Electronic device for providing information generated on basis of image

Abstract

An electronic device according to an embodiment may comprise a display, a memory for storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to: display, via the display, a still image including one or more image objects corresponding to text; acquire text information corresponding to the one or more image objects in the still image on the basis of an input for analyzing the still image being displayed; select a designated application corresponding to the still image from among a plurality of applications by using at least a portion of the text information; display, via the display, a screen including the still image and a visual object for launching the designated application; and, in response to an input related to the visual object, display a launch screen of the designated application via the display.

Description

An electronic device for providing information generated based on images

[0001] The descriptions below relate to electronic devices for providing information generated based on images.

[0002] Electronic devices, such as smartphones, can store image files. For example, electronic devices can store images captured through a camera, screenshots of the screen displayed on the display, and images downloaded from external sources. Electronic devices can provide applications that allow users to view and manage these images.

[0003] The above information may be provided as background information to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art in connection with the present disclosure.

[0004] According to an embodiment, an electronic device may include a display, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a still image including one or more image objects corresponding to text. The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the one or more image objects of the still image based on an input for analyzing the still image being displayed. The instructions, when executed by the processor, may cause the electronic device to select, from among a plurality of applications, a designated application corresponding to the still image using at least a portion of the text information. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a screen including a visual object for executing the designated application and the still image. The above instructions, when executed by the processor, may cause the electronic device to display, through the display, an execution screen of the specified application in response to an input related to the visual object.

[0005] According to one embodiment, an electronic device may include a display, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to display a screenshot image through the display. The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the screenshot image based on an input for analyzing the screenshot image being displayed. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a window including a visual object for executing an application based on identifying an application corresponding to the screenshot image using the text information, partially overlapping the screenshot image. The instructions, when executed by the processor, may cause the electronic device to display an execution screen of the application through the display in response to an input related to the visual object, based on identifying an application corresponding to the screenshot image using the text information.

[0006] According to one embodiment, an electronic device may include a display, a memory storing instructions, and a processor. The instructions, when executed by the processor, may cause the electronic device to display a first screen including an image through the display. The instructions, when executed by the processor, may cause the electronic device to extract text from the image by performing optical character recognition (OCR). The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the image from an artificial intelligence model into which the extracted text is input. The instructions, when executed by the processor, may cause the electronic device to identify a category of the image using the text information. The instructions, when executed by the processor, may cause the electronic device to display a second screen including the image and a visual object for executing the application based on identifying an application corresponding to the identified category. The instructions, when executed by the processor, may cause the electronic device to display a third screen different from the second screen by executing the application in response to an input related to the visual object.

[0007] According to one embodiment, a non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display, cause the electronic device to display a still image including one or more image objects corresponding to text through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain text information corresponding to the one or more image objects of the still image based on an input for analyzing the still image being displayed. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to select a designated application corresponding to the still image from among a plurality of applications using at least a portion of the text information. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a screen including a visual object for executing the designated application and the still image through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display, through the display, an execution screen of the specified application in response to an input related to the visual object.

[0008] According to one embodiment, a non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display, cause the electronic device to display a screenshot image through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain text information corresponding to the screenshot image based on an input for analyzing the screenshot image being displayed. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display, through the display, a window including a visual object for executing an application, partially overlapping the screenshot image, based on identifying an application corresponding to the screenshot image using the text information. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display, through the display, an execution screen of the application in response to an input related to the visual object, based on identifying an application corresponding to the screenshot image using the text information.

[0009] According to one embodiment, a non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display, cause the electronic device to display a first screen including an image through the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to extract text from the image by performing optical character recognition (OCR). The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain text information corresponding to the image from an artificial intelligence model into which the extracted text has been input. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to identify a category of the image using the text information. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a second screen including the image and a visual object for executing the application based on identifying an application corresponding to the identified category. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a third screen different from the second screen by executing the application in response to an input related to the visual object.

[0010] According to one embodiment, a method of an electronic device having a display may include an operation of displaying a still image including one or more image objects corresponding to text through the display. The method may include an operation of obtaining text information corresponding to the one or more image objects of the still image based on an input for analyzing the still image being displayed. The method may include an operation of selecting a designated application corresponding to the still image from among a plurality of applications using at least a portion of the text information. The method may include an operation of displaying a screen including a visual object for executing the designated application and the still image through the display. The method may include an operation of displaying an execution screen of the designated application through the display in response to an input related to the visual object.

[0011] According to one embodiment, a method of an electronic device having a display may include an operation of displaying a screenshot image through the display. The method may include an operation of obtaining text information corresponding to the screenshot image based on an input for analyzing the screenshot image being displayed. The method may include an operation of displaying a window including a visual object for executing the application, partially overlapping the screenshot image, through the display, based on identifying an application corresponding to the screenshot image using the text information. The method may include an operation of displaying an execution screen of the application through the display, in response to an input related to the visual object, based on identifying an application corresponding to the screenshot image using the text information.

[0012] According to one embodiment, a method of an electronic device having a display may include an operation of displaying a first screen including an image through the display. The method may include an operation of extracting text from the image by performing optical character recognition (OCR). The method may include an operation of obtaining text information corresponding to the image from an artificial intelligence model into which the extracted text is input. The method may include an operation of identifying a category of the image using the text information. The method may include an operation of displaying a second screen including the image and a visual object for executing the application based on identifying an application corresponding to the identified category. The method may include an operation of displaying a third screen different from the second screen by executing the application in response to an input related to the visual object.

[0013] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0014] FIG. 2A illustrates examples of screens displayed by an electronic device according to one embodiment.

[0015] FIG. 2b illustrates examples of screens displayed by an electronic device according to one embodiment.

[0016] FIG. 2c illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0017] FIG. 3A illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0018] FIG. 3b illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0019] FIG. 3c illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0020] FIG. 3D illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0021] FIG. 4A illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0022] FIG. 4b illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0023] FIG. 4c illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0024] FIG. 4d illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0025] FIG. 5A illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0026] FIG. 5b illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0027] FIG. 5c illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0028] FIG. 5d illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0029] FIG. 6 illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0030] FIG. 7 is a drawing showing an example of a screen displayed by an electronic device according to one embodiment.

[0031] FIG. 8 is a drawing showing an example of a screen displayed by an electronic device according to one embodiment.

[0032] FIG. 9 illustrates an example of a signal flow for a method of providing a function associated with text information generated based on an image, according to one embodiment.

[0033] FIG. 10 is a flowchart illustrating a method for providing functions associated with text information generated based on an image, according to one embodiment.

[0034] FIG. 11 illustrates examples of exemplary screens of an electronic device according to one embodiment.

[0035] FIG. 12A illustrates an example screen of an electronic device according to one embodiment.

[0036] FIG. 12b illustrates an example screen of an electronic device according to one embodiment.

[0037] FIG. 13 illustrates an example screen of an electronic device according to one embodiment.

[0038] FIG. 14a illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0039] FIG. 14b illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0040] FIG. 14c illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0041] FIG. 14d illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0042] FIG. 14e illustrates an example of a screen displayed by an electronic device according to one embodiment.

[0043] FIG. 15 is an exemplary block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0044] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0045] The processor (120) may control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) by executing, for example, software (e.g., a program (140)), and may perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculation, the processor (120) may store a command or data received from another component (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the command or data stored in the volatile memory (132), and store the resulting data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0046] The auxiliary processor (123) may control at least a part of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0047] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0048] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0049] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0050] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0051] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0052] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0053] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0054] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0055] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0056] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0057] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0058] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0059] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0060] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, Wi-Fi (wireless fidelity) direct, or IrDA (infrared data association)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0061] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) may support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0062] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). According to one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). According to one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the selected at least one antenna. According to some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0063] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0064] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0065] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service by itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0066] FIGS. 2A, 2B, and 2C illustrate examples of screens displayed by an electronic device according to an embodiment. The electronic device (101) of FIGS. 2A, 2B, and 2C may be an example of the electronic device (101) of FIG. 1 . For example, the display (260) of FIGS. 2A, 2B, and 2C may include at least a portion of the display module (160) of FIG. 1 .

[0067] In FIGS. 2a, 2b, and 2c, examples (251, 252, 253, 254) of how an electronic device (101) performs a function related to content within an image through analysis of the image are shown.

[0068] Referring to example (251), the electronic device (101) can display a screen (201) through the display (260). The screen (201) can include an image (210). The image (210) can include, for example, a still image, such as an image captured by a camera or an image downloaded or received from an external source. The image (210) can include, for example, a screenshot image (or capture image). The screenshot image can include an image classified as a screenshot image. For example, the screenshot image can be an image including a flag value indicating that it is a screenshot image. For example, but not limited to, the electronic device (101) can, in response to receiving an input for capturing the screen (201) being displayed through the display (260), obtain the captured image (210) and classify the image as the screenshot image. The screen (201) being displayed through the display (260) may include execution screens of various applications, such as SNS (social network service), messages, wallets, screen capture applications, and applications for playing photos or videos.

[0069] The image (210) may be distinguished from a preview image acquired through a camera of the electronic device (101) (e.g., the camera module (180) of FIG. 1). For example, the image (210) may be temporarily, permanently, or semi-permanently stored in a volatile memory (e.g., the volatile memory (132) of FIG. 1) or a non-volatile memory (e.g., the non-volatile memory (134) of FIG. 1) of the electronic device (101), such that substantially the same object may be displayed again through the display (260) based on a user's selection or some function of the electronic device (101).

[0070] In one embodiment, the image (210) may include a coupon image. The image (210) may include image objects representing the content of the coupon image. For non-limiting examples, the image (210) may include image objects (211, 212, 214, 216, 218). For example, the image objects (211, 212, 216) may correspond to text. For example, the image objects (214, 218) may correspond to non-text. For example, the image object (211) may represent the name of a product included in the coupon image. For example, the image object (212) may represent an attribute of the product, such as 'FREE'. Although not shown, the image (210) may include one or more image objects representing a category (e.g., food), a subcategory (e.g., hamburger), performance, and function as attributes of the product. For example, the image object (214) may represent the shape of the product. For example, the image object (216) may represent a coupon number. For example, the image object (218) may represent an image code of the coupon. For non-limiting examples, the image code may include an imaged code such as a QR code, a barcode, or a color code.

[0071] In one embodiment, the screen (201) may include a visual object (206) for performing recognition of an image (210) and / or displaying a recognition result for the image (210). For example, the electronic device (101) may obtain text corresponding to the image (210) based on receiving a user input for the visual object (206). For example, the electronic device (101) may perform optical character recognition (OCR) on the image (210) to extract text corresponding to image objects (211, 212, 214, 216, 218) included in the image (210). However, the method for extracting the text is not limited to the above-described OCR, and various technologies may be used.

[0072] In one embodiment, the electronic device (101) is described as obtaining text corresponding to an image (210) based on a user input for a visual object (206), but is not limited thereto. For example, the electronic device (101) may extract text included in a screen being displayed through the display (260) before receiving a user input for the visual object (206). For example, the electronic device (101) may extract text included in an image (210) before receiving a user input for the visual object (206). For example, the electronic device (101) may obtain text corresponding to image objects (211, 212, 214, 216, 218) included in the image (210) based on obtaining the image (210) (e.g., capturing a screen including the image (210)) and / or displaying the screen (201) including the image (210). In this case, the electronic device (101) may display a screen (205) to display texts extracted from an image (210) in response to a user input for a visual object (206).

[0073] Additionally or optionally, the visual object (206) may be selectively displayed depending on the image being displayed on the electronic device (101). For example, the electronic device (101) may identify whether a coupon image, such as the image (210), is being displayed through the screen (201). The electronic device (101) may identify that the image (210) is being displayed through the screen (201) and display the visual object (206). The electronic device (101) may identify that another image different from the coupon image is being displayed through the screen (201) and may not display the visual object (206).

[0074] Referring to example (252), the electronic device (101) may display a screen (205) based on receiving a user input for a visual object (206). The screen (205) may include an image (210). In one embodiment, the electronic device (101) may highlight and display image objects (211, 212, 214, 216, 218) identified through recognition of the image (210) through the screen (205). In example (252), an example is shown in which areas corresponding to the identified image objects (211, 212, 214, 216, 218) are displayed brighter than other areas of the screen (205), but the present invention is not limited thereto.

[0075] In one embodiment, the screen (205) may include a visual object (215) indicating that identified image objects (211, 212, 214, 216, 218) may be selected. Through input for each of the identified image objects (211, 212, 214, 216, 218), at least some of the identified image objects (211, 212, 214, 216, 218) may be selected.

[0076] In one embodiment, the electronic device (101) may obtain text information corresponding to the image (210) based on an input to a visual object (219) of the screen (205) to complete selection of identified image objects (211, 212, 214, 216, 218). The text information may include category information of the image (210), content information of the image (210), and summary information about the content information. The category information of the image (210) may include, for example, information indicating a subject of the image (210) (e.g., coupon). The content information of the image (210) may include, for example, at least one of a brand of a coupon provider, a coupon code (e.g., an identification number or barcode number of the coupon), content provided by the coupon, a start date and time, an expiration date and time, a usage method, and restrictions. The above summary information may be information that briefly summarizes the content information. In one embodiment, it has been described that the electronic device (101) obtains the text information based on an input to a visual object (219), but the present invention is not limited thereto. For example, the electronic device (101) may obtain the text information based on an input to a visual object (206) of the screen (201). As another example, the electronic device (101) may obtain text information corresponding to an image (210) by selecting at least some of the identified image objects (211, 212, 214, 216, 218) without an input to a visual object (219). For another example, the electronic device (101) may obtain the text information within the image (210) based on obtaining the image (210) (e.g., by capturing a screen including the image (210)) and / or displaying a screen (201) including the image (210).

[0077] For example, the electronic device (101) may obtain text information corresponding to the image (210) from an artificial intelligence model (e.g., a text information extraction model (903) of FIG. 9) into which the extracted text and a prompt are input from the identified image objects (211, 212, 214, 216, 218). The prompt may be a command in natural language for determining a category from the extracted text and generating information (e.g., the content information and the summary information) corresponding to the determined category in a designated format. For example, the artificial intelligence model may output the text information in a designated data format, such as JSON (javaScript object notation), in response to the prompt. For non-limiting examples, the artificial intelligence model may include a plurality of parameters for driving a neural network, such as BERT (bidirectional encoder representations from transformers) and / or GPT (generative pre-trained transformer). As a non-limiting example, the artificial intelligence model may include a large language model (LLM) for processing natural language based on massive parameters.

[0078] In one embodiment, when only some of the identified image objects (211, 212, 214, 216, 218) are selected through an input on the screen (205), the electronic device (101) may obtain the text information based on the selected image objects. For example, the electronic device (101) may obtain the text information corresponding to the image (210) from an artificial intelligence model into which text and a prompt extracted from the selected image objects among the identified image objects (211, 212, 214, 216, 218) are input. Additionally, in order for the electronic device (101) to obtain the text information, the image (210) may be input to the artificial intelligence model together with the extracted text.

[0079] In one embodiment, the electronic device (101) can identify the category of the image (210) using the text information. For example, the electronic device (101) can identify the category of the image (210) using the category information of the image (210) included in the text information.

[0080] Referring to example (253) of FIG. 2B, the electronic device (101) can display a screen (or window) (220) through the display (260). For example, the electronic device (101) can display the screen (220) together with the screen (205). For example, the electronic device (101) can display the screen (220) by partially overlapping the screen (205) through the display (260), but is not limited thereto. For example, the electronic device (101) can display the screen (205) having a first size in a portion of the entire display area of ​​the display (260), and display the screen (220) having a second size smaller than the first size in the remaining portion. For example, the screen (220) can be displayed as a split or pop-up for the screen (205).

[0081] In one embodiment, the screen (220) may include a visual object (221) representing the category information of the image (210). The visual object (221) may correspond to text such as, but is not limited to, "There is coupon information in the screenshot." For example, the screen (220) may include an icon related to the category of the image (210). In one embodiment, the electronic device (101) may display the screen (220) based on identifying an application corresponding to the category of the image (210). For a non-limiting example, the application corresponding to the coupon category of the image (210) may include an application for managing the coupon, such as a wallet application. As another example, the application corresponding to the coupon may include a payment application that supports payment using a technology such as near field communication (NFC) or magnetic secure transmission (MST).

[0082] Referring to example (254), in one embodiment, the electronic device (101) can display a screen (or window) (230) through the display (260). For example, the electronic device (101) can display the screen (230) together with the screen (205). For example, the electronic device (101) can display the screen (230) through the display (260) in response to receiving an input (e.g., a touch input or a swipe up gesture) for a visual object (221) of the screen (220). The screen (230) can have a larger size than the screen (220) and can be displayed as a partial overlap with the screen (205), but is not limited thereto. For example, the screen (230) can be displayed as a split or pop-up with respect to the screen (205). For example, the electronic device (101) may display the screen (205) in a portion of the entire display area of ​​the display (260) and display the screen (230) smaller than the screen (205) and larger than the screen (220) in the remaining portion. In one embodiment, the screen (230) is described as being displayed by an input to a visual object (221) of the screen (220), but is not limited thereto. For example, the screen (230) may be displayed after a certain period of time has elapsed after the display of the screen (220). In another example, the display of the screen (220) may be omitted, and the screen (230) may be displayed. For example, the example (253) of FIG. 2B may be omitted.

[0083] In one embodiment, the screen (230) may include a visual object (221) representing the category information of the image (210), a visual object (222) representing at least a portion of the summary information, and a visual object (223) for executing the application corresponding to the image (210). In FIG. 2B, one visual object (223) is illustrated as being displayed, but the present invention is not limited thereto. For example, as in the visual objects (732, 733) of FIG. 7, when there are multiple applications corresponding to the image (210), a plurality of visual objects (223) for executing them may also be displayed. For example, as in the visual objects (832, 833) of FIG. 8, when there are multiple categories corresponding to the image (210), visual objects for executing applications (e.g., a wallet application and a schedule application) corresponding to each of the multiple categories may be displayed. For example, a visual object (221) may be displayed in a first area of ​​a screen (230), a visual object (222) may be displayed in a second area of ​​a screen (230), and a visual object (223) may be displayed in a third area of ​​a screen (230). For example, the second area may be located between the first area and the third area. For example, with reference to the illustration in FIG. 2B, the third area may be located below the first area and the second area. In one embodiment, the electronic device (101) may display the screen (230) based on identifying an application corresponding to the category of the image (210).

[0084] The screen (230) including the visual object (223) has been described as being displayed by a user input to the screen (220), but is not limited thereto. For example, the electronic device (101) may, based on obtaining the text information about the image (210), display the visual object (223) or another screen including the same, instead of the screen (220) or the screen (230), through the display (260) (e.g., displayed together with the screen (201) or the screen (205) including the image (210).

[0085] Referring to example (255) of FIG. 2C, in one embodiment, the electronic device (101) may, in response to receiving an input related to a visual object (223), display a screen (240) different from the screen (230) through the display (260). For example, the electronic device (101) may display the screen (240) by executing the application corresponding to the image (210). The screen (240) may include an execution screen of the application. For example, the screen (240) may include an execution screen capable of performing a function of the application corresponding to the category of the image (210). For example, the screen (240) may include an execution screen of the application capable of storing the coupon of the image (210).

[0086] In one embodiment, the electronic device (101) may execute the application based at least in part on the text information corresponding to the image (210). For example, in response to an input for a visual object (223), the electronic device (101) may display a screen (240) in which at least a portion of the text information is entered into an input field of an execution screen of the application. For example, text information corresponding to objects selected from among image objects (211, 212, 214, 216, 218) may be entered into the input field of the execution screen of the application, but is not limited thereto.

[0087] Additionally, the electronic device (101) may display a screen (240) with an image (210) attached thereto, such as an image (241), in response to an input for a visual object (223). Additionally, before the screen (240) is displayed, a screen may be provided for the user to designate at least a portion of the image (210), such as the screen (1407) of FIG. 14D. In this case, the electronic device (101) may display a screen (240) with an image (241) attached thereto corresponding to the designated area of ​​the image (210).

[0088] FIGS. 3A, 3B, 3C, and 3D illustrate examples of screens displayed by an electronic device according to an embodiment. The electronic device (101) of FIGS. 3A, 3B, 3C, and 3D may be an example of the electronic device (101) of FIG. 1.

[0089] In FIGS. 3a, 3b, 3c and 3d, examples are shown of how an electronic device (101) performs a function related to content within an image through analysis of the image.

[0090] Referring to FIG. 3A, the electronic device (101) may display a screen (301) through the display (260). The screen (301) may include an image (310). The image (310) may include, for example, a still image and / or a screenshot image. For example, but not limited to, if the image (310) is the screenshot image, the electronic device (101) may acquire the image (310) in response to receiving an input for capturing the screen (301) being displayed through the display (260). In one embodiment, the image (310) may be a different image from the image (210) of FIG. 2A. As described below, the image (310) may be an image classified into a different category than the image (210). The screen (301) being displayed through the display (260) may include execution screens of various applications, such as SNS, messages, wallets, screen capture applications, and applications for playing photos or videos.

[0091] The image (310) may be distinguished from a preview image acquired through a camera of the electronic device (101) (e.g., the camera module (180) of FIG. 1). For example, the image (310) may be temporarily, permanently, or semi-permanently stored in the volatile memory or the non-volatile memory of the electronic device (101), and may be in a state where substantially the same object may be displayed again through the display (260) based on a user's selection or some function of the electronic device (101).

[0092] In one embodiment, the image (310) may include an image representing a specific area (or location). In FIG. 3A, the image (310) is illustrated as representing information about a specific restaurant, but is not limited to the illustrated example.

[0093] The image (310) may include image objects representing content within the image (310). For example, the image (310) may include image objects for describing the specific region. For non-limiting examples, the image (310) may include image objects (312, 314, 316, 318). For example, the image objects (312, 314, 316) may correspond to text. For example, the image object (318) may correspond to non-text. For example, the image object (312) may represent the name of a specific location, such as 'B coffee Yangjae branch'. For example, the image object (314) may represent the address of the specific location. For example, the image object (316) may represent the days and hours of operation of the specific location. For example, the image object (318) may represent a logo image of the specific location.

[0094] In one embodiment, the screen (301) may include a visual object (306) (e.g., a visual object (206)) for performing recognition of the image (310) and / or displaying a recognition result for the image (310). For example, the electronic device (101) may obtain text corresponding to the image (310) based on receiving a user input for the visual object (306). For example, the electronic device (101) may perform OCR on the image (310) to identify image objects (312, 314, 316, 318) included in the image (310) and extract text corresponding to the image objects (312, 314, 316, 318). However, the method for extracting the text is not limited to the above-described OCR, and various technologies may be used.

[0095] In one embodiment, the electronic device (101) is described as obtaining text corresponding to an image (310) based on a user input for a visual object (306), but is not limited thereto. For example, the electronic device (101) may extract text contained within the image (310) based on obtaining the image (310) (e.g., capturing a screen including the image (310)) and / or displaying a screen (301) including the image (310). In this case, the electronic device (101) may display a screen (305) for indicating texts extracted from the image (310) in response to a user input for the visual object (306). Additionally or alternatively, the visual object (306) may be selectively displayed depending on the image being displayed on the electronic device (101). For example, the electronic device (101) can identify whether an image representing a specific location, such as an image (310), is being displayed on the screen (301). The electronic device (101) can identify that the image (310) is being displayed on the screen (301) and display a visual object (306). The electronic device (101) can identify that a different image than the image representing the specific location is being displayed on the screen (301) and not display the visual object (306).

[0096] In one embodiment, the electronic device (101) may obtain text information corresponding to the image (310). For example, the electronic device (101) may obtain text information corresponding to the image (310) based on an input to a visual object (306) of the screen (301). As another example, the electronic device (101) may obtain the text information for the image (310) based on obtaining the image (310) (e.g., by capturing a screen including the image (310)) and / or displaying the screen (301) including the image (310). The text information may include category information of the image (310), content information of the image (310), and summary information about the content information. The category information of the image (310) may be information indicating a subject of the image (310), for example, a location. The content information of the image (310) is information about the content included in the image (310), and may include, for example, at least one of the name of the place, opening hours, closing hours, time zone, phone number, and address.

[0097] For example, the electronic device (101) can obtain the text information corresponding to the image (310) from the artificial intelligence model into which the extracted text and prompt are input from the image objects (312, 314, 316, 318). Additionally, in order for the electronic device (101) to obtain the text information, the image (310) together with the extracted text can be input to the artificial intelligence model.

[0098] Referring to FIG. 3B, the electronic device (101) can display a screen (or window) (320) through the display (260). For example, the electronic device (101) can display the screen (320) together with the screen (305) including the image (310). For example, the electronic device (101) can display the screen (320) by partially overlapping the screen (305) through the display (260), but is not limited thereto. For example, the electronic device (101) can display the screen (305) having a first size in a portion of the entire display area of ​​the display (260), and display the screen (320) having a second size smaller than the first size in the remaining portion. For example, the screen (320) can be displayed as a split or pop-up for the screen (305). In one embodiment, the screen (305) may be a screen including an image (310) and having visual objects (312, 314, 316) corresponding to text included in the image (310) highlighted. The electronic device (101) may display the screen (305) based on text extracted from the image (310).

[0099] Although not shown, prior to outputting the screen (320), an interface for selecting identified image objects (312, 314, 316, 318) (e.g., visual objects (215, 219) of the screen (205) of FIG. 2b) may be provided via the screen (305).

[0100] In one embodiment, the screen (320) may include a visual object (321) for indicating the category information of the image (310). The visual object (321) may correspond to text such as, but is not limited to, "There is place information in the screenshot." For example, the screen (320) may include an icon related to the category of the image (310). In one embodiment, the electronic device (101) may display the screen (320) based on identifying an application corresponding to the category of the image (310). For a non-limiting example, the application corresponding to the place, which is the category of the image (310), may include an application that can perform a function related to the place, such as a map application.

[0101] Referring to FIG. 3C, in one embodiment, the electronic device (101) may display a screen (or window) (330) through the display (260). For example, the electronic device (101) may display the screen (330) together with the screen (305). For example, the electronic device (101) may display the screen (330) through the display (260) in response to receiving an input (e.g., a touch input or a swipe-up gesture) for a visual object (321) of the screen (320). The screen (330) may have a larger size than the screen (320) and may be displayed as a partial overlap with the screen (305), but is not limited thereto. For example, the screen (330) may be displayed as a split or pop-up with respect to the screen (305). For example, the electronic device (101) may display the screen (305) in a portion of the entire display area of ​​the display (260) and display the screen (330) that is smaller than the screen (305) and larger than the screen (320) in the remaining portion. In one embodiment, the screen (330) is described as being displayed by an input to a visual object (321) of the screen (320), but is not limited thereto. For example, the screen (330) may be displayed after a certain period of time has elapsed after the display of the screen (320). In another example, the display of the screen (320) may be omitted and the screen (330) may be displayed.

[0102] In one embodiment, the screen (330) may include a visual object (321) representing the category information of the image (310), a visual object (322) representing at least a portion of the summary information, and a visual object (323) for executing the application corresponding to the image (310). In FIG. 3C, one visual object (323) is illustrated as being displayed, but the present invention is not limited thereto. For example, when there are multiple applications corresponding to the image (310), such as the visual objects (732, 733) of FIG. 7 or the visual objects (832, 833) of FIG. 8, respective visual objects for executing the multiple applications may be displayed. For example, the image (310) may include, in addition to text information about a specific location, text information about a schedule and / or appointment. In this case, the electronic device (101) may further display a visual object for executing an application associated with the text information about the schedule and / or appointment, for example, a calendar application, in addition to the visual object (323).

[0103] For example, a visual object (321) may be displayed in a first area of ​​a screen (330), a visual object (322) may be displayed in a second area of ​​a screen (330), and a visual object (323) may be displayed in a third area of ​​a screen (330). For example, the second area may be located between the first area and the third area. For example, with reference to the city of FIG. 3C, the third area may be located below the first area and the second area. In one embodiment, the electronic device (101) may display the screen (330) based on identifying an application corresponding to the category of the image (310).

[0104] The screen (330) including the visual object (323) has been described as being displayed by a user input to the screen (320), but is not limited thereto. For example, the electronic device (101) may, based on obtaining the text information about the image (310), display the visual object (323) or another screen including the same, instead of the screen (320) or the screen (330), through the display (260) (e.g., displayed together with the screen (301) or the screen (305) including the image (310).

[0105] Referring to FIG. 3D , in one embodiment, the electronic device (101) may, in response to receiving an input related to a visual object (323), display a screen (340) different from the screen (330) through the display (260). For example, the electronic device (101) may display the screen (340) by executing the application corresponding to the image (310). The screen (340) may include an execution screen of the application. For example, the screen (340) may include an execution screen capable of performing a function of the application corresponding to the category of the image (310). For example, the screen (340) may include an execution screen of the application capable of searching for the location of the image (310). In one embodiment, the application corresponding to the screen (340) (e.g., a map application) may be different from the application corresponding to the screen (240) (e.g., a wallet application).

[0106] In one embodiment, the electronic device (101) may execute the application based at least in part on the text information corresponding to the image (310). For example, the electronic device (101) may display a screen (340) showing a searched location of the image (310) in response to an input for a visual object (323). The search for the location may be performed through the application and / or an external server for providing a service using the application. The electronic device (101) may store map information corresponding to the screen (340) in the memory of the electronic device (101) (e.g., the memory (130) of FIG. 1) so that it may be provided without establishing a network with the external server. In FIG. 3D, a screen (340) indicating a path from a current location (341) to a location (342) corresponding to the location of the image (310) is illustrated, but the present invention is not limited to the illustrated example.

[0107] FIGS. 4A, 4B, 4C, and 4D illustrate examples of screens displayed by an electronic device according to an embodiment. The electronic device (101) of FIGS. 4A, 4B, 4C, and 4D may be an example of the electronic device (101) of FIG. 1.

[0108] In FIGS. 4a, 4b, 4c and 4d, examples are shown of how an electronic device (101) performs a function related to content within an image through analysis of the image.

[0109] Referring to FIG. 4A, the electronic device (101) may display a screen (401) through the display (260). The screen (401) may include an image (410). The image (410) may include, for example, a still image and / or a screenshot image. For example, but not limited to, if the image (410) is the screenshot image, the electronic device (101) may acquire the image (410) in response to receiving an input for capturing the screen (401) being displayed through the display (260). In one embodiment, the image (410) may be a different image from the images (210, 310). As described below, the image (410) may be an image classified into a different category from the images (210, 310). The screen (401) being displayed through the display (260) may include execution screens of various applications, such as SNS, messages, wallets, screen capture applications, and applications for playing photos or videos.

[0110] The image (410) may be distinguished from a preview image acquired through a camera of the electronic device (101) (e.g., the camera module (180) of FIG. 1). For example, the image (410) may be temporarily, permanently, or semi-permanently stored in the volatile memory or the non-volatile memory of the electronic device (101), and may be in a state where substantially the same object may be displayed again through the display (260) based on a user's selection or some function of the electronic device (101).

[0111] In one embodiment, the image (410) may include an image representing a specific product available for purchase. In FIG. 4A, the image (410) is illustrated as representing information about a toy castle available for purchase, but is not limited to the illustrated example.

[0112] The image (410) may include image objects representing content within the image (410). For example, the image (410) may include image objects for describing the specific product. For non-limiting examples, the image (410) may include image objects (412, 414, 416, 418). For example, the image objects (412, 414, 416) may correspond to text. For example, the image object (418) may correspond to non-text. For example, the image objects (412, 414) may represent attributes of the specific product, such as 'figure' or 'castle'. For example, the image object (416) may represent the price of the specific product. For example, the image object (418) may represent the shape of the specific product. Although not shown, the image (410) may include an image object indicating a path for purchasing the specific product, such as the address of a web page.

[0113] In one embodiment, the screen (401) may include a visual object (406) (e.g., a visual object (306)) for performing recognition of the image (410) and / or displaying a recognition result for the image (410). For example, the electronic device (101) may obtain text corresponding to the image (410) based on receiving a user input for the visual object (406). For example, the electronic device (101) may perform OCR on the image (410) to identify image objects (412, 414, 416, 418) included in the image (410) and extract text corresponding to the image objects (412, 414, 416, 418). However, the method for extracting the text is not limited to the above-described OCR, and various technologies may be used.

[0114] In one embodiment, the electronic device (101) is described as obtaining text corresponding to an image (410) based on a user input for a visual object (406), but is not limited thereto. For example, the electronic device (101) may extract text contained within the image (410) based on obtaining the image (410) (e.g., capturing a screen including the image (410)) and / or displaying a screen (401) including the image (410). In this case, the electronic device (101) may display a screen (405) for indicating texts extracted from the image (410) in response to a user input for the visual object (406).

[0115] Additionally or optionally, the visual object (406) may be selectively displayed depending on the image being displayed on the electronic device (101). For example, the electronic device (101) may identify whether an image representing a specific product, such as an image (410), is being displayed on the screen (401). The electronic device (101) may identify that the image (410) is being displayed on the screen (401) and display the visual object (406). The electronic device (101) may identify that a different image than the image representing the specific product is being displayed on the screen (401) and may not display the visual object (406).

[0116] In one embodiment, the electronic device (101) may obtain text information corresponding to the image (410). For example, the electronic device (101) may obtain text information corresponding to the image (410) based on an input to a visual object (406) of the screen (401). As another example, the electronic device (101) may obtain the text information for the image (410) based on obtaining the image (410) (e.g., by capturing a screen including the image (410)) and / or displaying the screen (401) including the image (410). The text information may include category information of the image (410), content information of the image (410), and summary information about the content information. The category information of the image (410) may be information indicating a subject of the image (410), for example, shopping. The content information of the image (410) is information about the content included in the image (410), and may include, for example, at least one of the brand name, product name, cost, offer price, sales promotion details, and purchase location of the specific product.

[0117] For example, the electronic device (101) can obtain the text information corresponding to the image (410) from the artificial intelligence model into which the extracted text and prompt are input from the image objects (412, 414, 416, 418). Additionally, in order for the electronic device (101) to obtain the text information, the image (410) together with the extracted text can be input to the artificial intelligence model.

[0118] Referring to FIG. 4B, the electronic device (101) can display a screen (or window) (420) through the display (260). For example, the electronic device (101) can display the screen (420) together with the screen (405). For example, the electronic device (101) can display the screen (420) by partially overlapping the screen (405) through the display (260), but is not limited thereto. For example, the electronic device (101) can display the screen (405) having a first size in a portion of the entire display area of ​​the display (260), and display the screen (420) having a second size smaller than the first size in the remaining portion. For example, the screen (420) can be displayed as a split or pop-up for the screen (405). In one embodiment, the screen (405) may be a screen including an image (410) and having visual objects (412, 414, 416, 418) identified within the image (410) highlighted. The electronic device (101) may display the screen (405) based on text extracted from the image (410).

[0119] Although not shown, an interface for selecting identified image objects (412, 414, 416, 418) prior to output on the screen (420) may be provided (e.g., visual objects (215, 219) on the screen (205) of FIG. 2b).

[0120] In one embodiment, the screen (420) may include a visual object (421) for indicating the category information of the image (410). The visual object (421) may correspond to text such as, but is not limited to, "There is shopping information in the screenshot." For example, the screen (420) may include an icon related to the category of the image (410). In one embodiment, the electronic device (101) may display the screen (420) based on identifying an application corresponding to the category of the image (410). For a non-limiting example, the application corresponding to the shopping category of the image (410) may include an application for providing a reminder for the shopping, such as a reminder application.

[0121] Referring to FIG. 4C, in one embodiment, the electronic device (101) may display a screen (or window) (430) through the display (260). For example, the electronic device (101) may display the screen (430) together with the screen (405). For example, the electronic device (101) may display the screen (430) through the display (260) in response to receiving an input (e.g., a touch input or a swipe-up gesture) for a visual object (421) of the screen (420). The screen (430) may have a larger size than the screen (420) and may be displayed as a partial overlap with the screen (405), but is not limited thereto. For example, the screen (430) may be displayed as a split or pop-up with respect to the screen (405). For example, the electronic device (101) may display the screen (405) in a portion of the entire display area of ​​the display (260) and display the screen (430) that is smaller than the screen (405) and larger than the screen (420) in the remaining portion. In one embodiment, the screen (430) is described as being displayed by an input to a visual object (421) of the screen (420), but is not limited thereto. For example, the screen (430) may be displayed after a certain period of time has elapsed after the display of the screen (420). In another example, the display of the screen (420) may be omitted and the screen (430) may be displayed.

[0122] In one embodiment, the screen (430) may include a visual object (421) representing the category information of the image (410), a visual object (422) representing at least a portion of the summary information, and a visual object (423) for executing the application corresponding to the image (410). In FIG. 4C, one visual object (423) is illustrated as being displayed, but the present invention is not limited thereto. For example, when there are multiple applications corresponding to the image (410), such as the visual objects (732, 733) of FIG. 7 or the visual objects (832, 833) of FIG. 8, respective visual objects for executing the multiple applications may be displayed.

[0123] For example, a visual object (421) may be displayed in a first area of ​​a screen (430), a visual object (422) may be displayed in a second area of ​​a screen (430), and a visual object (423) may be displayed in a third area of ​​a screen (430). For example, the second area may be located between the first area and the third area. For example, with reference to the illustration in FIG. 4C, the third area may be located below the first area and the second area. In one embodiment, the electronic device (101) may display the screen (430) based on identifying an application corresponding to the category of the image (410).

[0124] The screen (430) including the visual object (423) has been described as being displayed by a user input to the screen (420), but is not limited thereto. For example, the electronic device (101) may, based on obtaining the text information about the image (410), display the visual object (423) or another screen including the same, instead of the screen (420) or the screen (430), through the display (260) (e.g., displayed together with the screen (401) or the screen (405) including the image (410).

[0125] Referring to FIG. 4D, in one embodiment, the electronic device (101) may, in response to receiving an input related to a visual object (423), display a screen (440) different from the screen (430) through the display (260). For example, the electronic device (101) may display the screen (440) by executing the application corresponding to the image (410). The screen (440) may include an execution screen of the application. For example, the screen (440) may include an execution screen capable of performing a function of the application corresponding to the category of the image (410). For example, the screen (440) may include an execution screen of the application capable of storing shopping information corresponding to the image (410). In one embodiment, the application corresponding to screen (440) (e.g., a reminder application) may be different from the application corresponding to screen (240) (e.g., a wallet application) and the application corresponding to screen (340) (e.g., a map application).

[0126] In one embodiment, the electronic device (101) may execute the application based at least in part on the text information corresponding to the image (410). For example, in response to an input for a visual object (423), the electronic device (101) may display a screen (440) in which at least a portion of the text information corresponding to the image (410) is entered into an input field of an execution screen of the application. For example, text information corresponding to objects selected from among image objects (412, 414, 416, 418) may be entered into the input field of the execution screen of the application, but is not limited thereto.

[0127] Additionally, the electronic device (101) may display a screen (440) in response to an input for a visual object (423), with an image (441) attached, at least in part based on the image (410). Additionally, before the screen (440) is displayed, a screen may be provided for the user to designate at least a portion of the image (410), such as the screen (1407) of FIG. 14D. In this case, the screen (440) may be displayed with an image (441) attached corresponding to the designated portion of the image (410).

[0128] FIGS. 5A, 5B, 5C, and 5D illustrate examples of screens displayed by an electronic device according to an embodiment. The electronic device (101) of FIGS. 5A, 5B, 5C, and 5D may be an example of the electronic device (101) of FIG. 1.

[0129] In FIGS. 5a, 5b, 5c and 5d, examples are shown of how an electronic device (101) performs a function related to content within an image through analysis of the image.

[0130] Referring to FIG. 5A, the electronic device (101) may display a screen (501) through the display (260). The screen (501) may include an image (510). The image (510) may include, for example, a still image and / or a screenshot image. For example, but not limited to, if the image (510) is the screenshot image, the electronic device (101) may acquire the image (510) in response to receiving an input for capturing the screen (501) being displayed through the display (260). In one embodiment, the image (510) may be a different image from the images (210, 310, 410). As described below, the image (510) may be an image classified into a different category from the images (210, 310, 410). The screen (501) being displayed through the display (260) may include execution screens of various applications, such as SNS, messages, wallets, screen capture applications, and applications for playing photos or videos.

[0131] The image (510) may be distinguished from a preview image acquired through a camera of the electronic device (101) (e.g., the camera module (180) of FIG. 1). For example, the image (510) may be temporarily, permanently, or semi-permanently stored in the volatile memory or the non-volatile memory of the electronic device (101), and may be in a state where substantially the same object may be displayed again through the display (260) based on a user's selection or some function of the electronic device (101).

[0132] In one embodiment, the image (510) may include a ticket image. In FIG. 5A, the image (510) is illustrated as a movie ticket image, but is not limited to the illustrated example. For example, the image (510) may include an image representing a reserved schedule, such as an airline ticket.

[0133] The image (510) may include image objects representing content within the image (510). For example, the image (510) may include image objects representing information about a ticket. For non-limiting examples, the image (510) may include image objects (512, 514, 516, 517, 518). For example, the image objects (512, 514, 516, 517) may correspond to text. For example, the image object (518) may correspond to non-text. For example, the image object (512) may represent a movie title. For example, the image object (514) may represent a movie showing date and showing time. For example, the image object (516) may represent reservation details such as the theater, theater, number of people, seats, and amount. For example, the image object (517) may represent payment information. For example, an image object (518) may represent a movie poster.

[0134] In one embodiment, the screen (501) may include a visual object (506) (e.g., a visual object (406)) for performing recognition of the image (510) and / or displaying a recognition result for the image (510). For example, the electronic device (101) may obtain text corresponding to the image (510) based on receiving a user input for the visual object (506). For example, the electronic device (101) may perform OCR on the image (510) to identify image objects (512, 514, 516, 517, 518) included in the image (510) and extract text corresponding to the image objects (512, 514, 516, 517, 518). However, the method for extracting the text is not limited to the above-described OCR, and various technologies may be used.

[0135] In one embodiment, the electronic device (101) is described as obtaining text corresponding to an image (510) based on a user input for a visual object (506), but is not limited thereto. For example, the electronic device (101) may extract text included in a screen being displayed through the display (260) before receiving a user input for the visual object (506). For example, the electronic device (101) may extract text included in an image (510) before receiving a user input for the visual object (506). For example, the electronic device (101) may obtain text corresponding to image objects (512, 514, 516, 517, 518) included in the image (510) based on obtaining the image (510) (e.g., capturing a screen including the image (510)) and / or displaying a screen (501) including the image (510). In this case, the electronic device (101) may display a screen (505) to display texts extracted from an image (510) in response to a user input for a visual object (506).

[0136] Additionally or optionally, the visual object (506) may be selectively displayed depending on the image being displayed on the electronic device (101). For example, the electronic device (101) may identify whether a ticket image, such as the image (510), is being displayed through the screen (501). The electronic device (101) may identify that a ticket image, such as the image (510), is being displayed through the screen (501) and display the visual object (506). The electronic device (101) may identify that a different image than the ticket image is being displayed through the screen (501) and may not display the visual object (506).

[0137] In one embodiment, the electronic device (101) may obtain text information corresponding to the image (510). For example, the electronic device (101) may obtain text information corresponding to the image (510) based on an input to a visual object (506) of the screen (501). As another example, the electronic device (101) may obtain the text information for the image (510) based on obtaining the image (510) (e.g., by capturing a screen including the image (510)) and / or displaying the screen (501) including the image (510). The text information may include category information of the image (510), content information of the image (510), and summary information about the content information. The category information of the image (510) may be information indicating a subject of the image (510), for example, a schedule. The content information of the image (510) is information about the content included in the image (510), and may include, for example, at least one of a title, a start date, a start time, a start time zone, an end date, an end time zone, a start location, an end location, and a reservation location.

[0138] For example, the electronic device (101) can obtain the text information corresponding to the image (510) from the artificial intelligence model into which the extracted text and prompt are input from the image objects (512, 514, 516, 517, 518). Additionally, in order for the electronic device (101) to obtain the text information, the image (510) together with the extracted text can be input to the artificial intelligence model.

[0139] Referring to FIG. 5B, the electronic device (101) can display a screen (or window) (520) through the display (260). For example, the electronic device (101) can display the screen (520) together with the screen (505). For example, the electronic device (101) can display the screen (520) by partially overlapping the screen (505) through the display (260), but is not limited thereto. For example, the electronic device (101) can display the screen (505) having a first size in a portion of the entire display area of ​​the display (260), and display the screen (520) having a second size smaller than the first size in the remaining portion. For example, the screen (520) can be displayed as a split or pop-up for the screen (505). In one embodiment, the screen (505) may be a screen including an image (510) and having visual objects (512, 514, 516, 517) identified within the image (510) highlighted. The electronic device (101) may display the screen (505) based on text extracted from the image (510).

[0140] Although not shown, an interface for selecting identified image objects (512, 514, 516, 517, 518) prior to output on the screen (520) may be provided (e.g., visual objects (215, 219) on the screen (205) of FIG. 2b).

[0141] In one embodiment, the screen (520) may include a visual object (521) for indicating the category information of the image (510). The visual object (521) may correspond to text such as, but is not limited to, "There is reservation information in the screenshot." For example, the screen (520) may include an icon related to the category of the image (510). In one embodiment, the electronic device (101) may display the screen (520) based on identifying an application corresponding to the category of the image (510). For a non-limiting example, the application corresponding to the schedule, which is the category of the image (510), may include an application for storing and managing the schedule, such as a calendar application.

[0142] Referring to FIG. 5C, in one embodiment, the electronic device (101) may display a screen (or window) (530) through the display (260). For example, the electronic device (101) may display the screen (530) together with the screen (505). For example, the electronic device (101) may display the screen (530) through the display (260) in response to receiving an input (e.g., a touch input or a swipe-up gesture) for a visual object (521) of the screen (520). The screen (530) may have a larger size than the screen (520) and may be displayed as a partial overlap with the screen (505), but is not limited thereto. For example, the screen (530) may be displayed as a split or pop-up with respect to the screen (505). For example, the electronic device (101) may display the screen (505) in a portion of the entire display area of ​​the display (260) and display the screen (530) that is smaller than the screen (505) and larger than the screen (520) in the remaining portion. In one embodiment, the screen (530) is described as being displayed by an input to a visual object (521) of the screen (520), but is not limited thereto. For example, the screen (530) may be displayed after a certain period of time has elapsed after the display of the screen (520). In another example, the display of the screen (520) may be omitted and the screen (530) may be displayed.

[0143] In one embodiment, the screen (530) may include a visual object (521) representing the category information of the image (510), a visual object (522) representing at least a portion of the summary information, and a visual object (523) for executing the application corresponding to the image (510). In FIG. 5C, one visual object (523) is illustrated as being displayed, but the present invention is not limited thereto. For example, when there are multiple applications corresponding to the image (510), such as the visual objects (732, 733) of FIG. 7 or the visual objects (832, 833) of FIG. 8, respective visual objects for executing the multiple applications may be displayed.

[0144] For example, a visual object (521) may be displayed in a first area of ​​a screen (530), a visual object (522) may be displayed in a second area of ​​the screen (530), and a visual object (523) may be displayed in a third area of ​​the screen (530). For example, the second area may be located between the first area and the third area. For example, with reference to the illustration in FIG. 5C, the third area may be located below the first area and the second area. In one embodiment, the electronic device (101) may display the screen (530) based on identifying an application corresponding to the category of the image (510).

[0145] The screen (530) including the visual object (523) has been described as being displayed by a user input to the screen (520), but is not limited thereto. For example, the electronic device (101) may, based on obtaining the text information about the image (510), display the visual object (523) or another screen including it, instead of the screen (520) or the screen (530), through the display (260) (e.g., displayed together with the screen (501) or the screen (505) including the image (510).

[0146] Referring to FIG. 5D, in one embodiment, the electronic device (101) may, in response to receiving an input related to a visual object (523), display a screen (540) different from the screen (530) through the display (260). For example, the electronic device (101) may display the screen (540) by executing the application corresponding to the image (510). The screen (540) may include an execution screen of the application. For example, the screen (540) may include an execution screen capable of performing a function of the application corresponding to the category of the image (510). For example, the screen (540) may include an execution screen of the application capable of storing schedule information corresponding to the image (510). In one embodiment, an application corresponding to screen (540) (e.g., a calendar application) may be different from an application corresponding to screen (240) (e.g., a wallet application), an application corresponding to screen (340) (e.g., a map application), and an application corresponding to screen (440) (e.g., a reminder application).

[0147] In one embodiment, the electronic device (101) may execute the application based at least in part on the text information corresponding to the image (510). For example, in response to an input for a visual object (523), the electronic device (101) may display a screen (540) in which at least a portion of the text information corresponding to the image (510) is entered into an input field of an execution screen of the application. For example, text information corresponding to selected objects among image objects (e.g., image objects (512, 514, 516, 517)) identified in the image (510) may be entered into the input field of the execution screen of the application, but is not limited thereto.

[0148] In one embodiment, the electronic device (101) is described as obtaining the text information by using the extracted text from an image (e.g., images (210, 310, 410, or 510)) and the artificial intelligence model into which the prompt is input, but is not limited thereto. For example, before obtaining the text information, the electronic device (101) may obtain category information of the image. For example, the electronic device (101) may obtain the category information from the text extracted from the image and the artificial intelligence model into which a first prompt for generating category information from the extracted text is input. Thereafter, the electronic device (101) may obtain the text information from the artificial intelligence model into which a second prompt for generating content information corresponding to the category information from the extracted text is input. As another example, a machine learning-based model, rather than a prompt-based generative model, may be used to obtain the text information.

[0149] In one embodiment, the electronic device (101) may include the artificial intelligence model. For example, the artificial intelligence model may be stored in the memory of the electronic device (101) (e.g., the memory (130) of FIG. 1). In one embodiment, the artificial intelligence model may be included in a server (e.g., the server (108) of FIG. 1) external to the electronic device (101). For example, the artificial intelligence model may be stored in the memory of the server. In this case, the electronic device (101) may transmit text extracted from the image to the server via a communication circuit (e.g., the communication module (190) of FIG. 1). Thereafter, the electronic device (101) may receive the text information from the server via the communication circuit. Additionally, the electronic device (101) may transmit the image together with the text extracted from the image to the server to obtain the text information. In one embodiment, the electronic device (101) may obtain additional information about the image using the image and / or the text information about the image. For example, the electronic device (101) may obtain the additional information by searching for the image and at least a portion of the text information about the image through a network such as the Internet. For example, the additional information may include information that is not included in the text information about the image. For example, the image may be a coffee coupon (e.g., image (210) of FIG. 2A) and may not include information about a specific location where the coffee coupon can be used. In this case, the electronic device (101) may obtain information about the specific location as the additional information through a search using the image and / or the text information based on the image.

[0150] In one embodiment, the electronic device (101) can identify the category of the image using the text information. For example, the electronic device (101) can identify the category of the image using the category information of the image (510) included in the text information.

[0151] FIG. 6 illustrates examples of screens displayed by an electronic device according to one embodiment. The electronic device (101) of FIG. 6 may be an example of the electronic device (101) of FIG. 1.

[0152] Referring to FIG. 6, the electronic device (101) can display a screen (601) through the display (260). The screen (601) can include an image (610). The image (610) can include, for example, a still image and / or a screenshot image. For example, but not limited to, if the image (610) is the screenshot image, the electronic device (101) can acquire the image (610) in response to receiving an input for capturing the screen (601) being displayed through the display (260).

[0153] In one embodiment, the electronic device (101) can extract text corresponding to an image object included in an image (610). Using the extracted text, the electronic device (101) can obtain text information corresponding to the image (610).

[0154] The electronic device (101) can identify the category of the image (610) using the text information. For example, the image (610) may have a different category from the aforementioned images (210, 310, 410, 510). For example, the category of the image (610) may include movie information, food information, music information, technical information, news, chat, meeting log, document, personal information, rune information, SNS, or other categories. In this case, the electronic device (101) may determine that there is no application corresponding to the category of the image (610) detected using the text information. Based on the determination that there is no application corresponding to the category of the image (610), the electronic device (101) may display the screen (602) partially overlapping the screen (601) through the display (260).

[0155] In one embodiment, the screen (602) may include a visual object (621) indicating that there are no actions available for the image (610). As a non-limiting example, the visual object (621) may include text such as 'No actions available.'

[0156] Although not shown, the categories of images detected using the above text information and the applications corresponding to the categories may be changed according to the user's settings.

[0157] For example, even if the category of the image (610) is identified as the food information, the category of the image (610) may be changed to a different category (e.g., shopping) from the food information, depending on the user's settings. Thereafter, the electronic device (101) may use the text information to determine the category corresponding to the image (610) as the different category.

[0158] For example, a specific application corresponding to a category of an image (610) may be designated by the user's settings. After the specific application is designated, the electronic device (101) may display a visual object for executing the specific application using the text information of the image (610).

[0159] In one embodiment, the text information is described as being generated based on an image and text extracted from the image, but is not limited thereto. For example, the electronic device (101) may perform an Internet search using at least a portion of the text extracted from the image. In this case, the electronic device (101) may generate the text information including other information not included in the image based on the Internet search.

[0160] FIG. 7 is a diagram illustrating an example of a screen displayed by an electronic device according to one embodiment. The screen (730) illustrated in FIG. 7 may be a screen displayed through the display (260) of the electronic device (101) of FIGS. 2A to 6 .

[0161] In one embodiment, applications corresponding to categories of images (e.g., images (210, 310, 410, 510)) are described as being assigned one-to-one, but this is not limited thereto. For example, multiple applications may be assigned to correspond to one category.

[0162] For example, although not shown, an image may include image objects representing information about books available for purchase. In this case, the category of the image, detected using text information of the image, may be book information. In this case, the electronic device (101) may display a screen (730) including a visual object (731) representing the category of the image, a visual object (732) for executing a web browser application for purchasing the book shown in the image, and a visual object (733) for executing a reading application.

[0163] FIG. 8 is a diagram illustrating an example of a screen displayed by an electronic device according to one embodiment. The screen (830) illustrated in FIG. 8 may be a screen displayed through the display (260) of the electronic device (101) of FIGS. 2A to 6 .

[0164] In one embodiment, categories corresponding to images (e.g., images (210, 310, 410, 510)) are described as being assigned 1:1, but this is not limited thereto. For example, multiple categories may be assigned to a single image.

[0165] For example, although not shown, the image may include a coupon image, such as image (210), and an image of a specific purchasable product, such as image (310). In this case, the categories of the image, detected using text information of the image, may be the coupon and the shopping. In this case, the electronic device (101) may display a screen (830) including a visual object (832) for executing a wallet application corresponding to the coupon category and a visual object (833) for executing a reminder application corresponding to the shopping category.

[0166] In one embodiment, the electronic device (101) may display another visual object (not shown) for storing text information in an image in multiple applications. For example, in response to an input to the other visual object, the electronic device (101) may store the image and / or the text information corresponding to the image in the wallet application and the reminder application, or display a screen for storing the image and / or the text information corresponding to the image.

[0167] FIG. 9 illustrates an example of a signal flow for a method of providing a function associated with text information generated based on an image, according to one embodiment.

[0168] The first application (901), the image analysis model (902), the text information extraction model (903), and the second application (904) illustrated in FIG. 9 may be stored in the memory of the electronic device (101) of FIG. 1 (e.g., the memory (130) of FIG. 1). By being executed by the processor (120) of the electronic device (101), the operations of the first application (901), the image analysis model (902), the text information extraction model (903), and the second application (904) may be performed. In one embodiment, the electronic device (101) or the text information extraction model (903) may include at least one of the blocks of the artificial intelligence system of FIG. 15. Alternatively or optionally, the image analysis model (902) may be included in a server (e.g., the server (108) of FIG. 1) external to the electronic device (101) and executed by the server. Alternatively or optionally, the text information extraction model (903) may be included in a server (e.g., server (108) of FIG. 1) external to the electronic device (101) and executed by the server.

[0169] Referring to FIG. 9, in one embodiment, the first application (901) may include a capture application capable of capturing a screen being displayed through a display (260) of the electronic device (101).

[0170] In one embodiment, the image analysis model (902) may be configured to obtain text from image objects contained within an image. For example, the image analysis model (902) may extract the text by performing OCR on the input image.

[0171] In one embodiment, the text information extraction model (903) may be trained to generate text information based on text. The text information extraction model (903) may be trained to generate the text information based on images and text.

[0172] In one embodiment, the second application (904) may include at least one application corresponding to a category of an image detected using the text information.

[0173] In operation (905), the electronic device (101) can obtain a screenshot image using the first application (901). For example, the electronic device (101) can obtain an image (210) using the first application (901). For example, the electronic device (101) can obtain an image (310) using the first application (901). For example, the electronic device (101) can obtain an image (410) using the first application (901). For example, the electronic device (101) can obtain an image (510) using the first application (901).

[0174] In operation (910), the electronic device (101) may transmit the screenshot image from the first application (901) to the image analysis model (902). For example, the electronic device (101) may perform operation (910) based on a user input related to the screenshot image. For example, the electronic device (101) may obtain the screenshot image using the first application (901) based on receiving a user input for capturing a screen being displayed through the display (260), and transmit the obtained screenshot image from the first application (901) to the image analysis model (902). For example, the electronic device (101) may transmit the image (210) from the first application (901) to the image analysis model (902) based on an input for a visual object (206) of the screen (201).

[0175] For example, the electronic device (101) can transmit an image (310) from a first application (901) to an image analysis model (902) based on an input to a visual object (306) of a screen (301). For example, the electronic device (101) can transmit an image (410) from a first application (901) to an image analysis model (902) based on an input to a visual object (406) of a screen (401). For example, the electronic device (101) can transmit an image (510) from a first application (901) to an image analysis model (902) based on an input to a visual object (506) of a screen (501).

[0176] In operation (915), the electronic device (101) may extract text from the screenshot image using the image analysis model (902). For example, the electronic device (101) may extract text corresponding to image objects (211, 212, 216) in the image (210). For example, the electronic device (101) may extract text corresponding to image objects (312, 314, 316) in the image (310). For example, the electronic device (101) may extract text corresponding to image objects (412, 414, 416) in the image (410). For example, the electronic device (101) can extract text corresponding to image objects (512, 514, 516, 517) in the image (510). In operation (920), the electronic device (101) can transmit the extracted text from the image analysis model (902) to the first application (901). For example, the electronic device (101) can transmit the text extracted from the image objects (211, 212, 216) in the image (210) from the image analysis model (902) to the first application (901). For example, the electronic device (101) can transmit the text extracted from the image objects (312, 314, 316) in the image (310) from the image analysis model (902) to the first application (901). For example, the electronic device (101) can transmit text extracted from image objects (412, 414, 416) within the image (410) from the image analysis model (902) to the first application (901). For example, the electronic device (101) can transmit text extracted from image objects (512, 514, 516, 517) within the image (510) from the image analysis model (902) to the first application (901).

[0177] In operation (930), the electronic device (101) may transmit the extracted text from the first application (901) to the text information extraction model (903). Additionally, the electronic device (101) may transmit a screenshot image and / or a prompt of operation (910) together with the extracted text from the first application (901) to the text information extraction model (903). For example, the electronic device (101) may transmit a prompt in which the extracted text is embedded from the first application (901) to the text information extraction model (903).

[0178] In operation (935), the electronic device (101) may generate text information using a text information extraction model (903). The text information may include the category information, the content information, and the summary information.

[0179] For example, the text information for the image (210) may indicate that the image (210) falls into the 'coupon' category and may include information about at least one of the brand of the coupon provider, the coupon code (e.g., the coupon's identification number or barcode number), what the coupon provides, the start date and time, the expiration date and time, how to use it, and restrictions.

[0180] For example, the text information for the image (310) may indicate that the image (310) falls into the 'place' category and may include information about at least one of the place's name, opening hours, closing hours, time zone, phone number, and address.

[0181] For example, the text information for the image (410) may indicate that the image (410) belongs to the 'shopping' category and may include information on at least one of the brand name, product name, cost price, offer price, sales promotion details, and purchase location of a specific product.

[0182] For example, the text information for the image (410) may indicate that the image (410) falls into the 'schedule' category and may include information about at least one of a title, a start date, a start time, a start time zone, an end date, an end time zone, a start location, an end location, and a reservation location.

[0183] In one embodiment, the electronic device (101) is described as obtaining all of the text information from the text information extraction model (903) into which the prompt embedded with the extracted text is input, but is not limited thereto. For example, the electronic device (101) may transmit a first prompt for generating category information from the extracted text from the first application (901) to the text information extraction model (903). Thereafter, the electronic device (101) may transmit the category information from the text information extraction model (903) into which the first prompt is input to the first application (901). The electronic device (101) may transmit a second prompt for generating content information corresponding to the category information from the extracted text from the first application (901) to the text information extraction model (903).

[0184] In operation (940), the electronic device (101) can transmit the generated text information from the text information extraction model (903) to the first application (901).

[0185] In operation (950), the electronic device (101) may transmit text information generated from the first application (901) to the text information extraction model (903). For example, the electronic device (101) may display a visual object (e.g., visual objects (223, 323, 423, 523)) for executing at least one application corresponding to a category of an image identified using the generated text information using the first application (901). The electronic device (101) may perform operation (950) based on an input for the visual object (e.g., visual objects (223, 323, 423, 523)) for executing the at least one application.

[0186] In operation (960), the electronic device (101) may execute a second application (904) based on text information. For example, the second application (904) corresponding to the image (210) may include a wallet application. The electronic device (101) may execute the wallet application based on the generated text information for the image (210). For example, the electronic device (101) may display an execution screen (240) of the coupon application for storing a coupon corresponding to the image (210). At least a portion of the text information generated for the image (210) may be input into an input field of the execution screen (240) of the coupon application.

[0187] For example, the second application (904) corresponding to the image (310) may include a map application. The electronic device (101) may execute the map application based on the generated text information for the image (310). For example, the electronic device (101) may display an execution screen (340) of the map application to provide information about a location corresponding to the image (310). The execution screen (340) of the map application may display a route to the location corresponding to the image (310).

[0188] For example, the second application (904) corresponding to the image (410) may include a reminder application. The electronic device (101) may execute the reminder application based on the generated text information for the image (410). For example, the electronic device (101) may display an execution screen (440) of the reminder application for registering reminder shopping information corresponding to the image (410). At least a portion of the text information generated for the image (410) may be input into an input field of the execution screen (440) of the reminder application.

[0189] For example, the second application (904) corresponding to the image (510) may include a schedule application. The electronic device (101) may execute the schedule application based on the generated text information for the image (510). For example, the electronic device (101) may display an execution screen (540) of the schedule application for registering a schedule corresponding to the image (510). At least a portion of the text information generated for the image (510) may be input into an input field of the execution screen (540) of the schedule application.

[0190] FIG. 10 is a flowchart illustrating a method for providing functions related to text information generated based on an image, according to an embodiment. The operations illustrated in FIG. 10 may be performed by the electronic device (101) of FIG. 1 . For example, the processor (120) of the electronic device (101) may perform the operations of FIG. 10 . For example, instructions stored in a memory (e.g., the memory (130) of FIG. 1 ) may, when executed by the processor (120), cause the electronic device (101) to perform the operations of FIG. 10 .

[0191] Referring to FIG. 10, in operation (1010), the electronic device (101) can display an image (e.g., images (210, 310, 410, 510)) through a display.

[0192] In operation (1020), the electronic device (101) may obtain text information corresponding to the image based on an input for analyzing the image being displayed (e.g., input for objects (206, 306, 406, 506)). The text information may include category information of the image, content information, and summary information about the content information.

[0193] In operation (1030), the electronic device (101) may display a visual object (e.g., visual objects (233, 333, 433, 533)) for executing the application based on identifying an application corresponding to the image using text information. For example, the electronic device (101) may display a window (e.g., screen (230, 330, 430, 530)) including the visual object. For example, the electronic device (101) may display the visual object or the window including the visual object together with the image. For example, the electronic device (101) may display the visual object or the window including the visual object in a manner of partially overlapping the image, but is not limited thereto.

[0194] In operation (1040), the electronic device (101) may display an execution screen of the application (e.g., screens (240, 340, 440, 540)) in response to an input related to a visual object.

[0195] FIG. 11 illustrates examples of exemplary screens of an electronic device according to an embodiment. The electronic device (101) of FIG. 11 may be an example of the electronic device (101) of FIG. 1. For example, the display (260) may include the display module (160) of FIG. 1.

[0196] Referring to example (1150), the electronic device (101) can display a screen (1101) including an image (1110) through the display (260). The description of the image (1110) and the screen (1101) can be substantially identically applied to the description of the image (210) and the screen (201).

[0197] In one embodiment, the electronic device (101) may display, through the display (260), a screen (1130) that partially overlaps the screen (1101) and includes a visual object (1123). In one embodiment, the description of the visual object (1123) and the screen (1130) may be substantially identical to the description of the visual object (223) and the screen (230).

[0198] In one embodiment, the electronic device (101) may receive an input for a portion (1125) of a screen (1130). For example, the portion (1125) may include an area within the screen (1130) excluding an area corresponding to a visual object (1123). The input may include, but is not limited to, a swipe input moving left or right with respect to the city of FIG. 11.

[0199] Referring to example (1151), the electronic device (101) can control the display (260) to change the screen (1130) to the screen (1140) in response to an input to some area (1125).

[0200] In one embodiment, the screen (1140) may include selectable objects (1131) and may include a visual object (1141) for storing information (e.g., tags) corresponding to the selected objects and for classifying the image in association with the image (1110). The tags may be included within text information corresponding to the image (1110).

[0201] In one embodiment, in response to receiving an input for a visual object (1141), the electronic device (101) may store a tag corresponding to an activated object among selectable objects (1131) in association with the image (1110). For example, the electronic device (101) may store the tag in association with a gallery application for accessing the image (1110). In this case, images with the stored tag may be searched or classified within the gallery application. For example, the electronic device (101) may store the tag in the metadata of the image (1110).

[0202] FIGS. 12A and 12B illustrate exemplary screens of an electronic device according to an embodiment. The electronic device (101) of FIGS. 12A and 12B may be an example of the electronic device (101) of FIG. 1 . For example, the display (260) may include the display module (160) of FIG. 1 .

[0203] In FIGS. 12A and 12B , screens (1210, 1220) displayed through a display (260) of an electronic device (101) are illustrated. The screens (1210, 1220) may include, for example, an execution screen of an application for managing images and text information corresponding to the images. For example, but not limited to, the application may include a first application (901).

[0204] Referring to FIG. 12A, in one embodiment, a screen (1210) may display a list of images classified by category. For example, the screen (1210) may include images corresponding to a coupon category (e.g., image (1211)) and visual objects based on at least a portion of text information about the images (e.g., an object (1212) representing information about a coupon corresponding to the image and an object (1213) representing an expiration date of the coupon). For example, the screen (1210) may include images corresponding to a certain category (e.g., image (1221)) and visual objects based on at least a portion of text information about the images (e.g., an object (1222) representing information about a schedule corresponding to the image and an object (1223) representing a remaining period until the schedule). In one embodiment, it has been described that the categories designated for each image are determined based on the text information, but is not limited thereto. For example, a category of an image whose category has not been determined may be determined based on a user setting, or a category of an image determined based on the text information may be changed. Thereafter, the electronic device (101) may reflect the details of the category set or changed by the user when determining the category of the image using the text information. For example, the electronic device (101) may determine the category of a specific image (e.g., image (210) of FIG. 2A) as a first category (e.g., coupon) based on text information of the specific image. Thereafter, the category of the specific image may be changed by the user to a second category (e.g., schedule) different from the first category. Thereafter, the electronic device (101) may determine the category of an image that is identical or similar to the specific image as the second category.

[0205] In one embodiment, the electronic device (101) may provide a notification based on text information of an image using the application corresponding to the screen (1210). The notification function may be provided at least in part based on user behavior information stored in the electronic device (101), analyzed by the electronic device (101), or detected by the electronic device (101). The user behavior information may be obtained by analyzing information recorded in various applications of the electronic device (101), such as a wallet or calendar, and / or information detected using a sensor of the electronic device (101) (e.g., the sensor module (176) of FIG. 1), but is not limited thereto. For example, the electronic device (101) may provide a notification to the user when the expiration date corresponding to the image is approaching. For example, the image may be a coupon, and the user may mainly use the coupon on weekends. By analyzing such user behavior information, the electronic device (101) can provide a notification of the expiration date of the coupon on the weekend before the expiration date, rather than the day before the expiration date. For another example, the electronic device (101) can detect that the location of the electronic device (101) is adjacent to a location corresponding to the image (e.g., a coupon use location, a specific location), and provide a notification of this.

[0206] Although FIG. 12A illustrates an example of displaying images categorized by category, the present invention is not limited thereto. For example, the electronic device (101) can identify common text information from multiple images belonging to the same category and / or different categories. The electronic device (101) can display images having the common text information by classifying them into new categories or subcategories. For example, the electronic device (101) can classify images that expire within a specific period (e.g., day, week, month) into subcategories. For example, the electronic device (101) can classify images available at a specific location (e.g., a specific store) into subcategories. Accordingly, the user can conveniently manage and use coupons.

[0207] Referring to FIG. 12B, the screen (1220) may include a visual object (1230) for viewing classified screenshots. For example, the electronic device (101) may display the screen (1210) of FIG. 12A in response to an input to the visual object (1230). As another example, in response to the visual object (1230), an icon for displaying the screen (1210) may be created on the home screen of the electronic device (101).

[0208] FIG. 13 illustrates an example screen of an electronic device according to an embodiment. The electronic device (101) of FIG. 13 may be an example of the electronic device (101) of FIG. 1. For example, the display (260) may include the display module (160) of FIG. 1.

[0209] Referring to FIG. 13, the electronic device (101) can display a screen (1301) including an image (1310) through a display (260). The description of the image (1310) and the screen (1301) can be substantially identically applied to the description of the image (210) and the screen (201).

[0210] In one embodiment, the electronic device (101) may display a screen (1305) partially overlapping the screen (1301) via the display (260). In one embodiment, the screen (1305) may include a visual object (1306) to indicate that text is being analyzed.

[0211] Referring also to FIG. 2B, screen (1305) may be displayed before screen (220) and screen (230) are displayed. For example, referring also to FIG. 2A, in one embodiment, the electronic device (101) may display screen (1305) in response to a visual object (206) for analyzing an image (210). In one embodiment, the electronic device (101) may control the display (260) to change screen (1305) to screen (220) and display it based on obtaining the text information corresponding to the image (210).

[0212] FIGS. 14A, 14B, 14C, 14D, and 14E illustrate examples of screens displayed by an electronic device according to an embodiment. In FIGS. 14A, 14B, 14C, 14D, and 14E, screens (1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408, and 1409) displayed through a display (260) of an electronic device (101) are illustrated.

[0213] Referring to FIG. 14A, the electronic device (101) may display a first screen (1401) through the display (260). The first screen (1401) may include an execution screen of an application, for example, a gallery application, for playing, managing, and editing images and videos stored in the electronic device (101) (e.g., the memory). The first screen (1401) may include an image (1410). The image (1410) may be an image previously stored in the memory of the electronic device (101). For example, an image (1410) selected by a user from among a plurality of images stored in the electronic device (101) may be displayed through the first screen (1401). In one embodiment, the image (1410) may include a coupon image. The image (1410) may include a plurality of image objects. For example, image (1410) may include a plurality of image objects (1412, 1414, 1416, 1417, 1418). Objects (1412, 1414, 1417, 1418) may include objects corresponding to text, and object (1416) may include an object corresponding to non-text. Object (1412) may indicate the issuer of the coupon or the use of the coupon, such as 'E STORE'. Object (1412) may also indicate the value of goods provided by the coupon, such as '1000'. Object (1414) may indicate the nature of the coupon (e.g., a gift), such as 'a gift from my heart'. Object (1414) may indicate the website address of the issuer or use of the coupon, such as 'www.giftURL.com'. Object (1416) may represent a barcode image for the coupon, and object (1417) may represent a barcode number for the coupon. Object (1418) may include, for example, information about the name, use, and expiration date of the product provided by the coupon.However, the information contained in the image (1410) is not limited to the examples described above.

[0214] In one embodiment, the electronic device (101) may display a second screen (1402) through the display (260) after the first screen (1401) is displayed. The second screen (1402) may include an image (1410) and a visual object (1422).

[0215] In one embodiment, the visual object (1422) may be an object for providing a character recognition result for the image (1410). For example, the electronic device (101) may receive a user input for the visual object (1422). The electronic device (101) may display the character recognition result for the image (1410) in response to the user input for the visual object (1422). For example, the electronic device (101) may display the fifth screen (1405) of FIG. 14C, which shows the character recognition result for the image (1410). Character recognition of image objects included in the image (1410) may be performed, for example, in response to a user input for the visual object (1422). For another example, character recognition of image objects included in an image (1410) may be performed in response to displaying a first screen (1401) including the image (1410), regardless of an input for a visual object (1422). In one embodiment, the electronic device (101) may perform character recognition on the image (1410) to extract text included in the image (1410).

[0216] Referring to FIG. 14B, according to one embodiment, the electronic device (101) may display a third screen (1403) through the display (260). The third screen (1403) may include a captured image of the first screen (1401) or the second screen (1402). For example, the third screen (1403) may include an image (1410-1) that is substantially identical to the image (1410) displayed through the first screen (1401) or the second screen (1402). For example, the image (1410-1) may include visual objects (1412-1, 1414-1, 1416-1, 1417-1, 1418-1) that are substantially identical to the visual objects (1412, 1414, 1416, 1417, 1418) of the image (1410) of the first screen (1401) or the second screen (1402).

[0217] In one embodiment, when the third screen (1403) is a captured image of the second screen (1402), the third screen (1403) may include an image object (1432) corresponding to the visual object (1422) of the second screen (1402). When the third screen (1403) is a captured image of the first screen (1401), unlike the city, the third screen (1403) may not include an image object (1432) corresponding to the visual object (1422) of the second screen (1402). In one embodiment, the third screen (1403) may include a menu (1434) for editing, sharing, or creating a tag for the captured image (1410-1).

[0218] In one embodiment, the electronic device (101) may display a fourth screen (1404) through the display (260) after displaying the third screen (1403). The fourth screen (1404) may include an image (1410-1), a visual object (1442), and a visual object (1444). In one embodiment, within the fourth screen (1404), the visual object (1442) and the visual object (1444) are illustrated as being displayed to overlap the image (1410-1), but are not limited thereto. For example, the visual object (1442) and / or the visual object (1444) may be displayed in an area within the fourth screen (1404) that is separate from the image (1410-1).

[0219] In one embodiment, the visual object (1444) (e.g., the visual object (206) of FIG. 2A) may be an object for providing a character recognition result for the image (1410-1). For example, the electronic device (101) may receive a user input for the visual object (1444). The electronic device (101) may display the character recognition result for the image (1410-1) in response to the user input for the visual object (1444). For example, the electronic device (101) may display the fifth screen (1405) of FIG. 14C, which represents the character recognition result for the image (1410-1). Character recognition of image objects included in the image (1410-1) may be performed, for example, in response to a user input for the visual object (1444). For another example, character recognition of image objects included in the image (1410-1) may be performed in response to displaying a third screen (1403) or a fourth screen (1404) including the image (1410-1), regardless of an input for the visual object (1444). In one embodiment, the electronic device (101) may perform character recognition on the image (1410-1) to extract text included in the image (1410-1).

[0220] In one embodiment, the electronic device (101) may obtain text information including a category, content, and a summary of the content of the image (1410-1) by using text extracted from the image (1410-1). For example, the electronic device (101) may obtain the text information based on receiving a user input for a visual object (1422) of the second screen (1402) or obtaining the image (1410-1) (e.g., obtaining a captured image of the first screen (1401) or the second screen (1402).

[0221] In one embodiment, the visual object (1442) (e.g., the visual object (223) of FIG. 2B) may be an object for executing an application corresponding to the category of the image (1410-1). For example, the electronic device (101) may determine an application corresponding to the category of the image (1410-1) using the text information of the image (1410-1) and display the visual object (1442) for executing the determined application. In one embodiment, the electronic device (101) may execute the application corresponding to the category of the image (1410-1) in response to receiving a user input for the visual object (1442). For example, in response to receiving a user input for a visual object (1442), the electronic device (101) may display an execution screen of an application corresponding to the category of the image (1410-1) (e.g., the first execution screen (1406) of FIG. 14D). The application may include, for example, a payment application or a wallet application that can store, manage, and use coupons corresponding to the image (1410-1).

[0222] Referring to FIG. 14c, in one embodiment, the electronic device (101) may display a fifth screen (1405) through the display (260). For example, the electronic device (101) may display the fifth screen (1405) in response to a user input for a visual object (1422) of the second screen (1402) or a visual object (1444) of the fourth screen (1404).

[0223] In one embodiment, the fifth screen (1405) may include an image (1410-2) in which text included in image objects included in the image (1410) or the image (1410-1) is emphasized. For example, the electronic device (101) may display the image (1410-2) using text extracted from the image (1410) or the image (1410-1). For example, the image (1410-2) may display text (1452) included in the image object (1412) or the image object (1412-1), text (1454) included in the image object (1414) or the image object (1414-1), text (1457) included in the image object (1417) or the image object (1417-1), and text (1458) included in the image object (1418) or the image object (1418-1) with emphasis. In the illustrated example, text within the image (1410-2) is shown as being brighter than its surroundings, but the illustrated example is not limited thereto. For example, visual objects corresponding to non-text may be displayed as blurred.

[0224] In one embodiment, the fifth screen (1405) may include visual objects (1451, 1453, 1455, 1456). For example, the visual objects (1451, 1453, 1455) may be displayed within an area within the fifth screen (1405) that is separate from the image (1410-2). For example, the image (1410-2) being displayed may be displayed in a reduced size at the top of the fifth screen (1405), and the visual objects (1451, 1453, 1455) may be displayed at the bottom of the fifth screen (1405), but are not limited thereto.

[0225] In one embodiment, a visual object (1451) (e.g., visual object (1442)) may be configured to execute the payment application corresponding to a category of image (1410-2) (e.g., image (1410) or image (1410-1)). In one embodiment, a visual object (1456) may indicate that text within image (1410-2) may be selected.

[0226] In one embodiment, the visual object (1453) may be for executing an application different from the application executed by the visual object (1451), for example, a note application. In one embodiment, the electronic device (101) may execute the note application based on texts selected within the image (1410-2) based on a user input to the visual object (1453). For example, although not shown, the electronic device (101) may display a note (or page) in which texts selected within the image (1410-2) are entered as an execution screen of the note application based on a user input to the visual object (1453).

[0227] In one embodiment, the visual object (1455) may be for copying all text within the image (1410-2). In one embodiment, the electronic device (101) may, in response to a user input for the visual object (1455), copy all text within the image (1410-2) (or store it within an area of ​​the memory of the electronic device (101) where it is temporarily stored, such as a clipboard) and display the visual object (1459). The visual object (1459) may indicate an action that can be performed by the input for the visual object (1455), such as, "Copied. You can paste it here or on a nearby mobile phone." The visual object (1459) may be displayed for a specified period of time (e.g., several seconds).

[0228] Referring to FIGS. 14A, 14B, and 14D, an electronic device (101) according to an embodiment may display a first execution screen (1406) of the payment application corresponding to the category of the image (1410) or the image (1410-1). For example, the first execution screen (1406) may include a loading screen of the payment application. For example, the first execution screen (1406) may include an icon (1461) of the payment application.

[0229] In one embodiment, the electronic device (101) may execute the payment application based on text identified in the image (1410-1) of the fourth screen (1404) based on a user input for a visual object (1442) of the fourth screen (1404). For example, the electronic device (101) may display a first execution screen (1406) of the payment application in response to a user input for the visual object (1442) of the fourth screen (1404). After displaying the first execution screen (1406), the electronic device (101) may display an image (1480) including at least a portion of the image (1410-1) and a third execution screen (1408) in which texts identified in the image (1410-1) are input, as an execution screen of the payment application for storing a coupon corresponding to the image (1410-1). For example, a product name based on text identified in an image object (1418) may be entered in an input field (1486) of a third execution screen (1408). For example, a 'barcode' coupon type of an image (1410-1) may be entered in an input field (1487). For example, a barcode number based on text identified in an image object (1417) may be entered in an input field (1488).

[0230] In one embodiment, the electronic device (101) may execute the payment application based on a selected text among the texts identified within an image (1410-2) of the fifth screen (1405) (e.g., images (1410 or 1410-1)) based on a user input to a visual object (1451) of the fifth screen (1405). For example, the electronic device (101) may display a first execution screen (1406) of the payment application in response to a user input to a visual object (1451) of the fifth screen (1405). After displaying the first execution screen (1406), the electronic device (101) may display a third execution screen (1408) in which an image (1480) including at least a portion of the image (1410-1) and texts selected within the image (1410-2) are entered, as an execution screen of the application for storing a coupon corresponding to the image (1410-2). For example, a product name of the text (1458) may be entered within an input field (1486) of the third execution screen (1408). For example, a 'barcode', which is a coupon type of the image (1410-1), may be entered within an input field (1487). For example, a barcode number, which is the same as the text (1457), may be entered within the input field (1488).

[0231] Additionally or optionally, the electronic device (101) may display a second execution screen (1407) after displaying the first execution screen (1406) and before displaying the third execution screen (1408). The second execution screen (1407) may include a visual object (1473) (e.g., a crop tool) for specifying an area of ​​the image (1410-1) to be attached within the third execution screen (1408) and a visual object (1471) for completing the area specification for the image (1410-1). For example, a user may specify a specific area of ​​the image (1410-1) within the visual object (1473) by changing the size of the border of the visual object (1473). The visual object (1473) is illustrated as having a rectangular shape, but is not limited thereto. In one embodiment, the electronic device (101) may, in response to a user input for a visual object (1471), display a third execution screen (1408) with an image (1480) attached thereto corresponding to a specific area of ​​the image (1410-1) designated by the visual object (1473).

[0232] In one embodiment, the third execution screen (1408) may include a visual object (1481) for storing the entered coupon information and a visual object (1483) for canceling the storage of the coupon information.

[0233] In one embodiment, the electronic device (101) may store coupon information entered through the third execution screen (1408) based on an input to the visual object (1481), and display a fourth execution screen (1409) indicating the stored coupon information. The fourth execution screen (1409) may include coupon information such as, but not limited to, the name of the coupon, the name of the product provided by the coupon, the brand of the coupon provider, the coupon code (e.g., the coupon identification number or barcode number), the content provided by the coupon, the start date and time, the expiration date and time, the method of use, and restrictions. The fourth execution screen (1409) may display status information of the stored coupon, such as whether it has expired.

[0234] FIG. 15 is an exemplary block diagram illustrating a generative artificial intelligence system according to an embodiment. In FIG. 15, an exemplary artificial intelligence system is illustrated, including a user question response interface (1510), an AI framework (1520), a generative AI model (1530), a knowledge repository (1540), and an application / service component (1550). According to an embodiment, a processor (e.g., the processor 120 of FIG. 1) of an electronic device (e.g., the electronic device 101 of FIG. 1) may execute one or more programs defined by the blocks of FIG. 15 to execute functions related to generative artificial intelligence.

[0235] In one embodiment, a User Query / Response Interface (1510) may receive user input. The user input may be in the form of natural language, images, and / or videos. Context information may also be transmitted when the user input is transmitted. Context information may include various additional information at the time of user input. For example, information about the application currently being used by the user or information about the user's location. Furthermore, the user input may be in a mixed form of the aforementioned natural language, images, sounds, and context information. Furthermore, the user input may be in a non-natural language form, such as selecting a menu. The User Query / Response Interface (1510) may output the results of a generative artificial intelligence system to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The User Query / Response Interface (1510) may output the results of a generative artificial intelligence system to the user. The output can be in natural language form, in the form of specific content, or in the form of actions requested by the user.

[0236] The AI ​​framework (1520) can receive user input and coordinate and control each component necessary to perform the user's intent based on the user's query. The AI ​​framework (1520) can include a prompt design component (1521), an API / plug-in management component (1522), and a refinery component (1523).

[0237] User input received from the user question response interface (1510) can be transmitted to a prompt design component (1521). The prompt design component (1521) can be used to generate prompts suitable for inputting the user input into an LLM or LMM (Large Multimodal Model). The prompt design component (1521) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (1521) can access knowledge repositories (1540) containing user preference data, a prompt library, and prompt examples based on the user input to generate prompts and transmit the generated prompts to the LLM or LMM.

[0238] The API / plugin management component (1522) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / plugin management component (1522) can establish a channel for communicating with the outside of the AI ​​framework (1520) through an application programming interface (API) and can enable access to various data sources through the established channel. In addition, the API / plugin management component (1522) can request an action through the API that ultimately performs the user input, rather than an intermediate result, if the action needs to be performed by the application / service component (1550) (e.g., the program (140) or application (146) of FIG. 1). Information obtained from the outside can be used to generate a prompt in the prompt design component (1521) together with the user input or can be passed as input to the generative model.

[0239] The refinement component (1523) (or output modification component) can fine-tune the output from the generative AI model (1530). For example, the refinement component (1523) can verify that the content generated through the LLM and / or LMM is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the refinement component (1523) can determine to what extent the content matches the user's desired result and, if necessary, can perform additional processing. The refinement component (1523) can additionally configure and provide hints to the user to avoid undesired output.

[0240] A generative AI model (1530) can generally refer to an artificial intelligence neural network that creates new types of data based on user input information. A generative AI model (1530) can include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational autoencoder (VAE), and examples include a diffusion-based generative model that uses a VAE and a transformer structure. A language-generating model is a model trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, there are also large multimodal models (LMMs) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0241] An electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may include a display (e.g., the display module (160) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1) for storing instructions, and a processor (e.g., the processor (120) of FIG. 1). The instructions, when executed by the processor, may cause the electronic device to display, through the display, a still image including one or more image objects corresponding to text. The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the one or more image objects of the still image based on an input for analyzing the still image being displayed. The instructions, when executed by the processor, may cause the electronic device to select, from among a plurality of applications, a designated application corresponding to the still image, using at least a portion of the text information. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a screen including a visual object and the still image for executing the designated application. The instructions, when executed by the processor, may cause the electronic device to display, through the display, an execution screen of the designated application in response to an input related to the visual object.

[0242] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to, in response to an input for the visual object, execute a function of the specified application with respect to the still image.

[0243] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to store at least some text of the text information or other text related to the at least some text in association with the designated application.

[0244] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to provide a notification through the designated application based on the at least some text.

[0245] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to classify the still image into one or more of a plurality of categories based at least in part on the text information.

[0246] In one embodiment, the electronic device may include a communication circuit. The instructions, when executed by the processor, may cause the electronic device to obtain, through the communication circuit, additional information retrieved from an external server based at least in part on the text information. The instructions, when executed by the processor, may cause the electronic device to store the additional information in association with at least a portion of the text of the text information.

[0247] In one embodiment, the still image may further include another image object corresponding to non-text. The instructions, when executed by the processor, may cause the electronic device to store an image corresponding to the other image object in association with the specified application.

[0248] In one embodiment, the other image object may include an image code.

[0249] In one embodiment, the text information may include location information. The specified application may include a map application. The instructions, when executed by the processor, may cause the electronic device to execute the map application based at least in part on the location information.

[0250] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to select a first application from among the plurality of applications as the designated application, at least in part based on the text information including a first designated text. The instructions, when executed by the processor, may cause the electronic device to select a second application from among the plurality of applications, different from the first application, as the designated application, at least in part based on the text information including a second designated text different from the first designated text.

[0251] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display, through the display, the screen further including another visual object indicating that at least a portion of the still image corresponds to a coupon image or a ticket image.

[0252] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display the screen further including another visual object indicating that at least a portion of the still image corresponds to a map image.

[0253] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to obtain the text information from the still image based on a user input.

[0254] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to acquire the still image via a capture application.

[0255] In one embodiment, the electronic device may include a communication circuit. The instructions, when executed by the processor, may cause the electronic device to transmit the still image to an external server via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to receive, after transmitting the still image, at least a portion of the text information from the external server via the communication circuit.

[0256] In one embodiment, the memory may store an image training model for a plurality of images. The instructions, when executed by the processor, may cause the electronic device to generate at least a portion of the text information by applying the still image to the image training model.

[0257] In one embodiment, the still image may include a coupon image. The text information may include coupon information contained within the coupon image. The designated application may include an application for managing coupons. The instructions, when executed by the processor, may cause the electronic device to execute a function of the designated application based at least in part on the coupon information in response to the input related to the visual object.

[0258] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify whether the specified application is installed in the electronic device. The instructions, when executed by the processor, may cause the electronic device to display, together with the still image, the visual object for executing the specified application if the specified application is installed in the electronic device. The instructions, when executed by the processor, may cause the electronic device to display, together with the still image, another visual object for executing another application for installing the specified application if the specified application is not installed in the electronic device.

[0259] According to one embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may include a display (e.g., the display module (160) of FIG. 1), a memory (e.g., the memory (130) of FIG. 1) for storing instructions, and a processor (e.g., the processor (120) of FIG. 1). The instructions, when executed by the processor, may cause the electronic device to display a screenshot image through the display. The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the screenshot image based on an input for analyzing the screenshot image being displayed. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a window including a visual object for executing an application based on identifying an application corresponding to the screenshot image using the text information, so as to partially overlap the screenshot image. The instructions, when executed by the processor, may cause the electronic device to display an execution screen of the application through the display in response to an input related to the visual object, based on identifying an application corresponding to the screenshot image using the text information.

[0260] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to extract text corresponding to an image object included in the screenshot image. The instructions, when executed by the processor, may cause the electronic device to obtain text information from an artificial intelligence model into which the extracted text is input.

[0261] In one embodiment, the screenshot image may include another image object corresponding to the non-text. The instructions, when executed by the processor, may cause the electronic device to obtain the text information from the artificial intelligence model into which the other image object and the extracted text are input.

[0262] In one embodiment, the electronic device may include a communication circuit (e.g., the communication module (190) of FIG. 1). The instructions, when executed by the processor, may cause the electronic device to extract text corresponding to an image object included in the screenshot image. The instructions, when executed by the processor, may cause the electronic device to transmit the extracted text to a server (e.g., the server (108) of FIG. 1) via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to receive the text information from the server via the communication circuit after transmitting the extracted text.

[0263] In one embodiment, the electronic device may include the communication circuit. The instructions, when executed by the processor, may cause the electronic device to transmit the screenshot image to the server via the communication circuit in response to the input for analyzing the screenshot image. The instructions, when executed by the processor, may cause the electronic device to receive the text information from the server via the communication circuit after transmitting the screenshot image.

[0264] In one embodiment, the text information may include category information of the screenshot image, content information, and summary information about the content information.

[0265] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display, through the display, the window further including an object representing the category information.

[0266] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display, through the display, the window further including an object representing the summary information.

[0267] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display, through the display, another window including the category information and having a first size, partially overlapping the screenshot image. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a window having a second size larger than the first size, partially overlapping the screenshot image, in response to an input for the other window.

[0268] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to receive an input for capturing a screen being displayed through the display. The instructions, when executed by the processor, may cause the electronic device to obtain the screenshot image in response to the input for capturing the screen.

[0269] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to store, in association with the application, information for classifying the screenshot image using the text information.

[0270] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to store information for classifying the screenshot image using the text information within metadata of the screenshot image.

[0271] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to execute the application based on the text information.

[0272] In one embodiment, the screenshot image may include a coupon image. The text information may include coupon information contained within the coupon image. The application may include an application for managing coupons. The instructions, when executed by the processor, may cause the electronic device to execute the application based at least in part on the coupon information in response to the input related to the visual object.

[0273] In one embodiment, the screenshot image may include an image representing a specific region. The text information may include location information corresponding to the image representing the specific region. The application may include a map application. The instructions, when executed by the processor, may cause the electronic device to execute the map application based at least in part on the location information in response to the input related to the visual object.

[0274] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to identify whether the application is installed in the electronic device. The instructions, when executed by the processor, may cause the electronic device to display the window including the visual object for executing the application if the application is installed in the electronic device. The instructions, when executed by the processor, may cause the electronic device to display, in a partial overlap with the screenshot image, a window including another visual object for executing another application for installing the application if the application is not installed in the electronic device.

[0275] In one embodiment, the instructions, when executed by the processor, may cause the electronic device to display, through the display, another screenshot image different from the screenshot image. The instructions, when executed by the processor, may cause the electronic device to obtain, based on an input for analyzing the other screenshot image, other text information corresponding to the other screenshot image. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a window that includes another visual object for executing another application different from the application and corresponding to the other screenshot image identified using the other text information, partially overlapping the other screenshot image. The instructions, when executed by the processor, may cause the electronic device to display, through the display, an execution screen of the other application in response to an input related to the other visual object.

[0276] In one embodiment, the screenshot image may include a coupon image. The other screenshot image may include an image representing a specific region.

[0277] According to one embodiment, an electronic device (e.g., electronic device (101) of FIG. 1) may include a display (e.g., display module (160) of FIG. 1), a memory (e.g., memory (130) of FIG. 1) for storing instructions, and a processor (e.g., processor (120) of FIG. 1). The instructions, when executed by the processor, may cause the electronic device to display a first screen including an image through the display. The instructions, when executed by the processor, may cause the electronic device to extract text from the image by performing optical character recognition (OCR). The instructions, when executed by the processor, may cause the electronic device to obtain text information corresponding to the image from an artificial intelligence model into which the extracted text is input. The instructions, when executed by the processor, may cause the electronic device to identify a category of the image using the text information. The instructions, when executed by the processor, may cause the electronic device to display a second screen including the image and a visual object for executing the application based on identifying an application corresponding to the identified category. The instructions, when executed by the processor, may cause the electronic device to display a third screen different from the second screen by executing the application in response to an input related to the visual object.

[0278] In one embodiment, the electronic device may include a communication circuit (e.g., a communication module (190) of FIG. 1). The instructions, when executed by the processor, may cause the electronic device to transmit the extracted text to a server (e.g., a server (108) of FIG. 1) including the artificial intelligence model via the communication circuit. The instructions, when executed by the processor, may cause the electronic device to receive the text information from the server via the communication circuit.

[0279] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments disclosed in this document are not limited to the aforementioned devices.

[0280] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0281] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0282] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more commands stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one command among the one or more commands stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one command called. The one or more commands may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0283] According to one embodiment, the method according to the various embodiments disclosed in the present document may be provided as included in a computer program product. The computer program product may be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0284] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; Memory that stores instructions; Contains a processor, The above instructions, when executed by the processor, cause the electronic device to: Through the above display, a still image including one or more image objects corresponding to the text is displayed, Based on the input for analyzing the still image being displayed, text information corresponding to one or more image objects of the still image is obtained, Using at least a portion of the above text information, selecting a designated application corresponding to the still image from among a plurality of applications; Through the above display, a screen including a visual object and the still image for executing the above designated application is displayed, and In response to an input related to said visual object, causing the execution screen of said specified application to be displayed through said display. Electronic devices.

2. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: In response to an input to said visual object, causing a function of said designated application to be executed with respect to said still image, Electronic devices.

3. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: Causing to store at least a portion of the text of said text information or other text related to said at least a portion of the text in association with said specified application; Electronic devices.

4. In claim 3, The above instructions, when executed by the processor, cause the electronic device to: Causing a notification to be provided through said specified application based on at least some of said text; Electronic devices.

5. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: causing said still image to be classified into one or more of a plurality of categories based at least in part on said text information; Electronic devices.

6. In claim 3, Containing a communication circuit, The above instructions, when executed by the processor, cause the electronic device to: Obtaining additional information retrieved through an external server based at least in part on the above text information through the communication circuit; Causing said additional information to be stored in association with at least some text of said text information, Electronic devices.

7. In claim 1, The above still image further includes another image object corresponding to the non-text, The above instructions, when executed by the processor, cause the electronic device to: Causing an image corresponding to said other image object to be stored in association with said specified application; Electronic devices.

8. In claim 7, The above other image object contains the image code, Electronic devices.

9. In claim 1, The above text information includes location information, The above specified application includes a map application, The above instructions, when executed by the processor, cause the electronic device to: Causing said map application to be executed based at least in part on said location information; Electronic devices.

10. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: selecting a first application among the plurality of applications as the designated application based at least in part on the fact that the text information includes a first designated text; and Causing a second application, different from the first application among the plurality of applications, to be selected as the designated application, based at least in part on the fact that the text information includes a second designated text different from the first designated text. Electronic devices.

11. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: Causing the screen to be displayed, through the display, further including another visual object indicating that at least a portion of the still image corresponds to a coupon image or a ticket image; Electronic devices.

12. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: causing said screen to be displayed further including another visual object indicating that at least a portion of said still image corresponds to a map image; Electronic devices.

13. In claim 11, The above instructions, when executed by the processor, cause the electronic device to: causing said text information to be obtained from said still image based on user input; Electronic devices.

14. In claim 1, The above instructions, when executed by the processor, cause the electronic device to: causing said still image to be acquired through a capture application; Electronic devices.

15. In claim 1, Containing a communication circuit, The above instructions, when executed by the processor, cause the electronic device to: Transmitting the above still image to an external server through the above communication circuit; and After transmitting the still image, causing at least a part of the text information to be received from the external server through the communication circuit. Electronic devices.

Images