Electronic device for providing image-based generated information

Abstract

An electronic device according to an embodiment may include a display, a memory for storing instructions, and a processor. When executed by the processor, the instructions may cause the electronic device to: display a still image including one or more image objects corresponding to text via the display; obtain text information corresponding to one or more image objects in the still image based on input for analyzing the still image being displayed; select a specified application corresponding to the still image from a plurality of applications using at least a portion of the text information; display a screen via the display including the still image and a visual object for launching the specified application; and display a launch screen for the specified application via the display in response to input associated with the visual object.

Description

Technical Field

[0001] The following description relates to an electronic device for providing image-based information. Background Technology

[0002] Electronic devices, such as smartphones, can store image files. For example, they can store images captured by a camera, screenshots of the screen displayed on a monitor, and images downloaded from external sources. These devices can then provide users with applications to view or manage these images.

[0003] The information described above may be provided as relevant technology for the purpose of aiding understanding of this disclosure. No claim or determination is made as to whether any of the above descriptions can be used as prior art in relation to this disclosure. Summary of the Invention

[0004] Technical solution

[0005] An electronic device according to an embodiment may include a display, a memory storing instructions, and a processor. When executed by the processor, the instructions may cause the electronic device to display a still image, including one or more image objects corresponding to text, on the display. When executed by the processor, the instructions may cause the electronic device to obtain text information corresponding to one or more image objects of the still image, based on input for analyzing the still image being displayed. When executed by the processor, the instructions may cause the electronic device to select a specific application corresponding to the still image from a plurality of applications using at least a portion of the text information. When executed by the processor, the instructions may cause the electronic device to display a screen, including a visual object for executing the specified application and the still image, on the display. When executed by the processor, the instructions may cause the electronic device to display an execution screen for the specified application on the display in response to input associated with the visual object.

[0006] An electronic device according to an embodiment may include a display, a memory storing instructions, and a processor. When executed by the processor, the instructions may cause the electronic device to display a screenshot image on the display. When executed by the processor, the instructions may cause the electronic device to: obtain text information corresponding to the screenshot image based on input for analyzing the screenshot image being displayed; and, when executed by the processor, cause the electronic device to: identify an application corresponding to the screenshot image using the text information, and display a window including a visual object for executing the application on the display, partially overlapping the screenshot image. When executed by the processor, the instructions may cause the electronic device to: identify an application corresponding to the screenshot image using the text information, and, in response to input related to the visual object, display an application execution screen on the display.

[0007] An electronic device according to an embodiment may include a display, a memory storing instructions, and a processor. When executed by the processor, the instructions may cause the electronic device to display a first screen including images on the display. When executed by the processor, the instructions may cause the electronic device to extract text from the images by performing optical character recognition (OCR). When executed by the processor, the instructions may cause the electronic device to obtain text information corresponding to the images from an artificial intelligence model input to the extracted text. When executed by the processor, the instructions may cause the electronic device to use the text information to identify the category of the images. When executed by the processor, the instructions may cause the electronic device to display a second screen including images and visual objects for performing the applications, based on the identification of an application corresponding to the identified category. When executed by the processor, the instructions may cause the electronic device to display a third screen different from the second screen by performing the applications in response to input associated with the visual objects.

[0008] A non-transitory computer-readable storage medium according to an embodiment may store one or more programs. The one or more programs may include instructions, when executed by an electronic device including a display, to cause the electronic device to display a still image comprising one or more image objects corresponding to text. The one or more programs may include instructions, when executed by the electronic device, to cause the electronic device to: obtain text information corresponding to one or more image objects of the still image based on input for analyzing the still image being displayed. The one or more programs may include instructions, when executed by the electronic device, to cause the electronic device to select a specified application corresponding to the still image from a plurality of applications using at least a portion of the text information. The one or more programs may include instructions, when executed by the electronic device, to cause the electronic device to display a screen comprising a visual object for performing the specified application and the still image. The one or more programs may include instructions, when executed by the electronic device, to cause the electronic device to display an execution screen for the specified application via the display in response to input associated with the visual object.

[0009] The non-transitory computer-readable storage medium according to the embodiments may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display, cause the electronic device to display a screenshot image on the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to: obtain text information corresponding to the screenshot image based on input for analyzing the screenshot image being displayed. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to:, based on identifying an application corresponding to the screenshot image using the text information, display a window including a visual object for executing the application on the display, partially overlapping the screenshot image. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to:, based on identifying an application corresponding to the screenshot image using the text information, display an execution screen of the application on the display in response to input related to the visual object.

[0010] A non-transitory computer-readable storage medium according to an embodiment may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device including a display, cause the electronic device to display a first screen including an image via the display. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to extract text from the image by performing optical character recognition (OCR). The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to obtain text information corresponding to the image from an artificial intelligence model input to the extracted text. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to use the text information to identify a category of the image. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a second screen including an image and a visual object for performing an application based on the identified application. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display a third screen different from the second screen by performing an application in response to input associated with the visual object.

[0011] According to an embodiment, a method for an electronic device including a display may include: displaying a still image comprising one or more image objects corresponding to text on the display. The method may include: obtaining text information corresponding to one or more image objects of the still image based on input for analyzing the still image being displayed. The method may include selecting a specified application corresponding to the still image from a plurality of applications using at least a portion of the text information. The method may include displaying a screen comprising a visual object for performing the specified application and the still image on the display. The method may include: displaying an execution screen for the specified application on the display in response to input associated with the visual object.

[0012] According to an embodiment, a method for an electronic device including a display may include displaying a screenshot image on the display. The method may include: obtaining text information corresponding to the screenshot image based on input for analyzing the screenshot image being displayed. The method may include: identifying an application corresponding to the screenshot image using the text information, and displaying a window including a visual object for performing the application on the display, partially overlapping the screenshot image. The method may also include: identifying the application corresponding to the screenshot image using the text information, and displaying an execution screen of the application on the display in response to input associated with the visual object.

[0013] According to an embodiment, a method for an electronic device including a display may include displaying a first screen including an image on the display. The method may include extracting text from the image by performing optical character recognition (OCR). The method may include obtaining text information corresponding to the image from an artificial intelligence model input to the extracted text. The method may include identifying a category of the image using the text information. The method may include displaying a second screen including the image and a visual object for performing the application, based on the identified application corresponding to the identified category. The method may include displaying a third screen different from the second screen by performing the application in response to input related to the visual object. Attached Figure Description

[0014] Figure 1 This is a block diagram of an electronic device in a network environment according to various embodiments.

[0015] Figure 2a An example of a screen displayed by an electronic device according to an embodiment is shown.

[0016] Figure 2b An example of a screen displayed by an electronic device according to an embodiment is shown.

[0017] Figure 2c An example of a screen displayed by an electronic device according to an embodiment is shown.

[0018] Figure 3a An example of a screen displayed by an electronic device according to an embodiment is shown.

[0019] Figure 3b An example of a screen displayed by an electronic device according to an embodiment is shown.

[0020] Figure 3c An example of a screen displayed by an electronic device according to an embodiment is shown.

[0021] Figure 3d An example of a screen displayed by an electronic device according to an embodiment is shown.

[0022] Figure 4a An example of a screen displayed by an electronic device according to an embodiment is shown.

[0023] Figure 4b An example of a screen displayed by an electronic device according to an embodiment is shown.

[0024] Figure 4c An example of a screen displayed by an electronic device according to an embodiment is shown.

[0025] Figure 4d An example of a screen displayed by an electronic device according to an embodiment is shown.

[0026] Figure 5a An example of a screen displayed by an electronic device according to an embodiment is shown.

[0027] Figure 5b An example of a screen displayed by an electronic device according to an embodiment is shown.

[0028] Figure 5c An example of a screen displayed by an electronic device according to an embodiment is shown.

[0029] Figure 5d An example of a screen displayed by an electronic device according to an embodiment is shown.

[0030] Figure 6 An example of a screen displayed by an electronic device according to an embodiment is shown.

[0031] Figure 7 This is a diagram illustrating an example of a screen displayed by an electronic device according to an embodiment.

[0032] Figure 8 This is a diagram illustrating an example of a screen displayed by an electronic device according to an embodiment.

[0033] Figure 9 An example of a signal flow for a method according to an embodiment for providing functionality associated with image-generated text information is shown.

[0034] Figure 10 This is a flowchart illustrating a method for providing functionality associated with image-generated text information according to an embodiment.

[0035] Figure 11 An example of an exemplary screen of an electronic device according to an embodiment is shown.

[0036] Figure 12a An example of an exemplary screen of an electronic device according to an embodiment is shown.

[0037] Figure 12b An example of an exemplary screen of an electronic device according to an embodiment is shown.

[0038] Figure 13 An example of an exemplary screen of an electronic device according to an embodiment is shown.

[0039] Figure 14a An example of a screen displayed by an electronic device according to an embodiment is shown.

[0040] Figure 14b An example of a screen displayed by an electronic device according to an embodiment is shown.

[0041] Figure 14c An example of a screen displayed by an electronic device according to an embodiment is shown.

[0042] Figure 14d An example of a screen displayed by an electronic device according to an embodiment is shown.

[0043] Figure 14e An example of a screen displayed by an electronic device according to an embodiment is shown.

[0044] Figure 15 This is an exemplary block diagram illustrating a generative artificial intelligence system according to an embodiment. Detailed Implementation

[0045] Figure 1 This is a block diagram illustrating an electronic device 101 in a network environment 100 according to various embodiments. Reference Figure 1In network environment 100, electronic device 101 can communicate with electronic device 102 via a first network 198 (e.g., a short-range wireless communication network), or with at least one of electronic device 104 or server 108 via a second network 199 (e.g., a long-range wireless communication network). According to an embodiment, electronic device 101 can communicate with electronic device 104 via server 108. According to an embodiment, electronic device 101 may include a processor 120, memory 130, input module 150, sound output module 155, display module 160, audio module 170, sensor module 176, interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, user identification module (SIM) 196, or antenna module 197. In some embodiments, at least one of the components (e.g., connection terminal 178) may be omitted from electronic device 101, or one or more other components may be added to electronic device 101. In some embodiments, some of the components (e.g., sensor module 176, camera module 180, or antenna module 197) may be implemented as a single component (e.g., display module 160).

[0046] Processor 120 can execute, for example, software (e.g., program 140) to control at least one other component (e.g., hardware or software component) of electronic device 101 coupled to processor 120, and can perform various data processing or calculations. According to embodiments, as at least part of data processing or calculation, processor 120 can store commands or data received from another component (e.g., sensor module 176 or communication module 190) in volatile memory 132, process the commands or data stored in volatile memory 132, and store the resulting data in non-volatile memory 134. According to embodiments, processor 120 may include a main processor 121 (e.g., central processing unit (CPU) or application processor (AP)) or an auxiliary processor 123 (e.g., graphics processing unit (GPU), neural processing unit (NPU), image signal processor (ISP), sensor central processor, or communication processor (CP)) that is operationally independent of or combined with the main processor 121. For example, when electronic device 101 includes a main processor 121 and an auxiliary processor 123, the auxiliary processor 123 may be adapted to consume less power than the main processor 121, or adapted to perform a specific function. The auxiliary processor 123 may be implemented separately from the main processor 121, or may be implemented as part of the main processor 121.

[0047] When the main processor 121 is inactive (e.g., in sleep mode), the auxiliary processor 123 may control at least some of the functions or states associated with at least one component of the electronic device 101 (other than the main processor 121), such as display module 160, sensor module 176, or communication module 190. Alternatively, when the main processor 121 is active (e.g., running an application), the auxiliary processor 123 may work with the main processor 121 to control at least some of the functions or states associated with at least one component of the electronic device 101, such as display module 160, sensor module 176, or communication module 190. According to embodiments, the auxiliary processor 123 (e.g., an image signal processor or a communication processor) may be implemented as part of another component (e.g., camera module 180 or communication module 190) functionally associated with the auxiliary processor 123. According to embodiments, the auxiliary processor 123 (e.g., a neural processing unit) may include hardware architectures specified for processing artificial intelligence models. Artificial intelligence models can be generated through machine learning. This learning can be performed, for example, by an electronic device 101 performing artificial intelligence or via a separate server (e.g., server 108). The learning algorithm can include, but is not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning. The artificial intelligence model can include multiple layers of artificial neural networks. The artificial neural network can be a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of these, but is not limited thereto. Additionally or alternatively, the artificial intelligence model can include software structures in addition to hardware structures.

[0048] Memory 130 may store various data used by at least one component of electronic device 101 (e.g., processor 120 or sensor module 176). The various data may include, for example, software (e.g., program 140) and input or output data for commands associated therewith. Memory 130 may include volatile memory 132 or non-volatile memory 134.

[0049] Program 140 may be stored as software in memory 130 and may include, for example, an operating system (OS) 142, middleware 144, or application 146.

[0050] Input module 150 can receive commands or data from outside electronic device 101 (e.g., a user) to be used by another component of electronic device 101 (e.g., processor 120). Input module 150 may include, for example, a microphone, mouse, keyboard, keys (e.g., buttons), or digital pen (e.g., stylus).

[0051] The audio output module 155 can output audio signals to the outside of the electronic device 101. The audio output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as playing multimedia or playing records. The receiver can be used to receive incoming calls. According to an embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0052] Display module 160 can visually provide information to the outside of electronic device 101 (e.g., to a user). Display module 160 may include, for example, a display, a holographic device, or a projector, and control circuitry for controlling a respective one of the display, holographic device, and projector. According to an embodiment, display module 160 may include a touch sensor adapted to detect touch or a pressure sensor adapted to measure the intensity of the force caused by touch.

[0053] The audio module 170 can convert sound into electrical signals and vice versa. According to an embodiment, the audio module 170 can obtain sound via the input module 150, or output sound via the sound output module 155 or headphones of an external electronic device (e.g., electronic device 102) that is directly (e.g., wired) connected to the electronic device 101 or wirelessly connected.

[0054] Sensor module 176 can detect the operating state of electronic device 101 (e.g., power or temperature) or the environmental state outside electronic device 101 (e.g., user state), and then generate an electrical signal or data value corresponding to the detected state. According to embodiments, sensor module 176 may include, for example, a gesture sensor, gyroscope sensor, atmospheric pressure sensor, magnetic sensor, accelerometer, grip sensor, proximity sensor, color sensor, infrared (IR) sensor, biometric sensor, temperature sensor, humidity sensor, or illuminance sensor.

[0055] Interface 177 may support one or more specific protocols used to enable electronic device 101 to connect directly (e.g., wired) or wirelessly to external electronic device (e.g., electronic device 102). According to embodiments, interface 177 may include, for example, a High Definition Multimedia Interface (HDMI), a Universal Serial Bus (USB) interface, a Secure Digital Card (SD) interface, or an audio interface.

[0056] Connection end 178 may include a connector, via which electronic device 101 can be physically connected to an external electronic device (e.g., electronic device 102). According to embodiments, connection end 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0057] The haptic module 179 can convert electrical signals into mechanical stimuli (e.g., vibration or motion) or electrical stimuli that can be recognized by a user through his touch or kinesthesia. According to embodiments, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulator.

[0058] Camera module 180 can capture still images or moving images. According to an embodiment, camera module 180 may include one or more lenses, an image sensor, an image signal processor, or a flash.

[0059] The power management module 188 manages the power supply to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least part of, for example, a power management integrated circuit (PMIC).

[0060] Battery 189 can power at least one component of electronic device 101. According to an embodiment, battery 189 may include, for example, a non-rechargeable primary battery, a rechargeable rechargeable battery, or a fuel cell.

[0061] Communication module 190 can support the establishment of a direct (e.g., wired) or wireless communication channel between electronic device 101 and external electronic devices (e.g., electronic device 102, electronic device 104, or server 108), and perform communication via the established communication channel. Communication module 190 may include one or more communication processors that can operate independently of processor 120 (e.g., application processor (AP)) and support direct (e.g., wired) or wireless communication. According to embodiments, communication module 190 may include wireless communication module 192 (e.g., cellular communication module, short-range wireless communication module, or Global Navigation Satellite System (GNSS) communication module) or wired communication module 194 (e.g., local area network (LAN) communication module or power line communication (PLC) module). One of these communication modules can communicate with an external electronic device via a first network 198 (e.g., a short-range communication network such as Bluetooth™, Wi-Fi Direct, or Infrared Data Association (IrDA)) or a second network 199 (e.g., a long-range communication network such as a traditional cellular network, 5G network, next-generation communication network, the Internet, or a computer network (e.g., a LAN or a wide area network (WAN))). These various types of communication modules can be implemented as a single component (e.g., a single chip) or as multiple components separate from each other (e.g., multiple chips). The wireless communication module 192 can use user information (e.g., an International Mobile Subscriber Identity (IMSI)) stored in the user identification module 196 to identify and verify the electronic device 101 in the communication network (e.g., the first network 198 or the second network 199).

[0062] Wireless communication module 192 can support 5G networks and next-generation communication technologies, such as New Radio (NR) access technologies, following 4G networks. NR access technologies can support enhanced mobile broadband (eMBB), massive machine-type communications (mMTC), or ultra-reliable and low-latency communications (URLLC). Wireless communication module 192 can support high-frequency bands (e.g., millimeter-wave bands) to achieve, for example, high data transmission rates. Wireless communication module 192 can support various technologies used to ensure performance in high-frequency bands, such as beamforming, massive MIMO, full-dimensional MIMO (FD-MIMO), array antennas, analog beamforming, or massive antennas. Wireless communication module 192 can support various requirements specified in electronic device 101, external electronic device (e.g., electronic device 104), or network system (e.g., second network 199). According to an embodiment, the wireless communication module 192 may support peak data rates (e.g., 20 Gbps or higher) for implementing eMBB, loss coverage (e.g., 164 dB or lower) for implementing mMTC, or U-plane delay (e.g., 0.5 ms or less for each of the downlink (DL) and uplink (UL), or 1 ms or less round trip) for implementing URLLC.

[0063] Antenna module 197 can transmit or receive signals or power to or from the outside of electronic device 101 (e.g., external electronic device). According to an embodiment, antenna module 197 may include an antenna comprising a radiating element formed of a conductive material or conductive pattern formed in or on a substrate (e.g., a printed circuit board (PCB)). According to an embodiment, antenna module 197 may include multiple antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication scheme used in a communication network (such as a first network 198 or a second network 199) can be selected from the multiple antennas, for example by communication module 190 (e.g., wireless communication module 192). Signals or power can then be transmitted or received between communication module 190 and external electronic device via the selected at least one antenna. According to an embodiment, another component besides the radiating element (e.g., a radio frequency integrated circuit (RFIC)) may be additionally incorporated into antenna module 197.

[0064] According to various embodiments, antenna module 197 can form a millimeter-wave antenna module. According to embodiments, the millimeter-wave antenna module may include: a printed circuit board; an RFIC disposed on or adjacent to a first surface (e.g., bottom surface) of the printed circuit board and capable of supporting a specified high-frequency band (e.g., millimeter-wave band); and a plurality of antennas (e.g., array antennas) disposed on or adjacent to a second surface (e.g., top or side surface) of the printed circuit board and capable of transmitting or receiving signals in the specified high-frequency band.

[0065] At least some of the aforementioned components may be coupled to each other and transmit signals (e.g., commands or data) between them via a peripheral communication scheme (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industrial processor interface (MIPI)).

[0066] According to an embodiment, commands or data can be sent or received between electronic device 101 and external electronic device 104 via server 108 connected to a second network 199. Each of electronic devices 102 or 104 can be a device of the same or different type as electronic device 101. According to an embodiment, all or some operations to be performed at electronic device 101 can be performed at one or more of external electronic devices 102, 104, or 108. For example, if electronic device 101 is required to automatically perform a function or service, or in response to a request from a user or another device, electronic device 101 may request one or more external electronic devices to perform at least a portion of the function or service, instead of performing the function or service, or may request one or more external electronic devices to perform at least a portion of the function or service in addition to performing the function or service. Upon receiving the request, one or more external electronic devices may perform at least a requested portion of the function or service, or perform additional functions or services related to the request, and transmit the result of the performance to electronic device 101. Electronic device 101 may provide the result, with or without further processing, as at least part of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technologies can be used, for example. Electronic device 101 can use, for example, distributed computing or mobile edge computing to provide ultra-low latency services. In another embodiment, external electronic device 104 may include Internet of Things (IoT) devices. Server 108 may be an intelligent server using machine learning and / or neural networks. According to embodiments, external electronic device 104 or server 108 may be included in a second network 199. Electronic device 101 can be applied to intelligent services based on 5G communication technology or IoT-related technologies (e.g., smart homes, smart cities, smart cars, or healthcare).

[0067] Figure 2a , Figure 2b and Figure 2c An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 2a , Figure 2b and Figure 2c The electronic device 101 may be Figure 1 An example of electronic device 101. For example, Figure 2a , Figure 2b and Figure 2c The display 260 may include Figure 1 At least a portion of the display module 160.

[0068] exist Figure 2a , Figure 2b and Figure 2c Examples 251, 252, 253, and 254 illustrate methods by which electronic device 101 performs functions related to the content in an image through image analysis.

[0069] Referring to Example 251, electronic device 101 can display screen 201 via display 260. Screen 201 may include image 210. Image 210 may include, for example, still images, such as images captured by a camera or images downloaded or received from an external source. Image 210 may include, for example, screenshot images (or captured images). Screenshot images may include images classified as screenshot images. For example, a screenshot image may be an image that includes a flag value indicating that it is a screenshot image. As a non-limiting example, electronic device 101 may obtain captured image 210 in response to receiving input for capturing screen 201 displayed via display 260 and classify the image as a screenshot image. Screen 201 displayed via display 260 may include execution screens for various applications, such as social networking services (SNS), messaging, wallets, screen capture applications, and applications for playing pictures or videos.

[0070] Image 210 can be displayed with a camera (e.g., via electronic device 101) Figure 1 The preview image obtained by the camera module 180 can be distinguished. For example, image 210 can be in a state where it is temporarily, permanently, or semi-permanently stored in the volatile memory of electronic device 101 (e.g., ...). Figure 1 volatile memory 132) or non-volatile memory (e.g., Figure 1 In the non-volatile memory 134, based on the user's selection or certain functions of the electronic device 101, the essentially the same object can be displayed again on the display 260.

[0071] In an embodiment, image 210 may include a coupon image. Image 210 may include image objects representing the content of the coupon image. As a non-limiting example, image 210 may include image objects 211, 212, 214, 216, and 218. For example, image objects 211, 212, and 216 may correspond to text. For example, image objects 214 and 218 may correspond to non-text. For example, image object 211 may represent the name of a product included in the coupon image. For example, image object 212 may represent a product attribute, such as "Free". Although not shown, image 210 may include one or more image objects representing categories (e.g., food), subcategories (e.g., hamburgers), features, and functions as attributes of the product. For example, image object 214 may represent the shape of the product. For example, image object 216 may represent a coupon number. For example, image object 218 may represent an image code for the coupon. As a non-limiting example, the image code may include an imaging code, such as a QR code, barcode, or color code.

[0072] In an embodiment, screen 201 may include a visual object 206 for performing image 210 recognition and / or displaying the recognition results of image 210. For example, electronic device 101 may obtain text corresponding to image 210 based on user input received on visual object 206. For example, electronic device 101 may extract text corresponding to image objects 211, 212, 214, 216, and 218 included in image 210 by performing optical character recognition (OCR) on image 210. However, the method for extracting text is not limited to the above-described OCR, and various techniques may be used.

[0073] In the embodiments, it has been described that electronic device 101 obtains text corresponding to image 210 based on user input to visual object 206, but is not limited thereto. For example, electronic device 101 may extract text included in a screen displayed via display 260 before receiving user input to visual object 206. For example, electronic device 101 may extract text included in image 210 before receiving user input to visual object 206. For example, electronic device 101 may obtain text corresponding to image objects 211, 212, 214, 216, and 218 included in image 210 based on obtaining image 210 (e.g., capturing a screen including image 210) and / or displaying screen 201 including image 210. In this case, electronic device 101 may display screen 205 for representing text extracted from image 210 in response to user input to visual object 206.

[0074] Additionally or optionally, visual objects 206 may be selectively displayed based on the image displayed on electronic device 101. For example, electronic device 101 may recognize whether a coupon image, such as image 210, is being displayed on screen 201. Electronic device 101 may recognize that image 210 is being displayed on screen 201 and may display visual object 206. Electronic device 101 may recognize that another image different from the coupon image is being displayed on screen 201 and may not display visual object 206.

[0075] Referring to Example 252, electronic device 101 may display screen 205 based on received user input regarding visual object 206. Screen 205 may include image 210. In an embodiment, electronic device 101 may highlight and display image objects 211, 212, 214, 216, and 218 identified by recognizing image 210 on screen 205. In Example 252, an example is shown where the area corresponding to the identified image objects 211, 212, 214, 216, and 218 is displayed brighter than another area of ​​screen 205, but this is not a limitation.

[0076] In an embodiment, screen 205 may include a visual object 215 indicating that identified image objects 211, 212, 214, 216, and 218 can be selected. By inputting on each of the identified image objects 211, 212, 214, 216, and 218, at least some of the identified image objects 211, 212, 214, 216, and 218 can be selected.

[0077] In an embodiment, electronic device 101 can obtain text information corresponding to image 210 based on input to visual object 219 on screen 205 to complete the selection of identified image objects 211, 212, 214, 216, and 218. The text information may include category information of image 210, content information of image 210, and summary information about the content information. Category information of image 210 may include, for example, information indicating the subject of image 210 (e.g., coupon). Content information of image 210 is information about the content included in image 210 and may include, for example, at least one of the following: coupon provider's brand, coupon code (e.g., coupon identification number or barcode number), content offered by the coupon, start date and time, expiration date and time, usage method, and restrictions. Summary information may be information that briefly summarizes the content information. In this embodiment, it has been described that electronic device 101 obtains text information based on input to visual object 219, but it is not limited thereto. For example, electronic device 101 may obtain text information based on input to visual object 206 on screen 201. In another example, electronic device 101 can obtain text information corresponding to image 210 by selecting at least some of the identified image objects 211, 212, 214, 216, and 218 without input to visual object 219. In another example, electronic device 101 can obtain text information in image 210 based on obtaining image 210 (e.g., by capturing a screen including image 210) and / or displaying screen 201 including image 210.

[0078] For example, electronic device 101 can draw upon artificial intelligence models (e.g., Figure 9 The text information extraction model 903 obtains text information corresponding to image 210, with text and prompts extracted from identified image objects 211, 212, 214, 216, and 218 being input into the AI ​​model. The prompts can be commands formed from natural language, used to determine categories from the extracted text and generate information corresponding to the determined categories (e.g., content information and overview information) in a specified format. For example, the AI ​​model can output text information in a specified data format (such as JavaScript Object Notation (JSON)) in response to the prompts. As a non-limiting example, the AI ​​model can include multiple parameters for driving the neural network, such as bidirectional encoder representations from a transformer (BERT) or generative pre-trained transformer (GPT). As a non-limiting example, the AI ​​model can include a large language model (LLM) for processing natural language based on a large number of parameters.

[0079] In this embodiment, when only some of the identified image objects 211, 212, 214, 216, and 218 are selected through input to screen 205, electronic device 101 can obtain text information based on the selected image objects. For example, electronic device 101 can obtain text information corresponding to image 210 from an artificial intelligence model by inputting text and prompts extracted from the image objects selected from the identified image objects 211, 212, 214, 216, and 218 into the artificial intelligence model. Alternatively, to enable electronic device 101 to obtain text information, image 210 can be input together with the extracted text into the artificial intelligence model.

[0080] In an embodiment, electronic device 101 can identify the category of image 210 by using text information. For example, electronic device 101 can identify the category of image 210 by using category information of image 210 included in text information.

[0081] refer to Figure 2b In Example 253, electronic device 101 can display screen (or window) 220 via display 260. For example, electronic device 101 can display screen 220 together with screen 205. For example, electronic device 101 can display screen 220 that partially overlaps with screen 205 via display 260, but is not limited thereto. For example, electronic device 101 can display screen 205 of a first size in a portion of the entire display area of ​​display 260, and display screen 220 of a second size smaller than the first size in the remaining area. For example, screen 220 can be displayed as a split or pop-up window of screen 205.

[0082] In an embodiment, screen 220 may include a visual object 221 representing category information of image 210. Visual object 221 may correspond to text such as “Coupon information is in the screenshot,” but is not limited thereto. For example, screen 220 may include an icon associated with the category of image 210. In an embodiment, electronic device 101 may display screen 220 based on checking an application corresponding to the category of image 210. As a non-limiting example, the application corresponding to a coupon as a category of image 210 may include an application for managing coupons, such as a wallet application. For another example, the application corresponding to a coupon may include a payment application that supports payments using technologies such as Near Field Communication (NFC) or Magnetic Secure Transmission (MST).

[0083] Referring to Example 254, in an embodiment, electronic device 101 can display screen (or window) 230 via display 260. For example, electronic device 101 can display screen 230 together with screen 205. For example, electronic device 101 can display screen 230 via display 260 in response to receiving input (e.g., touch input or swipe-up gesture) to visual object 221 on screen 220. Screen 230 may have a larger size than screen 220 and may be displayed on screen 205 by partially overlapping, but is not limited thereto. For example, screen 230 may be displayed as a split or pop-up window of screen 205. For example, electronic device 101 may display screen 205 in a portion of the entire display area of ​​display 260 and display screen 230, which is smaller than screen 205 but larger than screen 220, in the remaining area. In the embodiment, displaying screen 230 by input to visual object 221 on screen 220 has been described, but is not limited thereto. For example, screen 230 may be displayed after a predetermined time following the display of screen 220. As another example, screen 220 can be omitted, and screen 230 can be displayed. For example, it can be omitted. Figure 2b Example 253.

[0084] In an embodiment, screen 230 may include a visual object 221 representing category information of image 210, a visual object 222 representing at least a portion of overview information, and a visual object 223 for performing an application corresponding to image 210. Although Figure 2b The image 223 is shown, but is not limited to it. For example, in the presence of multiple applications corresponding to image 210 (such as...) Figure 7 In the case of visual objects 732 and 733), multiple visual objects 223 can also be displayed to perform their functions. For example, in the case where there are multiple categories corresponding to image 210, such as Figure 8 Visual objects 832 and 833 can display visual objects used to perform applications corresponding to each of multiple categories (e.g., a wallet application and a calendar application). As a non-limiting example, visual object 221 can be displayed in a first area of ​​screen 230, visual object 222 can be displayed in a second area of ​​screen 230, and visual object 223 can be displayed in a third area of ​​screen 230. As a non-limiting example, the second area can be located between the first and third areas. As a non-limiting example, based on... Figure 2b As illustrated, the third region may be located below the first and second regions. In an embodiment, the electronic device 101 may display the screen 230 based on an application that corresponds to the category of the image 210.

[0085] It has been described that screen 230, which includes visual object 223, is displayed through user input to screen 220, but is not limited thereto. For example, electronic device 101 may display visual object 223 or another screen including visual object 223 on display 260, instead of screen 220 or screen 230 (e.g., displayed together with screen 201 or screen 205 including image 210), based on obtaining text information about image 210.

[0086] refer to Figure 2c In Example 255, in this embodiment, electronic device 101 may display a screen 240, different from screen 230, via display 260 in response to receiving input associated with visual object 223. For example, electronic device 101 may display screen 240 by executing an application corresponding to image 210. Screen 240 may include an application execution screen. For example, screen 240 may include an execution screen capable of performing functions of an application corresponding to the category of image 210. For example, screen 240 may include an application execution screen capable of storing coupons for image 210.

[0087] In an embodiment, electronic device 101 may execute an application based at least in part on text information corresponding to image 210. For example, in response to input to visual object 223, electronic device 101 may display screen 240 while at least a portion of text information is input into an input field of the application's execution screen. For example, text information corresponding to an object selected from image objects 211, 212, 214, 216, and 218 may be input into the input field of the application's execution screen, but is not limited thereto.

[0088] Additionally, the electronic device 101 can display the screen 240 in response to input to the visual object 223, while an image 210 (e.g., image 241) is attached. Furthermore, prior to displaying the screen 240, a screen may be provided for at least a portion of the image 210 to be specified by the user, such as... Figure 14d The screen 1407. In this case, the electronic device 101 can display a screen 240 with an image 241 attached that corresponds to a designated area of ​​the image 210.

[0089] Figure 3a , Figure 3b , Figure 3c and Figure 3d An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 3a , Figure 3b , Figure 3c and Figure 3d The electronic device 101 may be Figure 1 Example of electronic device 101.

[0090] exist Figure 3a , Figure 3b , Figure 3c and Figure 3d The image shows an example of a method by which electronic device 101 performs functions related to the content in an image by analyzing the image.

[0091] refer to Figure 3a Electronic device 101 can display screen 301 via display 260. Screen 301 may include image 310. Image 310 may include, for example, still images and / or screenshots. As a non-limiting example, if image 310 is a screenshot, electronic device 101 may acquire image 310 in response to receiving input for capturing screen 301 displayed via display 260. In embodiments, image 310 may be... Figure 2a Image 210 is a different image. As described later, image 310 may be an image classified into a different category than image 210. Screen 301 displayed via display 260 may include execution screens for various applications such as SNS, messaging, wallet, screen capture applications, and applications for playing pictures or videos.

[0092] Image 310 can be connected to a camera (e.g., via electronic device 101) Figure 1 The preview image obtained by the camera module 180 is distinguishable. For example, image 310 may be temporarily, permanently or semi-permanently stored in the volatile or non-volatile memory of electronic device 101, and may be in a state in which substantially the same object can be displayed again by display 260 based on user selection or some function of electronic device 101.

[0093] In one embodiment, image 310 may include an image representing a specific area (or location). Figure 3a Image 310 has been shown to represent information about a specific restaurant, but is not limited to the example shown.

[0094] Image 310 may include image objects representing the content within image 310. For example, image 310 may include image objects used to describe a specific area. As a non-limiting example, image 310 may include image objects 312, 314, 316, and 318. For example, image objects 312, 314, and 316 may correspond to text. For example, image object 318 may correspond to non-text. For example, image object 312 may represent the name of a specific location, such as "B coffee Yangjae branch". For example, image object 314 may represent the address of a specific location. For example, image object 316 may represent the operating date and time of a specific location. For example, image object 318 may represent a logo image of a specific location.

[0095] In an embodiment, screen 301 may include a visual object 306 (e.g., visual object 206) for performing image 310 recognition and / or displaying the recognition results of image 310. For example, electronic device 101 may obtain text corresponding to image 310 based on user input received on visual object 306. For example, electronic device 101 may identify image objects 312, 314, 316, and 318 included in image 310 and extract text corresponding to image objects 312, 314, 316, and 318 by performing OCR on image 310. However, the method for extracting text is not limited to the above-described OCR, and various techniques may be used.

[0096] In the embodiments, it has been described that electronic device 101 obtains text corresponding to image 310 based on user input to visual object 306, but is not limited thereto. For example, electronic device 101 may extract text included in image 310 based on obtaining image 310 (e.g., capturing a screen including image 310) and / or displaying screen 301 including image 310. In this case, electronic device 101 may display screen 305 for representing text extracted from image 310 in response to user input to visual object 306. Additionally or alternatively, visual object 306 may be selectively displayed based on the image displayed in electronic device 101. For example, electronic device 101 may recognize whether an image representing a specific location, such as image 310, is being displayed on screen 301. Electronic device 101 may recognize that image 310 is being displayed on screen 301 and may display visual object 306. Electronic device 101 may recognize that another image different from the image representing a specific location is being displayed on screen 301 and may not display visual object 306.

[0097] In an embodiment, electronic device 101 can obtain text information corresponding to image 310. For example, electronic device 101 can obtain text information corresponding to image 310 based on input to visual object 306 on screen 301. For another example, electronic device 101 can obtain text information about image 310 based on obtaining image 310 (e.g., by capturing a screen including image 310) and / or displaying screen 301 including image 310. The text information may include category information of image 310, content information of image 310, and summary information about the content information. The category information of image 310 is information indicating the subject of image 310 and may be, for example, a location. The content information of image 310 is information about the content included in image 310 and may include, for example, at least one of the following: location name, opening hours, closing hours, time zone, telephone number, and address.

[0098] For example, electronic device 101 can obtain text information corresponding to image 310 from an artificial intelligence model, with text and prompts extracted from image objects 312, 314, 316, and 318 being input into the artificial intelligence model. Alternatively, in order for electronic device 101 to obtain text information, image 310 can be input into the artificial intelligence model along with the extracted text.

[0099] refer to Figure 3b Electronic device 101 can display screen (or window) 320 via display 260. For example, electronic device 101 can display screen 320 together with screen 305 including image 310. For example, electronic device 101 can display screen 320 that partially overlaps with screen 305 via display 260, but is not limited thereto. For example, electronic device 101 can display screen 305 with a first size in a portion of the entire display area of ​​display 260, and display screen 320 with a second size smaller than the first size in the remaining area. For example, screen 320 can be displayed as a split or pop-up window of screen 305. In embodiments, screen 305 may include image 310, and may be a screen in which visual objects 312, 314, and 316 corresponding to text included in image 310 are highlighted. Electronic device 101 can display screen 305 based on text extracted from image 310.

[0100] Although not shown, an interface for selecting the identified image objects 312, 314, 316, and 318 can be provided via screen 305 before the output on screen 320 (e.g., Figure 2b Visual objects 215 and 219 on screen 205.

[0101] In an embodiment, screen 320 may include a visual object 321 for representing category information of image 310. Visual object 321 may correspond to text such as “Place information is in the screenshot,” but is not limited thereto. For example, screen 320 may include icons associated with the category of image 310. In an embodiment, electronic device 101 may display screen 320 based on checking an application corresponding to the category of image 310. As a non-limiting example, the application corresponding to a location as a category of image 310 may include an application capable of performing location-related functions, such as a map application.

[0102] refer to Figure 3cIn this embodiment, electronic device 101 may display screen (or window) 330 via display 260. For example, electronic device 101 may display screen 330 together with screen 305. For example, electronic device 101 may display screen 330 via display 260 in response to receiving input (e.g., touch input or swipe-up gesture) to visual object 321 on screen 320. Screen 330 may have a larger size than screen 320 and may be displayed on screen 305 by partially overlapping, but is not limited thereto. For example, screen 330 may be displayed as a split or pop-up window of screen 305. For example, electronic device 101 may display screen 305 in a portion of the entire display area of ​​display 260 and display screen 330, which is smaller than screen 305 but larger than screen 320, in the remaining area. In this embodiment, displaying screen 330 by input to visual object 321 on screen 320 has been described, but is not limited thereto. For example, screen 330 may be displayed after a predetermined time has elapsed since screen 320 was displayed. As another example, the display of screen 320 can be omitted, and screen 330 can be displayed.

[0103] In an embodiment, screen 330 may include a visual object 321 representing category information of image 310, a visual object 322 representing at least a portion of overview information, and a visual object 323 for performing an application corresponding to image 310. Although Figure 3c The image 323 is shown, but is not limited to it. For example, in the presence of multiple applications corresponding to the image 310 (such as...) Figure 7 Visual objects 732 and 733 or Figure 8 In the case of visual objects 832 and 833, each of the visual objects used to perform multiple applications can be displayed. For example, in addition to text information about a specific location, image 310 may also include text information about schedules and / or appointments. In this case, electronic device 101 can display not only visual object 323, but also applications associated with the text information about schedules and / or appointments, such as visual objects used to perform calendar applications.

[0104] As a non-limiting example, visual object 321 can be displayed in a first area of ​​screen 330, visual object 322 can be displayed in a second area of ​​screen 330, and visual object 323 can be displayed in a third area of ​​screen 330. As a non-limiting example, the second area can be located between the first and third areas. As a non-limiting example, based on... Figure 3c As illustrated, the third region may be located below the first and second regions. In an embodiment, the electronic device 101 may display the screen 330 based on checking the application corresponding to the category of the image 310.

[0105] It has been described that screen 330, which includes visual object 323, is displayed through user input to screen 320, but is not limited thereto. For example, electronic device 101 may display visual object 323 or another screen including visual object 323 on display 260, instead of screen 320 or screen 330 (e.g., displayed together with screen 301 or screen 305 including image 310), based on obtaining text information about image 310.

[0106] refer to Figure 3d In an embodiment, electronic device 101 may display a screen 340, different from screen 330, via display 260 in response to receiving input related to visual object 323. For example, electronic device 101 may display screen 340 by executing an application corresponding to image 310. Screen 340 may include an application execution screen. For example, screen 340 may include an execution screen capable of performing functions of an application corresponding to the category of image 310. For example, screen 340 may include an execution screen of an application capable of retrieving the location of image 310. In an embodiment, the application corresponding to screen 340 (e.g., a map application) may be different from the application corresponding to screen 240 (e.g., a wallet application).

[0107] In embodiments, electronic device 101 may execute applications based at least in part on text information corresponding to image 310. For example, electronic device 101 may display screen 340 in a state where the location of image 310 is retrieved in response to input to visual object 323. Location retrieval may be performed via an external server for providing applications and / or services for using applications. Electronic device 101 may store map information corresponding to screen 340 in the memory of electronic device 101 (e.g., Figure 1 The map information is stored in memory 130, allowing it to be provided without establishing a network connection with an external server. Figure 3d The image shows a screen 340 that indicates the path from the current position 341 to the position 342 corresponding to the position of the image 310, but is not limited to the example shown.

[0108] Figure 4a , Figure 4b , Figure 4c and Figure 4d An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 4a , Figure 4b , Figure 4c and Figure 4d The electronic device 101 may be Figure 1 Example of electronic device 101.

[0109] exist Figure 4a , Figure 4b , Figure 4c and Figure 4d The image shows an example of a method by which electronic device 101 performs functions related to the content in an image by analyzing the image.

[0110] refer to Figure 4a Electronic device 101 can display screen 401 via display 260. Screen 401 may include image 410. Image 410 may include, for example, still images and / or screenshots. As a non-limiting example, if image 410 is a screenshot, electronic device 101 may acquire image 410 in response to receiving input for capturing screen 401 displayed via display 260. In embodiments, image 410 may be an image different from images 210 and 310. As described later, image 410 may be an image categorized into a different class than images 210 and 310. Screen 401 displayed via display 260 may include execution screens of various applications such as SNS, messaging, wallet, screen capture applications, and applications for playing pictures or videos.

[0111] Image 410 can be connected to a camera (e.g., via electronic device 101) Figure 1 The preview image obtained by the camera module 180 is distinguishable. For example, image 410 may be temporarily, permanently or semi-permanently stored in the volatile or non-volatile memory of electronic device 101, and may be in a state in which substantially the same object can be displayed again by display 260 based on user selection or some function of electronic device 101.

[0112] In one embodiment, image 410 may include an image representing a specific purchasable product. Figure 4a Image 410 has been shown to represent information about toy castles available for purchase, but is not limited to the example shown.

[0113] Image 410 may include image objects representing the content in image 410. For example, image 410 may include image objects describing a particular product. As a non-limiting example, image 410 may include image objects 412, 414, 416, and 418. For example, image objects 412, 414, and 416 may correspond to text. For example, image object 418 may correspond to non-text. For example, image objects 412 and 414 may represent attributes of a particular product, such as "figure" and "castle." For example, image object 416 may represent the price of a particular product. For example, image object 418 may represent the shape of a particular product. Although not shown, image 410 may include image objects indicating the path to purchasing a particular product, such as a webpage address.

[0114] In an embodiment, screen 401 may include a visual object 406 (e.g., visual object 306) for performing image 410 recognition and / or displaying the recognition results of image 410. For example, electronic device 101 may obtain text corresponding to image 410 based on user input received on visual object 406. For example, electronic device 101 may identify image objects 412, 414, 416, and 418 included in image 410 and extract text corresponding to image objects 412, 414, 416, and 418 by performing OCR on image 410. However, the method for extracting text is not limited to the above-described OCR, and various techniques may be used.

[0115] In the embodiments, it has been described that electronic device 101 obtains text corresponding to image 410 based on user input to visual object 406, but is not limited thereto. For example, electronic device 101 may extract text included in image 410 based on obtaining image 410 (e.g., capturing a screen including image 410) and / or displaying screen 401 including image 410. In this case, electronic device 101 may display screen 405 for representing text extracted from image 410 in response to user input to visual object 406.

[0116] Additionally or optionally, visual objects 406 may be selectively displayed based on the images displayed in electronic device 101. For example, electronic device 101 may recognize whether an image representing a specific product (such as image 410) is being displayed on screen 401. Electronic device 101 may recognize that image 410 is being displayed on screen 401 and may display visual object 406. Electronic device 101 may recognize that another image different from the image representing a specific product is being displayed on screen 401 and may not display visual object 406.

[0117] In an embodiment, electronic device 101 can obtain text information corresponding to image 410. For example, electronic device 101 can obtain text information corresponding to image 410 based on input to visual object 406 on screen 401. For another example, electronic device 101 can obtain text information about image 410 based on obtaining image 410 (e.g., by capturing a screen including image 410) and / or displaying screen 401 including image 410. The text information may include category information of image 410, content information of image 410, and summary information about the content information. Category information of image 410 is information indicating the subject of image 410, and may be, for example, shopping. Content information of image 410 is information about the content included in image 410, and may include at least one of, for example, brand name, product name, cost, supply price, promotional content, and purchase destination for a particular product.

[0118] For example, electronic device 101 can obtain text information corresponding to image 410 from an artificial intelligence model, with text and prompts extracted from image objects 412, 414, 416, and 418 being input into the artificial intelligence model. Alternatively, in order for electronic device 101 to obtain text information, image 410 can be input into the artificial intelligence model along with the extracted text.

[0119] refer to Figure 4b Electronic device 101 can display screen (or window) 420 via display 260. For example, electronic device 101 can display screen 420 together with screen 405. For example, electronic device 101 can display screen 420 that partially overlaps with screen 405 via display 260, but is not limited thereto. For example, electronic device 101 can display screen 405 with a first size in a portion of the entire display area of ​​display 260, and display screen 420 with a second size smaller than the first size in the remaining area. For example, screen 420 can be displayed as a split or pop-up window of screen 405. In embodiments, screen 405 may include image 410, and may be a screen in which visual objects 412, 414, 416, and 418 identified in image 410 are highlighted. Electronic device 101 can display screen 405 based on text extracted from image 410.

[0120] Although not shown, an interface for selecting the identified image objects 412, 414, 416, and 418 may be provided before the output on screen 420 (e.g., Figure 2b Visual objects 215 and 219 on screen 205.

[0121] In an embodiment, screen 420 may include a visual object 421 for representing category information of image 410. Visual object 421 may correspond to text such as “Shopping information is in the screenshot,” but is not limited thereto. For example, screen 420 may include icons associated with the category of image 410. In an embodiment, electronic device 101 may display screen 420 based on checking an application corresponding to the category of image 410. As a non-limiting example, the application corresponding to shopping as the category of image 410 may include an application for providing reminders for shopping, such as a reminder app.

[0122] refer to Figure 4c In this embodiment, electronic device 101 may display screen (or window) 430 via display 260. For example, electronic device 101 may display screen 430 together with screen 405. For example, electronic device 101 may display screen 430 via display 260 in response to receiving input (e.g., touch input or swipe-up gesture) to visual object 421 on screen 420. Screen 430 may have a larger size than screen 420 and may be displayed on screen 405 by partially overlapping, but is not limited thereto. For example, screen 430 may be displayed as a split or pop-up window of screen 405. For example, electronic device 101 may display screen 405 in a portion of the entire display area of ​​display 260 and display screen 430, which is smaller than screen 405 but larger than screen 420, in the remaining area. In this embodiment, displaying screen 430 by input to visual object 421 on screen 420 has been described, but is not limited thereto. For example, screen 430 may be displayed after a predetermined time has elapsed since screen 420 was displayed. As another example, the display of screen 420 can be omitted, and screen 430 can be displayed.

[0123] In an embodiment, screen 430 may include a visual object 421 representing category information of image 410, a visual object 422 representing at least a portion of overview information, and a visual object 423 for performing an application corresponding to image 410. Although Figure 4c The image shown is a visual object 423, but it is not limited to this. For example, in the presence of multiple applications corresponding to image 410 (such as...) Figure 7 Visual objects 732 and 733 or Figure 8 In the case of visual objects 832 and 833, each of the visual objects used to perform multiple applications can be displayed.

[0124] As a non-limiting example, visual object 421 can be displayed in a first area of ​​screen 430, visual object 422 can be displayed in a second area of ​​screen 430, and visual object 423 can be displayed in a third area of ​​screen 430. As a non-limiting example, the second area can be located between the first and third areas. As a non-limiting example, based on Figure 4c As illustrated, the third region may be located below the first and second regions. In an embodiment, the electronic device 101 may display the screen 430 based on checking the application corresponding to the category of the image 410.

[0125] It has been described that screen 430, which includes visual object 423, is displayed via user input to screen 420, but is not limited thereto. For example, electronic device 101 may display visual object 423 or another screen including visual object 423 via display 260, instead of screen 420 or screen 430 (e.g., displayed together with screen 401 or screen 405 including image 410), based on obtaining text information about image 410.

[0126] refer to Figure 4d In an embodiment, electronic device 101 may display a screen 440, different from screen 430, via display 260 in response to receiving input related to visual object 423. For example, electronic device 101 may display screen 440 by executing an application corresponding to image 410. Screen 440 may include an application execution screen. For example, screen 440 may include an execution screen capable of performing the functions of an application corresponding to the category of image 410. For example, screen 440 may include an execution screen capable of storing shopping information corresponding to image 410. In an embodiment, the application corresponding to screen 440 (e.g., a reminder application) may be different from the applications corresponding to screen 240 (e.g., a wallet application) and screen 340 (e.g., a map application).

[0127] In an embodiment, electronic device 101 may execute an application based at least in part on text information corresponding to image 410. For example, in response to input to visual object 423, electronic device 101 may display screen 440 while at least a portion of the text information corresponding to image 410 is input into an input field of the application's execution screen. For example, text information corresponding to objects selected from image objects 412, 414, 416, and 418 may be input into the input field of the application's execution screen, but is not limited thereto.

[0128] Additionally, in response to input to visual object 423, electronic device 101 can display screen 440 with an image (such as image 441) at least partially based on image 410 attached. Furthermore, prior to displaying screen 440, a screen may be provided for user-specified at least a portion of image 410, such as... Figure 14d Screen 1407. In this case, screen 440 may be displayed with an image 441 attached to a designated area corresponding to image 410.

[0129] Figure 5a , Figure 5b , Figure 5c and Figure 5d An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 5a , Figure 5b , Figure 5c and Figure 5d The electronic device 101 may be Figure 1 Example of electronic device 101.

[0130] exist Figure 5a , Figure 5b , Figure 5c and Figure 5d The image shows an example of a method by which electronic device 101 performs functions related to the content in an image by analyzing the image.

[0131] refer to Figure 5a Electronic device 101 can display screen 501 via display 260. Screen 501 may include image 510. Image 510 may include, for example, still images and / or screenshots. As a non-limiting example, if image 510 is a screenshot, electronic device 101 may acquire image 510 in response to receiving input for capturing screen 501 displayed via display 260. In embodiments, image 510 may be an image different from images 210, 310, and 410. As described later, image 510 may be an image categorized into a different class than images 210, 310, and 410. Screen 501 displayed via display 260 may include execution screens of various applications such as SNS, messaging, wallets, screen capture applications, and applications for playing pictures or videos.

[0132] Image 510 can be connected to a camera (e.g., via electronic device 101) Figure 1 The preview image obtained by the camera module 180 is distinguishable. For example, the image 510 may be temporarily, permanently or semi-permanently stored in the volatile or non-volatile memory of the electronic device 101, and may be in a state in which substantially the same object can be displayed again by the display 260 based on the user's selection or some function of the electronic device 101.

[0133] In one embodiment, image 510 may include a ticket image. Figure 5a In this context, image 510 is illustrated as a movie ticket image, but is not limited to the illustrated example. For example, image 510 may include an image representing a booking schedule, such as an airline ticket.

[0134] Image 510 may include image objects representing the content in image 510. For example, image 510 may include image objects representing information about tickets. As a non-limiting example, image 510 may include image objects 512, 514, 516, 517, and 518. For example, image objects 512, 514, 516, and 517 may correspond to text. For example, image object 518 may correspond to non-text. For example, image object 512 may represent a movie title. For example, image object 514 may represent the movie screening date and time. For example, image object 516 may represent booking details such as theater, screening room, number of people, seat, and price. For example, image object 517 may represent payment information. For example, image object 518 may represent a movie poster.

[0135] In an embodiment, screen 501 may include a visual object 506 (e.g., visual object 406) for performing image 510 recognition and / or displaying the recognition results of image 510. For example, electronic device 101 may obtain text corresponding to image 510 based on user input received on visual object 506. For example, electronic device 101 may recognize image objects 512, 514, 516, 517, and 518 included in image 510, and extract text corresponding to image objects 512, 514, 516, 517, and 518 by performing OCR on image 510. However, the method for extracting text is not limited to the above-described OCR, and various techniques may be used.

[0136] In the embodiments, it has been described that electronic device 101 obtains text corresponding to image 510 based on user input to visual object 506, but is not limited thereto. For example, electronic device 101 may extract text included in a screen displayed via display 260 before receiving user input to visual object 506. For example, electronic device 101 may extract text included in image 510 before receiving user input to visual object 506. For example, electronic device 101 may obtain text corresponding to image objects 512, 514, 516, 517, and 518 included in image 510 based on obtaining image 510 (e.g., capturing a screen including image 510) and / or displaying screen 501 including image 510. In this case, electronic device 101 may display screen 505 for representing text extracted from image 510 in response to user input to visual object 506.

[0137] Additionally or optionally, visual objects 506 may be selectively displayed based on the images displayed in electronic device 101. For example, electronic device 101 may recognize whether a ticket image, such as image 510, is being displayed on screen 501. Electronic device 101 may recognize that a ticket image, such as image 510, is being displayed on screen 501 and may display visual object 506. Electronic device 101 may recognize that another image different from the ticket image is being displayed on screen 501 and may not display visual object 506.

[0138] In an embodiment, electronic device 101 can obtain text information corresponding to image 510. For example, electronic device 101 can obtain text information corresponding to image 510 based on input to visual object 506 on screen 501. For another example, electronic device 101 can obtain text information about image 510 based on obtaining image 510 (e.g., by capturing a screen including image 510) and / or displaying screen 501 including image 510. The text information may include category information of image 510, content information of image 510, and overview information about the content. Category information of image 510 is information indicating the subject of image 510 and may be, for example, a schedule. Content information of image 510 is information about the content included in image 510 and may include at least one of, for example, title, start date, start time, start time zone, end date, end time, end time zone, start location, end location, and reservation location.

[0139] For example, electronic device 101 can obtain text information corresponding to image 510 from an artificial intelligence model, with text and prompts extracted from image objects 512, 514, 516, 517, and 518 being input into the artificial intelligence model. Alternatively, in order for electronic device 101 to obtain text information, image 510 can be input into the artificial intelligence model along with the extracted text.

[0140] refer to Figure 5bElectronic device 101 can display screen (or window) 520 via display 260. For example, electronic device 101 can display screen 520 together with screen 505. For example, electronic device 101 can display screen 520 that partially overlaps with screen 505 via display 260, but is not limited thereto. For example, electronic device 101 can display screen 505 with a first size in a portion of the entire display area of ​​display 260, and display screen 520 with a second size smaller than the first size in the remaining area. For example, screen 520 can be displayed as a split or pop-up window of screen 505. In embodiments, screen 505 may include image 510, and may be a screen in which visual objects 512, 514, 516, and 517 identified in image 510 are highlighted. Electronic device 101 can display screen 505 based on text extracted from image 510.

[0141] Although not shown, an interface for selecting the identified image objects 512, 514, 516, 517, and 518 may be provided before the output on screen 520 (e.g., Figure 2b Visual objects 215 and 219 on screen 205.

[0142] In an embodiment, screen 520 may include a visual object 521 for representing category information of image 510. Visual object 521 may correspond to text such as “Reservation information is in the screenshot,” but is not limited thereto. For example, screen 520 may include an icon associated with the category of image 510. In an embodiment, electronic device 101 may display screen 520 based on checking an application corresponding to the category of image 510. As a non-limiting example, the application corresponding to a schedule, which is a category of image 510, may include an application for storing and managing schedules, such as a calendar application.

[0143] refer to Figure 5cIn this embodiment, electronic device 101 may display screen (or window) 530 via display 260. For example, electronic device 101 may display screen 530 together with screen 505. For example, electronic device 101 may display screen 530 via display 260 in response to receiving input (e.g., touch input or swipe-up gesture) to visual object 521 on screen 520. Screen 530 may have a larger size than screen 520 and may be displayed on screen 505 by partially overlapping, but is not limited thereto. For example, screen 530 may be displayed as a split or pop-up window of screen 505. For example, electronic device 101 may display screen 505 in a portion of the entire display area of ​​display 260 and display screen 530, which is smaller than screen 505 but larger than screen 520, in the remaining area. In this embodiment, displaying screen 530 by input to visual object 521 on screen 520 has been described, but is not limited thereto. For example, screen 530 may be displayed after a predetermined time has elapsed since screen 520 was displayed. As another example, the display of screen 520 can be omitted, and screen 530 can be displayed.

[0144] In an embodiment, screen 530 may include a visual object 521 representing category information of image 510, a visual object 522 representing at least a portion of overview information, and a visual object 523 for performing an application corresponding to image 510. Although Figure 5c The image 523 is shown, but is not limited to it. For example, in the presence of multiple applications corresponding to the image 510 (such as...) Figure 7 Visual objects 732 and 733 or Figure 8 In the case of visual objects 832 and 833, each of the visual objects used to perform multiple applications can be displayed.

[0145] As a non-limiting example, visual object 521 can be displayed in a first area of ​​screen 530, visual object 522 can be displayed in a second area of ​​screen 530, and visual object 523 can be displayed in a third area of ​​screen 530. As a non-limiting example, the second area can be located between the first and third areas. As a non-limiting example, based on Figure 5c As illustrated, the third region may be located below the first and second regions. In an embodiment, the electronic device 101 may display the screen 530 based on checking the application corresponding to the category of the image 510.

[0146] It has been described that screen 530, which includes visual object 523, is displayed through user input to screen 520, but is not limited thereto. For example, electronic device 101 may display visual object 523 or another screen including visual object 523 on display 260, instead of screen 520 or screen 530 (e.g., displayed together with screen 501 or screen 505 including image 510), based on obtaining text information about image 510.

[0147] refer to Figure 5d In an embodiment, electronic device 101 may display a screen 540, different from screen 530, via display 260 in response to receiving input related to visual object 523. For example, electronic device 101 may display screen 540 by executing an application corresponding to image 510. Screen 540 may include an execution screen for an application. For example, screen 540 may include an execution screen capable of performing the functions of an application corresponding to the category of image 510. For example, screen 540 may include an execution screen for an application capable of storing schedule information corresponding to image 510. In an embodiment, the application corresponding to screen 540 (e.g., a calendar application) may be different from the application corresponding to screen 240 (e.g., a wallet application), the application corresponding to screen 340 (e.g., a map application), and the application corresponding to screen 440 (e.g., a reminder application).

[0148] In embodiments, electronic device 101 may execute an application based at least in part on text information corresponding to image 510. For example, in response to input to visual object 523, electronic device 101 may display screen 540 while at least a portion of the text information corresponding to image 510 is input into an input field of the application's execution screen. For example, text information corresponding to an object selected from image objects recognized from image 510 (e.g., image objects 512, 514, 516, and 517) may be input into the input field of the application's execution screen, but is not limited thereto.

[0149] In the embodiments, it has been described that electronic device 101 uses an artificial intelligence model to obtain text information, wherein text extracted from images (e.g., images 210, 310, 410, or 510) and prompts are input into the artificial intelligence model, but are not limited thereto. For example, before obtaining text information, electronic device 101 may obtain category information of the image. For example, electronic device 101 may obtain category information from the artificial intelligence model by inputting text extracted from the image and a first prompt for generating category information from the extracted text into the artificial intelligence model. Subsequently, electronic device 101 may obtain text information from the artificial intelligence model by inputting a second prompt for generating content information corresponding to the category information from the extracted text into the artificial intelligence model. As another example, to obtain text information, a machine learning-based model may be used instead of a prompt-based generative model.

[0150] In this embodiment, the electronic device 101 may include an artificial intelligence model. For example, the artificial intelligence model may be stored in the memory of the electronic device 101 (e.g., ...). Figure 1 The artificial intelligence model may be stored in a server (e.g., memory 130) outside the electronic device 101. In an embodiment, the artificial intelligence model may be included in a server (e.g., memory 130) outside the electronic device 101. Figure 1 In server 108). For example, an artificial intelligence model can be stored in the server's memory. In this case, electronic device 101 can communicate via communication circuits (e.g., Figure 1 The communication module 190) sends the text extracted from the image to the server. Thereafter, the electronic device 101 can receive text information from the server via the communication circuit. Furthermore, to obtain the text information, the electronic device 101 can send the image along with the text extracted from the image to the server. In embodiments, the electronic device 101 can obtain additional information about the image by using the image and / or text information about the image. For example, the electronic device 101 can obtain the additional information by retrieving the image and at least a portion of the text information about the image via a network such as the Internet. For example, the additional information may include information not included in the text information about the image. For example, the image is a coffee coupon (e.g., Figure 2a Image 210 may not include information about specific locations where coffee coupons can be used. In this case, electronic device 101 can obtain information about specific locations as supplementary information by using images and / or image-based text information retrieval.

[0151] In an embodiment, electronic device 101 may use text information to identify the category of an image. For example, electronic device 101 may identify the category of an image by using category information of image 510 included in text information.

[0152] Figure 6An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 6 The electronic device 101 may be Figure 1 Example of electronic device 101.

[0153] refer to Figure 6 Electronic device 101 can display screen 601 via display 260. Screen 601 may include image 610. Image 610 may include, for example, still images and / or screenshots. As a non-limiting example, if image 610 is a screenshot, electronic device 101 may acquire image 610 in response to receiving input for capturing screen 601 displayed via display 260.

[0154] In this embodiment, electronic device 101 can extract text corresponding to an image object included in image 610. Electronic device 101 can obtain text information corresponding to image 610 by using the extracted text.

[0155] Electronic device 101 can use text information to identify the category of image 610. For example, image 610 may have a different category than images 210, 310, 410, and 510 described above. For example, the category of image 610 may include movie information, food information, music information, technical information, news, casual conversation, meeting logs, documents, personal information, running information, SNS, or other categories. In this case, electronic device 101 can determine that there is no application corresponding to the category of image 610 detected using text information. Based on the determination that there is no application corresponding to the category of image 610, electronic device 101 can partially overlap screen 602 on screen 601 via display 260.

[0156] In an embodiment, screen 602 may include a visual object 621 indicating that there is no action that can be performed in association with image 610. As a non-limiting example, visual object 621 may include text such as “No action that may be performed”.

[0157] Although not shown, the category of the image detected using text information and the corresponding application can be changed according to the user's settings.

[0158] For example, even if the category of image 610 is identified as food information, the category of image 610 can be changed to a different category (e.g., shopping) through user settings. Afterward, electronic device 101 can determine the category corresponding to image 610 as a different category by using text information.

[0159] For example, a specific application corresponding to the category of image 610 can be specified through user settings. After specifying the specific application, electronic device 101 can display a visual object for performing the specific application by using the text information of image 610.

[0160] In the embodiments, the generation of text information based on images and text extracted from images has been described, but the invention is not limited thereto. For example, electronic device 101 can perform a retrieval on the Internet using at least a portion of the text extracted from an image. In this case, electronic device 101 can generate text information that includes additional information not included in the image based on the retrieval on the Internet.

[0161] Figure 7 This is a diagram illustrating an example of a screen displayed by an electronic device according to an embodiment. Figure 7 The screen 730 shown can be via Figures 2a to 6 The screen of the display 260 of the electronic device 101.

[0162] In the embodiments, it has been described that applications corresponding to the categories of images (e.g., images 210, 310, 410, and 510) are designated 1:1, but this is not a limitation. For example, multiple applications may be designated corresponding to one category.

[0163] For example, although not shown, an image may include an image object representing information about books available for purchase. In this case, the category of the image detected using textual information of the image could be book information. In this case, electronic device 101 can display screen 730, which includes a visual object 731 representing the category of the image, a visual object 732 for executing a web browser application for purchasing the books represented in the image, and a visual object 733 for executing a reading application.

[0164] Figure 8 This is a diagram illustrating an example of a screen displayed by an electronic device according to an embodiment. Figure 8 The screen 830 shown can be via Figures 2a to 6 The screen of the display 260 of the electronic device 101.

[0165] In the embodiments, it has been described that the categories corresponding to the images (such as images 210, 310, 410, and 510) are assigned 1:1, but this is not a limitation. For example, multiple categories may be assigned to an image.

[0166] For example, although not shown, images may include coupon images such as image 210 and images of specific purchasable products such as image 310. In this case, the image category detected using text information of the image could be coupons and shopping. In this case, electronic device 101 can display screen 830 including visual object 832 for performing a wallet application corresponding to the coupon category and visual object 833 for performing a reminder application corresponding to the shopping category.

[0167] In an embodiment, electronic device 101 may display another visual object (not shown) for storing text information of an image in multiple applications. For example, electronic device 101 may, in response to input on another visual object, store the image and / or the corresponding text information in a wallet application and an alert application, or display a screen for storing them.

[0168] Figure 9 An example of a signal flow for a method according to an embodiment for providing functionality associated with image-generated text information is shown.

[0169] Figure 9 The first application 901, image analysis model 902, text information extraction model 903, and second application 904 shown can be stored in [the following text is missing from the original] Figure 1 The memory of electronic device 101 (e.g., Figure 1 The memory 130 is used for execution. The operations of the first application 901, the image analysis model 902, the text information extraction model 903, and the second application 904 can be executed by the processor 120 of the electronic device 101. In an embodiment, the electronic device 101 or the text information extraction model 903 may include... Figure 15 At least one of the blocks in the artificial intelligence system. Alternatively or optionally, the image analysis model 902 may include a server (e.g., external to the electronic device 101) Figure 1 The text information extraction model 903 may be included in a server (e.g., server 108) and executed by the server. Optionally or alternatively, the text information extraction model 903 may be included in a server outside the electronic device 101 (e.g., server 108). Figure 1 The server (108) is in the server and is executed by the server.

[0170] refer to Figure 9 In an embodiment, the first application 901 may include a capture application capable of capturing the screen displayed on the display 260 of the electronic device 101.

[0171] In an embodiment, image analysis model 902 can be configured to obtain text from image objects included in an image. For example, image analysis model 902 can extract text by performing OCR on the input image.

[0172] In this embodiment, a text information extraction model 903 can be trained to generate text information based on text. The text information extraction model 903 can also be trained to generate text information based on images and text.

[0173] In an embodiment, the second application 904 may include at least one application corresponding to the category of the image detected using text information.

[0174] In operation 905, electronic device 101 can use the first application 901 to obtain screenshot images. For example, electronic device 101 can use the first application 901 to obtain image 210. For example, electronic device 101 can use the first application 901 to obtain image 310. For example, electronic device 101 can use the first application 901 to obtain image 410. For example, electronic device 101 can use the first application 901 to obtain image 510.

[0175] In operation 910, electronic device 101 can send a screenshot image from first application 901 to image analysis model 902. For example, electronic device 101 can perform operation 910 based on user input related to the screenshot image. For example, electronic device 101 can use first application 901 to obtain a screenshot image and send the obtained screenshot image from first application 901 to image analysis model 902 based on received user input for capturing the screen displayed through display 260. For example, electronic device 101 can send image 210 from first application 901 to image analysis model 902 based on input of visual object 206 on screen 201.

[0176] For example, electronic device 101 can send image 310 from first application 901 to image analysis model 902 based on input to visual object 306 on screen 301. For example, electronic device 101 can send image 410 from first application 901 to image analysis model 902 based on input to visual object 406 on screen 401. For example, electronic device 101 can send image 510 from first application 901 to image analysis model 902 based on input to visual object 506 on screen 501.

[0177] In operation 915, electronic device 101 can extract text from screenshot images using image analysis model 902. For example, electronic device 101 can extract text corresponding to image objects 211, 212, and 216 in image 210. For example, electronic device 101 can extract text corresponding to image objects 312, 314, and 316 in image 310. For example, electronic device 101 can extract text corresponding to image objects 412, 414, and 416 in image 410. For example, electronic device 101 can extract text corresponding to image objects 512, 514, 516, and 517 in image 510. In operation 920, electronic device 101 can send the extracted text from image analysis model 902 to first application 901. For example, electronic device 101 can send text extracted from image objects 211, 212, and 216 in image 210 from image analysis model 902 to first application 901. For example, electronic device 101 can send text extracted from image objects 312, 314, and 316 in image 310 from image analysis model 902 to first application 901. For example, electronic device 101 can send text extracted from image objects 412, 414, and 416 in image 410 from image analysis model 902 to first application 901. For example, electronic device 101 can send text extracted from image objects 512, 514, 516, and 517 in image 510 from image analysis model 902 to first application 901.

[0178] In operation 930, electronic device 101 can send the extracted text from first application 901 to text information extraction model 903. Additionally, electronic device 101 can send a screenshot image and / or prompt from operation 910 along with the text extracted from first application 901 to text information extraction model 903. For example, electronic device 101 can send a prompt containing the extracted text from first application 901 to text information extraction model 903.

[0179] In operation 935, electronic device 101 can generate text information using text information extraction model 903. The text information may include category information, content information, and overview information.

[0180] For example, textual information about image 210 may indicate that image 210 belongs to the "coupon" category and may include information about at least one of the following: the coupon provider's brand, coupon code (e.g., coupon identification number or barcode number), the content offered by the coupon, start date and time, expiry date and time, usage method and restrictions.

[0181] For example, text information about image 310 may indicate that image 310 belongs to the "location" category and may include information about at least one of the location's name, opening hours, closing hours, time zone, telephone number, and address.

[0182] For example, text information about image 410 may indicate that image 410 belongs to the "shopping" category and may include information about at least one of the following: brand name, product name, cost, supply price, promotional content, and purchase destination for a particular product.

[0183] For example, the text information about image 410 may indicate that image 410 belongs to the “Schedule” category and may include information about at least one of the following: title, start date, start time, start time zone, end date, end time, end time zone, start location, end location, and booking location.

[0184] In the embodiments, it has been described that the electronic device 101 obtains all text information from the text information extraction model 903, which is input with a prompt containing embedded extracted text, but is not limited thereto. For example, the electronic device 101 may send a first prompt for generating category information from the text extracted from the first application 901 to the text information extraction model 903. Thereafter, the electronic device 101 may send the category information from the text information extraction model 903, which is input with the first prompt, to the first application 901. The electronic device 101 may also send a second prompt for generating content information corresponding to the category information from the text extracted from the first application 901 to the text information extraction model 903.

[0185] In operation 940, electronic device 101 can send text information generated from text information extraction model 903 to first application 901.

[0186] In operation 950, electronic device 101 may send the generated text information from first application 901 to text information extraction model 903. For example, electronic device 101 may use first application 901 to display visual objects (e.g., visual objects 223, 323, 423, and 523) for performing at least one application corresponding to the category of the image recognized using the generated text information. Electronic device 101 may perform operation 950 based on input to the visual objects (e.g., visual objects 223, 323, 423, and 523) for performing at least one application.

[0187] In operation 960, electronic device 101 can execute a second application 904 based on text information. For example, the second application 904 corresponding to image 210 may include a wallet application. Electronic device 101 can execute the wallet application based on the generated text information about image 210. For example, electronic device 101 can display an execution screen 240 for a coupon application that stores coupons corresponding to image 210. At least a portion of the text information generated for image 210 can be input into the input field of the coupon application execution screen 240.

[0188] For example, the second application 904 corresponding to image 310 may include a map application. Electronic device 101 may execute the map application based on generated text information about image 310. For example, electronic device 101 may display an execution screen 340 for the map application, providing information about locations corresponding to image 310. On the execution screen 340 of the map application, paths to locations corresponding to image 310 may be displayed.

[0189] For example, the second application 904 corresponding to image 410 may include a reminder application. Electronic device 101 may execute the reminder application based on generated text information about image 410. For example, electronic device 101 may display an execution screen 440 of the reminder application for registering shopping information corresponding to image 410 as a reminder. At least a portion of the text information generated for image 410 may be input into the input field of the execution screen 440 of the reminder application.

[0190] For example, the second application 904 corresponding to image 510 may include a scheduling application. Electronic device 101 can execute the scheduling application based on the generated text information about image 510. For example, electronic device 101 may display an execution screen 540 for the scheduling application used to register the schedule corresponding to image 510. At least a portion of the text information generated for image 510 can be input into the input fields of the execution screen 540 of the scheduling application.

[0191] Figure 10 This is a flowchart illustrating a method for providing functionality associated with image-generated text information according to an embodiment. Figure 10 The operations shown can be performed by Figure 1 The electronic device 101 performs the execution. For example, the processor 120 of the electronic device 101 can perform the execution. Figure 10 The operation. Stored in memory (e.g., Figure 1 Instructions in memory 130 can be executed by electronic device 101 when executed by processor 120. Figure 10 The operation.

[0192] refer to Figure 10In operation 1010, electronic device 101 may display images (e.g., images 210, 310, 410 and 510) on a display.

[0193] In operation 1020, electronic device 101 can obtain text information corresponding to the image based on input used to analyze the image being displayed (e.g., input for objects 206, 306, 406, and 506). The text information may include category information, content information, and summary information about the content of the image.

[0194] In operation 1030, based on checking the application corresponding to the image using text information, electronic device 101 can display visual objects (e.g., visual objects 233, 333, 433, and 533) for performing the application. For example, electronic device 101 can display windows including visual objects (e.g., screens 230, 330, 430, or 530). For example, electronic device 101 can display visual objects or windows including them together with images. For example, electronic device 101 can display visual objects or windows including them by partially overlapping them with images, but is not limited to this.

[0195] In operation 1040, electronic device 101 may display an application execution screen (e.g., screen 240, 340, 440, or 540) in response to input associated with a visual object.

[0196] Figure 11 An example of an exemplary screen of an electronic device according to an embodiment is shown. Figure 11 The electronic device 101 may be Figure 1 Examples of electronic devices 101. For example, display 260 may include... Figure 1 The display module 160.

[0197] Referring to Example 1150, electronic device 101 can display a screen 1101 including image 1110 via display 260. The descriptions of image 210 and screen 201 can be applied substantially the same to the descriptions of image 1110 and screen 1101.

[0198] In one embodiment, the electronic device 101 may display a screen 1130 that partially overlaps with the screen 1101 and includes a visual object 1123 via a display 260. In another embodiment, the descriptions of the visual object 223 and the screen 230 may be applied substantially the same to the descriptions of the visual object 1123 and the screen 1130.

[0199] In an embodiment, the electronic device 101 may receive input to a portion 1125 of the screen 1130. For example, the portion 1125 may include a region within the screen 1130, excluding the region corresponding to the visual object 1123. As a non-limiting example, the input may include input based on... Figure 11 The icon is a sliding input that moves to the left or right.

[0200] Referring to Example 1151, electronic device 101 can control display 260 such that in response to input to a portion of area 1125, screen 1130 changes to screen 1140.

[0201] In an embodiment, screen 1140 may include selectable objects 1131 and may include visual objects 1141 for storing information corresponding to the selected objects (e.g., tags) and for classifying images in association with image 1110. Tags may be included in text information corresponding to image 1110.

[0202] In an embodiment, in response to receiving input for visual object 1141, electronic device 101 may store tags corresponding to active objects in optional objects 1131 in association with image 1110. For example, electronic device 101 may store tags in association with a gallery application for accessing image 1110. In this case, images with stored tags can be retrieved or categorized in the gallery application. For example, electronic device 101 may store tags in the metadata of image 1110.

[0203] Figure 12a and Figure 12b An example of an exemplary screen of an electronic device according to an embodiment is shown. Figure 12a and Figure 12b The electronic device 101 may be Figure 1 Examples of electronic devices 101. For example, display 260 may include... Figure 1 The display module 160.

[0204] exist Figure 12a and Figure 12b The image shows screens 1210 and 1220 displayed on a display 260 of an electronic device 101. Screens 1210 and 1220 may include, for example, execution screens for an application used to manage images and text information corresponding to the images. As a non-limiting example, the application may include a first application 901.

[0205] refer to Figure 12aIn embodiments, screen 1210 may represent a list of images categorized for each category. For example, screen 1210 may include an image corresponding to a coupon category (e.g., image 1211) and visual objects based on at least a portion of text information on the image (e.g., object 1212 representing information about a coupon corresponding to the image and object 1213 representing the coupon's expiration date). For example, screen 1210 may include an image corresponding to a schedule category (e.g., image 1221) and visual objects based on at least a portion of text information on the image (e.g., object 1222 representing schedule information corresponding to the image and object 1223 representing the remaining time until the schedule). In embodiments, determining the category assigned to each image based on text information has been described, but is not limited thereto. For example, the category of an image whose category has not yet been determined may be determined by user settings, or the category of an image determined based on text information may be changed. Subsequently, when determining the category of an image using text information, electronic device 101 may reflect the details of the user setting or changing the category. For example, based on a specific image (e.g., Figure 2a Based on the text information of image 210, electronic device 101 can determine the category of a specific image as a first category (e.g., coupon). Subsequently, the user can change the category of the specific image to a second category different from the first category (e.g., schedule). Afterwards, electronic device 101 can determine the category of images that are the same as or similar to the specific image as the second category.

[0206] In an embodiment, electronic device 101 can provide notifications based on image-based text information using an application corresponding to screen 1210. The notification function can be provided at least in part based on user behavior information stored in electronic device 101, analyzed by electronic device 101, or detected by electronic device 101. It can be achieved by analyzing information recorded in various applications of electronic device 101 (e.g., wallet, calendar, etc.) and / or by using sensors of electronic device 101 (e.g., [missing information]). Figure 1 The electronic device 101 obtains user behavior information by detecting information from the sensor module 176, but is not limited to this. For example, if the expiration date corresponding to an image is approaching, the electronic device 101 can provide a notification to the user. For example, the image is a coupon, and the user can primarily use the coupon on weekends. By analyzing this user behavior information, the electronic device 101 can provide a notification about the coupon's expiration date on the weekend before the expiration date, rather than on the day before the coupon's expiration date. As another example, it can detect that the location of the electronic device 101 is adjacent to a location corresponding to the image (e.g., the location where the coupon is used or a scheduled location), and can provide this as a notification.

[0207] exist Figure 12aThe illustration shows an example of images categorized for each category, but is not limited to this. For example, electronic device 101 can identify common text information from multiple images corresponding to the same and / or different categories. Electronic device 101 can classify images with common text information and display them as new categories or subcategories. For example, electronic device 101 can classify images expiring within a specific time period (e.g., a day, a week, or a month) as subcategories. For example, electronic device 101 can classify images available at a specific location (e.g., a specific store) as subcategories. Therefore, users can conveniently manage and use coupons.

[0208] refer to Figure 12b Screen 1220 may include a visual object 1230 for viewing a categorized screenshot. For example, electronic device 101 may display a visual object 1230 in response to input to the visual object 1230. Figure 12a Screen 1210. In another example, in response to visual object 1230, an icon for displaying screen 1210 can be generated on the main screen of electronic device 101.

[0209] Figure 13 An example of an exemplary screen of an electronic device according to an embodiment is shown. Figure 13 The electronic device 101 may be Figure 1 Examples of electronic devices 101. For example, display 260 may include... Figure 1 The display module 160.

[0210] refer to Figure 13 The electronic device 101 can display a screen 1301 including an image 1310 via a display 260. The descriptions of the image 210 and the screen 201 can be applied substantially the same to the descriptions of the image 1310 and the screen 1301.

[0211] In one embodiment, electronic device 101 can display screen 1305 on screen 1301 via partial overlap with display 260. In another embodiment, screen 1305 may include a visual object 1306 for notifying that text is being analyzed.

[0212] Let's refer to each other. Figure 2b Screen 1305 can be displayed before screens 220 and 230. (See also: [link to relevant documentation]). Figure 2a In one embodiment, the electronic device 101 may display screen 1305 in response to input from visual object 206 for analyzing image 210. In another embodiment, the electronic device 101 may control display 260 to change screen 1305 and display it as screen 220 based on obtained text information corresponding to image 210.

[0213] Figure 14a , Figure 14b , Figure 14c , Figure 14d and Figure 14e An example of a screen displayed by an electronic device according to an embodiment is shown. Figure 14a , Figure 14b , Figure 14c , Figure 14d and Figure 14e The image shows screens 1401, 1402, 1403, 1404, 1405, 1406, 1407, 1408 and 1409 displayed on a display 260 of an electronic device 101.

[0214] refer to Figure 14a Electronic device 101 can display a first screen 1401 via display 260. The first screen 1401 may include an execution screen for playing, managing, and editing images and videos stored in electronic device 101 (e.g., memory) (e.g., a gallery application). The first screen 1401 may include an image 1410. Image 1410 may be an image pre-stored in the memory of electronic device 101. For example, image 1410 selected by a user from a plurality of images stored in electronic device 101 may be displayed on the first screen 1401. In an embodiment, image 1410 may include a coupon image. Image 1410 may include multiple image objects. For example, image 1410 may include multiple image objects 1412, 1414, 1416, 1417, and 1418. Objects 1412, 1414, 1417, and 1418 may include objects corresponding to text, and object 1416 may include objects corresponding to non-text. Object 1412 (such as E STORE) may indicate the location where the coupon is issued or used. Additionally, object 1412 may indicate the value of the offer provided by the coupon, such as "1000". Object 1414 (such as "A gift from my heart") may indicate a characteristic of the coupon (e.g., a gift). Object 1414, such as "www.giftURL.com", may indicate the website address of the coupon's issuing or usage location. Object 1416 may indicate a barcode image of the coupon, and object 1417 may indicate the coupon's barcode number. Object 1418 may include information such as the name of the product offered by the coupon, the location of use, and the expiration date. However, the information included in image 1410 is not limited to the examples above.

[0215] In one embodiment, the electronic device 101 may display a second screen 1402 via a display 260 after displaying the first screen 1401. The second screen 1402 may include an image 1410 and a visual object 1422.

[0216] In an embodiment, the visual object 1422 may be an object used to provide character recognition results for image 1410. For example, electronic device 101 may receive user input to visual object 1422. Electronic device 101 may display character recognition results for image 1410 in response to user input to visual object 1422. For example, electronic device 101 may display an indicator of character recognition results on image 1410. Figure 14c The fifth screen 1405. Character recognition of image objects included in image 1410 can be performed, for example, in response to user input to visual object 1422. Alternatively, regardless of the input to visual object 1422, character recognition of image objects contained in image 1410 can be performed in response to displaying the first screen 1401 including image 1410. In an embodiment, electronic device 101 can extract text included in image 1410 by performing character recognition on image 1410.

[0217] refer to Figure 14b According to an embodiment, the electronic device 101 can display a third screen 1403 via a display 260. The third screen 1403 may include captured images from the first screen 1401 or the second screen 1402. For example, the third screen 1403 may include an image 1410-1 that is substantially the same as the image 1410 displayed on the first screen 1401 or the second screen 1402. For example, image 1410-1 may include visual objects 1412-1, 1414-1, 1416-1, 1417-1, and 1418-1 that are substantially the same as visual objects 1412, 1414, 1416, 1417, and 1418 of image 1410 on the first screen 1401 or the second screen 1402.

[0218] In an embodiment, when the third screen 1403 is a captured image of the second screen 1402, the third screen 1403 may include an image object 1432 corresponding to the visual object 1422 of the second screen 1402. When the third screen 1403 is a captured image of the first screen 1401, unlike the illustration, the third screen 1403 may not include the image object 1432 corresponding to the visual object 1422 of the second screen 1402. In an embodiment, the third screen 1403 may include a menu 1434 for editing or sharing the captured image 1410-1 or for generating labels on the image 1410-1.

[0219] In an embodiment, after displaying the third screen 1403, the electronic device 101 can display a fourth screen 1404 via the display 260. The fourth screen 1404 may include an image 1410-1, a visual object 1442, and a visual object 1444. In an embodiment, visual objects 1442 and 1444 are shown overlapping on the image 1410-1 within the fourth screen 1404, but this is not a limitation. For example, visual objects 1442 and / or 1444 may be displayed in an area of ​​the fourth screen 1404 separate from the image 1410-1.

[0220] In this embodiment, visual object 1444 (e.g., Figure 2a The visual object 206 can be an object used to provide character recognition results for image 1410-1. For example, electronic device 101 can receive user input to visual object 1444. Electronic device 101 can display the character recognition results of image 1410-1 in response to user input to visual object 1444. For example, electronic device 101 can display an indicator of the character recognition results on image 1410-1. Figure 14c The fifth screen 1405. Character recognition of image objects included in image 1410-1 can be performed, for example, in response to user input on visual object 1444. Alternatively, regardless of the input on visual object 1444, character recognition of image objects contained in image 1410-1 can be performed in response to displaying a third screen 1403 or a fourth screen 1404 including image 1410-1. In an embodiment, electronic device 101 can extract text included in image 1410-1 by performing character recognition on image 1410-1.

[0221] In an embodiment, electronic device 101 may use text extracted from image 1410-1 to obtain text information including the category, content, and summary of the content of image 1410-1. For example, electronic device 101 may obtain text information based on receiving user input to visual object 1422 on second screen 1402 or by obtaining image 1410-1 (e.g., obtaining a captured image of first screen 1401 or second screen 1402).

[0222] In this embodiment, visual object 1442 (e.g., Figure 2bThe visual object 223 can be an object for performing an application corresponding to the category of image 1410-1. For example, electronic device 101 can determine an application corresponding to the category of image 1410-1 and display a visual object 1442 for performing the determined application using text information of image 1410-1. In an embodiment, electronic device 101 can perform an application corresponding to the category of image 1410-1 in response to receiving user input on visual object 1442. For example, electronic device 101 can display an execution screen (e.g., [image description missing]) for the application corresponding to the category of image 1410-1 in response to receiving user input on visual object 1442. Figure 14d The first execution screen 1406). The application may include, for example, a payment application or wallet application capable of storing, managing, and using coupons corresponding to image 1410-1.

[0223] refer to Figure 14c In one embodiment, the electronic device 101 may display a fifth screen 1405 via a display 260. For example, the electronic device 101 may display the fifth screen 1405 in response to user input to a visual object 1422 on the second screen 1402 or a visual object 1444 on the fourth screen 1404.

[0224] In an embodiment, the fifth screen 1405 may include image 1410-2, in which text included in image objects included in image 1410 or image 1410-1 is highlighted. For example, electronic device 101 may use text extracted from image 1410 or image 1410-1 to display image 1410-2. For example, in image 1410-2, text 1452 included in image object 1412 or image object 1412-1, text 1454 included in image object 1414 or image object 1414-1, text 1457 included in image object 1417 or image object 1417-1, and text 1458 included in image object 1418 or image object 1418-1 may be highlighted. In the example shown, an example is shown where the text in image 1410-2 is displayed brighter than the surrounding environment, but this is not a limitation of the example shown. For example, visual objects corresponding to non-text may be displayed as blurred.

[0225] In an embodiment, the fifth screen 1405 may include visual objects 1451, 1453, 1455, and 1456. For example, visual objects 1451, 1453, and 1455 may be displayed in an area within the fifth screen 1405 separate from the image 1410-2. For example, the image 1410-2 being displayed may be scaled down and displayed at the top of the fifth screen 1405, and visual objects 1451, 1453, and 1455 may be displayed at the bottom of the fifth screen 1405, but are not limited thereto.

[0226] In one embodiment, visual object 1451 (e.g., visual object 1442) may be used to perform a payment application corresponding to the category of image 1410-2 (e.g., image 1410 or image 1410-1). In another embodiment, visual object 1456 may indicate that text in image 1410-2 can be selected.

[0227] In an embodiment, visual object 1453 can be used to execute an application different from the application executed by visual object 1451, such as a note application. In an embodiment, electronic device 101 can execute the note application based on user input to visual object 1453, based on text selected in images 1410-2. For example, although not shown, electronic device 101 can display notes (or pages) based on user input to visual object 1453, with the text selected in images 1410-2 as the execution screen for the note application.

[0228] In one embodiment, visual object 1455 can be used to copy all text in images 1410-2. In another embodiment, in response to user input to visual object 1455, electronic device 101 can copy all text in images 1410-2 (or store it temporarily in an area of ​​memory of electronic device 101, such as the clipboard) and display visual object 1459. Visual object 1459 can indicate an action that can be performed by input to visual object 1455, such as “Copied. You may paste it here or to a nearby mobile phone.” Visual object 1459 can display a specified time (e.g., a few seconds).

[0229] refer to Figure 14a , Figure 14b and Figure 14dAccording to the embodiment, the electronic device 101 can display a first execution screen 1406 of a payment application corresponding to the category of image 1410 or image 1410-1. For example, the first execution screen 1406 may include a loading screen for the payment application. For example, the first execution screen 1406 may include an icon 1461 of the payment application.

[0230] In an embodiment, electronic device 101 can execute a payment application based on user input to visual object 1442 of fourth screen 1404, based on text recognized in image 1410-1 of fourth screen 1404. For example, electronic device 101 can display a first execution screen 1406 of the payment application in response to user input to visual object 1442 of fourth screen 1404. After displaying the first execution screen 1406, electronic device 101 can display image 1480, including at least a portion of image 1410-1, and a third execution screen 1408 in a state where the text recognized in image 1410-1 is used as the execution screen for a payment application that stores coupons corresponding to image 1410-1. For example, in input field 1486 of third execution screen 1408, a product name based on text recognized in image object 1418 can be entered. For example, in input field 1487, a "barcode" as the coupon type of image 1410-1 can be entered. For example, in input field 1488, a barcode number based on text identified in image object 1417 can be entered.

[0231] In an embodiment, electronic device 101 can execute a payment application based on text selected from text recognized in images 1410-2 (e.g., images 1410 or 1410-1) on the fifth screen 1405, based on user input to visual object 1451 on the fifth screen 1405. For example, electronic device 101 can display a first execution screen 1406 for the payment application in response to user input to visual object 1451 on the fifth screen 1405. After displaying the first execution screen 1406, electronic device 101 can display an image 1480 including at least a portion of image 1410-1 and a third execution screen 1408 in a state where the text selected in image 1410-2 is used as the execution screen for an application to store coupons corresponding to image 1410-2. For example, the product name in text 1458 can be entered into input field 1486 of the third execution screen 1408. For example, a "barcode" of the coupon type of image 1410-1 can be entered into input field 1487. For example, in input field 1488, you can enter the same barcode number as text 1457.

[0232] Additionally or optionally, the electronic device 101 may display a second execution screen 1407 after displaying the first execution screen 1406 and before displaying the third execution screen 1408. The second execution screen 1407 may include a visual object 1473 (e.g., a cropping tool) for specifying a region of the image 1410-1 to be attached to the third execution screen 1408, and a visual object 1471 for completing the region specification of the image 1410-1. For example, a user may specify a specific region of the image 1410-1 within the visual object 1473 by changing the size of the boundaries of the visual object 1473. The visual object 1473 is shown in a quadrilateral shape, but is not limited thereto. In an embodiment, the electronic device 101 may, in response to user input on the visual object 1471, display the third execution screen 1408 to which the image 1480, specified by the visual object 1473, corresponds to a specific region of the image 1410-1, is attached.

[0233] In an embodiment, the third execution screen 1408 may include a visual object 1481 for storing input coupon information and a visual object 1483 for canceling the stored coupon information.

[0234] In an embodiment, electronic device 101 can store coupon information input via a third execution screen 1408 based on input to a visual object 1481, and display a fourth execution screen 1409 indicating the stored coupon information. The fourth execution screen 1409 may include, but is not limited to, coupon information such as, for example, the coupon name, the product name offered by the coupon, the brand of the coupon provider, the coupon code (e.g., the coupon identification number or barcode number), the content offered by the coupon, the start date and time, the expiration date and time, the method of use, and restrictions. The fourth execution screen 1409 may indicate the status information of the stored coupon, such as whether it has expired.

[0235] Figure 15 This is an exemplary block diagram illustrating a generative artificial intelligence system according to an embodiment. Figure 15 The image illustrates an exemplary artificial intelligence system including a user query / response interface 1510, an AI framework 1520, a generative AI model 1530, a knowledge base 1540, and an application / service component 1550. According to embodiments, electronic devices (e.g., Figure 1 The processor of electronic device 101 (e.g., Figure 1 The processor 120) can execute by Figure 15 Each block is a separate program that performs functions related to generative artificial intelligence.

[0236] In this embodiment, the user query / response interface 1510 can receive user input. User input can be in the form of natural language, images, and / or video. Additionally, contextual information can be sent along with the user input. Contextual information can include various supplementary information provided during user input. For example, contextual information can include information about the application the user is currently using or information about the user's location. User input can also be a mixture of the aforementioned natural language, images, sound, and contextual information. Furthermore, user input can be in the form of non-natural language, such as a selection menu. The user query / response interface 1510 can output the results of the generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of a user-requested action.

[0237] The AI ​​framework 1520 can receive user input and can coordinate and control each component needed to execute user intent based on user queries. The AI ​​framework 1520 may include a prompt design component 1521, an API / plugin management component 1522, and a refinement component 1523.

[0238] User input received from the user query / response interface 1510 can be sent to the prompt design component 1521. The prompt design component 1521 can be used to generate prompts suitable for inputting user input into an LLM or a large multimodal model (LMM). The prompt design component 1521 can be an AI component that uses machine learning algorithms or neural networks to develop better prompts over time. Based on user input, the prompt design component 1521 can generate prompts by accessing a knowledge base 1540 that includes user preference data, a prompt library, and prompt examples, and the generated prompts can be passed to the LLM or LMM.

[0239] When a request for additional information is received, the API / plugin management component 1522 can perform the role of communicating with external information, while simultaneously passing user input as input to the generative model. The API / plugin management component 1522 can establish a channel for external communication with the AI ​​framework 1520 via an application programming interface (API), and can allow access to various data sources through the established channel. Additionally, when a request for additional information is needed from the application / service component 1550 (e.g., ...), Figure 1When performing an action in program 140 or application 146 to ultimately execute user input rather than as an intermediate result, the API / plugin management component 1522 can request the corresponding action via API. Information obtained from external sources can be used with user input to generate prompts in the prompt design component 1521, or it can be passed as input to the generative model.

[0240] The refining component 1523 (or output modification component) can fine-tune the results output from the generative AI model 1530. For example, the refining component 1523 can verify whether the content generated by LLM and / or LMM is irrelevant, contains biased content, or contains harmful content. Furthermore, the refining component 1523 can determine how well the output matches the user's expected results, and can perform additional processing if necessary. The refining component 1523 can be additionally configured with hints to avoid unwanted output and can provide these hints to the user.

[0241] Generative AI models 1530 can generally refer to artificial intelligence neural networks that generate new forms of data based on user input information. Generative AI models 1530 can include models that generate images and / or models that generate language. Examples of image-generating models typically include generative adversarial networks (GANs) and variational autoencoders (VAEs), and can also include diffusion-based generative models using VAEs and transformer architectures. Language-generating models can be models trained to output statistically optimal output values ​​based on input values, and representative examples can include models such as CHAT-GPT 3 and CHAT-GPT 4. Furthermore, there may be large-scale multimodal models (LMMs) capable of recognizing various forms of data input (such as text, images, and speech) and generating corresponding new data.

[0242] An electronic device according to an embodiment (e.g., Figure 1 The electronic device 101 may include a display (e.g., Figure 1 The display module 160), and the memory for storing instructions (e.g., Figure 1 The memory 130) and the processor (e.g., Figure 1The processor 120 has instructions that, when executed by the processor, cause the electronic device to display a still image comprising one or more image objects corresponding to text on a display. The instructions, when executed by the processor, cause the electronic device to: obtain text information corresponding to one or more image objects of the still image based on input for analyzing the still image being displayed. When executed by the processor, the instructions cause the electronic device to select a specified application corresponding to the still image from multiple applications using at least a portion of the text information. When executed by the processor, the instructions cause the electronic device to display a screen comprising a visual object for executing the specified application and the still image on a display. When executed by the processor, the instructions cause the electronic device to display an execution screen for the specified application on a display in response to input associated with the visual object.

[0243] In one embodiment, the instructions, when executed by a processor, can cause an electronic device to perform a function for a specified application of a still image in response to input to a visual object.

[0244] In one embodiment, when the processor executes instructions, the instructions may cause the electronic device to store at least some text or another text associated with at least some text in relation to a specified application.

[0245] In one embodiment, the instructions, when executed by the processor, can enable the electronic device to provide notifications via a specified application based on at least some text.

[0246] In one embodiment, when the processor executes instructions, the instructions may cause the electronic device to classify still images into one or more of a plurality of categories, at least in part, based on textual information.

[0247] In one embodiment, the electronic device may include communication circuitry. When executed by a processor, the instructions may cause the electronic device to obtain, via the communication circuitry, additional information retrieved from an external server, at least in part, based on text information. When executed by the processor, the instructions may cause the electronic device to store the additional information in association with at least some text of the text information.

[0248] In an embodiment, the still image may also include another image object corresponding to non-text. When executed by a processor, instructions may cause the electronic device to store the image corresponding to the other image object in association with a specified application.

[0249] In an embodiment, another image object may include image code.

[0250] In this embodiment, the text information may include location information. The specified application may include a map application. When executed by a processor, the instructions may cause the electronic device to execute the map application at least in part based on the location information.

[0251] In an embodiment, when the processor executes instructions, the instructions may cause the electronic device to select a first application as a designated application from a plurality of applications, at least in part, based on text information including first specified text. When the processor executes instructions, the instructions may cause the electronic device to select a second application, different from the first application, as a designated application from a plurality of applications, at least in part, based on text information including second specified text different from the first specified text.

[0252] In one embodiment, when executed by a processor, the instructions can cause an electronic device to display a screen that also includes another visual object, which indicates that at least a portion of the still image corresponds to a coupon image or a ticket image.

[0253] In one embodiment, the instructions, when executed by a processor, can cause an electronic device to display a screen that also includes another visual object indicating that at least a portion of a still image corresponds to a map image.

[0254] In one embodiment, the instructions, when executed by the processor, enable the electronic device to obtain text information from a still image based on user input.

[0255] In one embodiment, the instructions, when executed by the processor, enable the electronic device to acquire still images via a capture application.

[0256] In one embodiment, the electronic device may include communication circuitry. When executed by a processor, instructions may cause the electronic device to send a still image to an external server via the communication circuitry. The instructions, when executed by the processor, may also cause the electronic device to receive at least a portion of text information from the external server via the communication circuitry after sending the still image.

[0257] In one embodiment, an image training model for multiple images may be stored in memory. When executed by a processor, instructions may cause an electronic device to generate at least a portion of textual information by applying still images to the image training model.

[0258] In this embodiment, the still image may include a coupon image. Text information may include coupon information included in the coupon image. The designated application may include an application for managing coupons. Instructions, when executed by a processor, may cause the electronic device to respond to input associated with a visual object, performing functions of the designated application at least in part based on the coupon information.

[0259] In an embodiment, the instructions, when executed by a processor, can cause the electronic device to identify whether a specified application is installed on the electronic device. When executed by a processor, the instructions can cause the electronic device to display a visual object for executing the specified application, along with a still image, if the specified application is installed on the electronic device. When executed by a processor, the instructions can cause the electronic device to display another visual object and a still image for executing another application, which is used to install the specified application, if the specified application is not installed on the electronic device.

[0260] An electronic device according to an embodiment (e.g., Figure 1 The electronic device 101 may include a display (e.g., Figure 1 The display module 160), and the memory for storing instructions (e.g., Figure 1 The memory 130) and the processor (e.g., Figure 1 The processor 120 has instructions that, when executed by the processor, cause the electronic device to display a screenshot image on a display. The instructions, when executed by the processor, cause the electronic device to: obtain text information corresponding to the screenshot image based on input for analyzing the screenshot image being displayed; and, when executed by the processor, cause the electronic device to: identify the application corresponding to the screenshot image using the text information, and display a window including a visual object for executing the application on the display, partially overlapping the screenshot image. The instructions, when executed by the processor, cause the electronic device to: identify the application corresponding to the screenshot image using the text information, and, in response to input related to the visual object, display the application's execution screen on the display.

[0261] In one embodiment, the instructions, when executed by a processor, enable the electronic device to extract text corresponding to image objects included in a screenshot. When executed by a processor, the instructions enable the electronic device to obtain text information from an artificial intelligence model that has been input with the extracted text.

[0262] In one embodiment, the screenshot image may include another image object corresponding to non-text. When executed by a processor, the instructions enable the electronic device to obtain text information from an artificial intelligence model that has been given input of another image object and extracted text.

[0263] In an embodiment, the electronic device may include communication circuitry (e.g., Figure 1 The communication module 190). When executed by the processor, the instructions can cause the electronic device to extract text corresponding to image objects included in a screenshot. When executed by the processor, the instructions can cause the electronic device to send the extracted text to a server via communication circuitry (e.g., Figure 1Server 108). When executed by the processor, the instructions enable the electronic device to receive text information from the server via communication circuitry after sending the extracted text.

[0264] In one embodiment, the electronic device may include communication circuitry. When executed by a processor, the instructions may cause the electronic device to send a screenshot image to a server via the communication circuitry in response to input for analyzing the screenshot image. When executed by a processor, the instructions may also cause the electronic device to receive text information from the server via the communication circuitry after sending the screenshot image.

[0265] In this embodiment, the text information may include category information of the screenshot image, content information, and summary information about the content information.

[0266] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to display a window on a screen that also includes objects representing category information.

[0267] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to display a window on a screen that also includes objects representing overview information.

[0268] In one embodiment, the instructions, when executed by a processor, cause the electronic device to display a second window that partially overlaps with the screenshot image on a display. This second window includes category information and has a first size. When executed by the processor, the instructions cause the electronic device to: in response to input to the second window, display a second window with a size larger than the first size that partially overlaps with the screenshot image on the display.

[0269] In one embodiment, the instructions, when executed by a processor, can cause the electronic device to receive input for capturing a screen displayed on a monitor. When executed by the processor, the instructions can cause the electronic device to obtain a screenshot image in response to the input for capturing the screen.

[0270] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to store information for classifying screenshot images in association with an application that uses text information.

[0271] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to store information using textual information to classify the screenshot image in the screenshot image's metadata.

[0272] In one embodiment, the instructions, when executed by the processor, can enable the electronic device to execute applications based on text information.

[0273] In this embodiment, the screenshot image may include a coupon image. Text information may include coupon information included in the coupon image. The application may include an application for managing coupons. Instructions, when executed by a processor, may cause the electronic device to respond to input associated with a visual object, executing the application at least in part based on the coupon information.

[0274] In this embodiment, the screenshot image may include an image representing a specific area. The text information may include location information corresponding to the image representing the specific area. The application may include a map application. Instructions, when executed by a processor, may cause the electronic device to respond to input associated with a visual object, at least in part based on the location information, to perform the map application.

[0275] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to identify whether an application is installed on the device. When executed by the processor, the instructions can cause the electronic device to display a window including a visual object for executing the application if the application is installed on the device. The instructions, when executed by the processor, can also cause the electronic device to display a window including another visual object for executing another application installed on the device, partially overlapping a screenshot image, if no application is installed on the device.

[0276] In one embodiment, the instructions, when executed by the processor, can cause the electronic device to display a different screenshot image than the original screenshot image on a display. When executed by the processor, the instructions can also cause the electronic device to obtain additional text information corresponding to the other screenshot image based on input used to analyze it. When executed by the processor, the instructions can also cause the electronic device to display a window corresponding to the other screenshot image identified using the additional text information, the window including another visual object for executing a different application than the original application, the window partially overlapping the other screenshot image. When executed by the processor, the instructions can also cause the electronic device to display the execution screen of the other application on a display in response to input associated with the other visual object.

[0277] In this embodiment, a screenshot image may include a coupon image. Another screenshot image may include an image representing a specific area.

[0278] An electronic device according to an embodiment (e.g., Figure 1 The electronic device 101 may include a display (e.g., Figure 1 The display module 160), and the memory for storing instructions (e.g., Figure 1 The memory 130) and the processor (e.g., Figure 1The processor 120 has the following instructions, when executed by the processor, that enable the electronic device to display a first screen, including images, via a display. The instructions, when executed by the processor, enable the electronic device to extract text from the images by performing optical character recognition (OCR). The instructions, when executed by the processor, enable the electronic device to obtain text information corresponding to the images from an artificial intelligence model that has been input to the extracted text. When executed by the processor, the instructions enable the electronic device to use the text information to identify the category of the images. The instructions, when executed by the processor, enable the electronic device to display a second screen, including images and visual objects for executing applications, based on the identification of applications corresponding to the identified categories. The instructions, when executed by the processor, enable the electronic device to display a third screen, different from the second screen, in response to input associated with the visual objects, by executing applications.

[0279] In an embodiment, the electronic device may include communication circuitry (e.g., Figure 1 The communication module 190). When executed by the processor, the instructions can cause the electronic device to send the extracted text to a server including an artificial intelligence model (e.g., communication module 190) via communication circuitry. Figure 1 (Server 108). When executed by the processor, the instructions enable the electronic device to receive text information from the server via communication circuitry.

[0280] The electronic device according to various embodiments can be one of a variety of types of electronic devices. The electronic device may include, for example, a portable communication device (e.g., a smartphone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. According to embodiments of this disclosure, the electronic device is not limited to those described above.

[0281] It should be understood that the various embodiments of this disclosure and the terminology used therein are not intended to limit the technical features set forth herein to the specific embodiments, but rather to include various changes, equivalents, or substitutions to the respective embodiments. Regarding the description of the drawings, similar reference numerals may be used to refer to similar or related elements. It should be understood that, unless the relevant context clearly indicates otherwise, the singular form of the noun corresponding to an item may include one or more things. As used herein, each of the phrases such as “A or B,” “at least one of A and B,” “at least one of A or B,” “A, B, or C,” “at least one of A, B, and C,” and “at least one of A, B, or C” may include any or all possible combinations of the items listed together in the corresponding phrase. As used herein, terms such as “first” and “second” or “first” and “second” may be used simply to distinguish the respective component from another component and do not limit the components in other respects (e.g., importance or order). It will be understood that, whether the terms “operably” or “communically” are used or not, if an element (e.g., a first element) is referred to as being “combined” or “connected” to another element (e.g., a second element), it means that the first element can be directly (e.g., wired) connected to the other element, wirelessly connected to the other element, or connected to the other element via a third element.

[0282] As used in conjunction with various embodiments of this disclosure, the term "module" may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with other terms (e.g., "logic," "logic block," "part," or "circuit"). A module may be a single integrated component adapted to perform one or more functions, or its smallest unit or part. For example, according to an embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0283] The various embodiments set forth herein can be implemented as software (e.g., program 140) containing one or more instructions readable by a machine (e.g., electronic device 101) stored in a storage medium (e.g., internal memory 136 or external memory 138). For example, under the control of a processor, a processor (e.g., processor 120) of the machine (e.g., electronic device 101) can invoke and execute at least one of the one or more instructions stored in the storage medium, with or without the use of one or more other components. This allows the machine to operate to perform at least one function according to the invoked at least one instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. The term "non-transitory" means only that the storage medium is a tangible device and does not include signals (e.g., electromagnetic waves), but this term does not distinguish between cases where data is stored semi-permanently in the storage medium and cases where data is temporarily stored in the storage medium.

[0284] According to embodiments, methods according to various embodiments of this disclosure may be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., an optical disc read-only memory (CD-ROM)) or via an app store (e.g., the Play Store). TM The computer program product may be distributed online (e.g., downloaded or uploaded) or directly between two user devices (e.g., smartphones). If distributed online, at least a portion of the computer program product may be temporarily generated or at least temporarily stored in a machine-readable storage medium, such as the memory of a manufacturer's server, an app store's server, or a relay server.

[0285] According to various embodiments, each of the above components (e.g., a module or program) may include a single entity or multiple entities, and some of the multiple entities may be arranged separately in different components. According to various embodiments, one or more of the above components may be omitted, or one or more other components may be added. Alternatively or additionally, multiple components (e.g., modules or programs) may be integrated into a single component. In this case, according to various embodiments, the integrated component may still perform one or more functions of each of the multiple components in the same or similar manner as the corresponding components in the multiple components before integration. According to various embodiments, operations performed by a module, program, or other component may be performed sequentially, in parallel, repeatedly, or heuristically, or one or more operations may be performed in a different order or omitted, or one or more other operations may be added.

Claims

1. An electronic device, comprising: monitor; Memory that stores instructions; and processor, The instructions, when executed by the processor, cause the electronic device to: The display shows a still image, which includes one or more image objects corresponding to the text. Based on the input used to analyze the still image being displayed, text information corresponding to the one or more image objects of the still image is obtained; Using at least a portion of the text information, select a specific application corresponding to the still image from among multiple applications; The display shows a screen including visual objects for executing the specified application and the still image; as well as In response to input related to the visual object, the execution screen of the specified application is displayed on the display.

2. The electronic device as claimed in claim 1, The instructions, when executed by the processor, cause the electronic device to: In response to input to the visual object, perform the function of the specified application for the still image.

3. The electronic device as described in claim 1, The instructions, when executed by the processor, cause the electronic device to: At least some texts of the text information are stored in association with the specified application, or another text related to the at least some texts.

4. The electronic device as described in claim 3, The instructions, when executed by the processor, cause the electronic device to: Notifications are provided through the designated application based on at least some of the text.

5. The electronic device as claimed in claim 1, The instructions, when executed by the processor, cause the electronic device to: The still image is classified into one or more categories, at least in part, based on the text information.

6. The electronic device as claimed in claim 3, Including communication circuits, The instructions, when executed by the processor, cause the electronic device to: The communication circuit obtains additional information retrieved from an external server, at least in part based on the text information; and The additional information is stored in association with at least some of the text in the text information.

7. The electronic device as claimed in claim 1, The still image mentioned above also includes another image object corresponding to the non-text, and The instructions, when executed by the processor, cause the electronic device to: The image corresponding to the other image object is stored in association with the specified application.

8. The electronic device as claimed in claim 7, The other image object mentioned above includes image code.

9. The electronic device as claimed in claim 1, The text information includes location information, and The specified applications include map applications, and The instructions, when executed by the processor, cause the electronic device to: The map application is performed based at least in part on the location information.

10. The electronic device as claimed in claim 1, characterized in that, The instructions, when executed by the processor, cause the electronic device to: Selecting a first application as the designated application from the plurality of applications, at least in part, based on the text information including the first designated text; and The designated application is selected from the plurality of applications based at least in part on the text information, which includes a second designated text that is different from the first designated text.

11. The electronic device as claimed in claim 1, The instructions, when executed by the processor, cause the electronic device to: The screen is displayed via the monitor, and the screen also includes another visual object indicating that at least a portion of the still image corresponds to a coupon image or a ticket image.

12. The electronic device as claimed in claim 1, The instructions, when executed by the processor, cause the electronic device to: The display also includes a screen with another visual object indicating that at least a portion of the still image corresponds to a map image.

13. The electronic device as claimed in claim 11, The instructions, when executed by the processor, cause the electronic device to: The text information is obtained from the still image based on user input.

14. The electronic device as claimed in claim 1, The instructions, when executed by the processor, cause the electronic device to: The still image is obtained by capturing the application.

15. The electronic device as claimed in claim 1, Including communication circuits, The instructions, when executed by the processor, cause the electronic device to: The still image is sent to an external server via the communication circuit; and After sending the still image, at least a portion of the text information is received from the external server via the communication circuit.

Images