Screen capture method and electronic device therefor

Abstract

Various embodiments of the present invention may comprise a display (160), a processor (120), and a memory (130). When executed by the processor, instructions stored in the memory may instruct the electronic device (101) to: display, on the display, a first execution screen of a first application; display, on the display, a second execution screen of a voice assistant application, the second execution screen being overlaid on a partial area of the first execution screen of the first application; receive a screen capture command through the voice assistant application while the second execution screen is displayed overlaid on the partial area of the first execution screen; and perform screen capture on the basis of the screen capture command while the second execution screen is displayed overlaid on the partial area of the first execution screen, the screen capture being performed by saving an image of the first execution screen, on which an image of the second execution screen is not overlaid, in a partial area of the first execution screen. Various embodiments are possible.

Description

Screen capture method and electronic device thereof

[0001] Various embodiments of the present disclosure relate to a screen capture method and an electronic device thereof.

[0002] With the advancement of digital technology, various types of electronic devices, such as mobile terminals, personal digital assistants (PDAs), electronic notebooks, smartphones, tablet PCs (personal computers), and wearable devices, are becoming widely used. These electronic devices are constantly undergoing improvements in their hardware and / or software to support and enhance their functionality.

[0003] Recently, technologies for understanding language and summarizing or condensing long sentences through large language models (LLMs) have been advancing. Electronic devices acquire data from user queries via LLM servers and provide the resulting data to users. However, the resulting data often contains a vast amount of information, requiring users to navigate through the data to find the information they need.

[0004] In one embodiment, when a screen capture command is received through a voice assistant application, a method and device for analyzing a user's intention to capture the execution screen of a first application, capture the execution screen of a voice assistant application, or capture the entire screen of a display including the execution screen of the first application and the execution screen of the voice assistant application may be disclosed.

[0005] An electronic device (101) according to an embodiment of the present disclosure includes a display (160), a processor (120), and a memory (130), and instructions stored in the memory, when executed by the processor, may cause the electronic device to display a first execution screen of a first application on the display, display a second execution screen of a voice assistant application overlaid on a portion of the first execution screen of the first application on the display, receive a screen capture command through the voice assistant application while the second execution screen is overlaid and displayed on a portion of the first execution screen, and store an image of the first execution screen, in which the image of the second execution screen is not overlaid, on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen, to perform a screen capture.

[0006] According to an embodiment of the present disclosure, an operating method of an electronic device (101) may include an operation of displaying a first execution screen of a first application on a display (160) of the electronic device, an operation of displaying a second execution screen of a voice assistant application overlaid on a portion of the first execution screen of the first application on the display, an operation of receiving a screen capture command through the voice assistant application while the second execution screen is overlaid and displayed on a portion of the first execution screen, and an operation of performing a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen.

[0007] According to one embodiment, the intention of the user's screen capture request can be inferred by determining the correlation between the execution screen of the voice assistant application and the execution screen of the first application.

[0008] According to one embodiment, user convenience can be increased by inferring the intent of the user's screen capture request and capturing the execution screen of the application desired by the user.

[0009] According to one embodiment, in response to a screen capture request through speech, it may be determined whether to capture the screen including the running screen of the voice assistant application or to capture the screen excluding the running screen of the voice assistant application.

[0010] FIG. 1 is a block diagram of an electronic device within a network environment according to one embodiment.

[0011] FIG. 2 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0012] FIGS. 3A and 3B are diagrams illustrating an example of processing a user's screen capture request in an electronic device according to one embodiment.

[0013] Figure 4 is a flowchart illustrating an operating method of an electronic device according to one embodiment.

[0014] FIG. 5A is a diagram illustrating an example of capturing an execution screen of a voice assistant application in an electronic device according to one embodiment.

[0015] FIG. 5b is a diagram illustrating an example of capturing the entire screen of a display in an electronic device according to one embodiment.

[0016] FIG. 5c is a diagram illustrating an example of capturing an execution screen of a first application or an execution screen of a voice assistant application in an electronic device according to one embodiment.

[0017] FIG. 5D is a diagram illustrating an example of capturing the entire screen of a display in an electronic device according to one embodiment.

[0018] FIG. 6 is a flowchart illustrating a method for capturing a screen based on a user's last interaction in an electronic device according to one embodiment.

[0019] FIG. 7A is a diagram illustrating an example of capturing a screen based on a user's last interaction in an electronic device according to one embodiment.

[0020] FIG. 7b is a diagram illustrating an example of capturing a screen based on an association between a voice assistant application and a first application in an electronic device according to one embodiment.

[0021] FIG. 7c is a diagram illustrating an example of capturing an execution screen of a voice assistant application and an execution screen of a first application in an electronic device according to one embodiment.

[0022] FIG. 8 is a flowchart illustrating a method for capturing a screen based on an intention to share a screen capture in an electronic device according to an embodiment.

[0023] FIG. 9A is a diagram illustrating an example of capturing a screen based on an intention to share a screen capture in an electronic device according to one embodiment.

[0024] FIG. 9b is a diagram illustrating an example of capturing a screen by specifying a screen to be captured in an electronic device according to one embodiment.

[0025] FIG. 10 is a flowchart illustrating a method for capturing a screen by identifying the intention of a user who requested a screen capture in an electronic device according to an embodiment.

[0026] FIG. 11 is a diagram illustrating an example of selectively storing a captured screen in an electronic device according to one embodiment.

[0027] FIG. 12 is a diagram illustrating an example of providing a screen captured image through a voice assistant application in an electronic device according to one embodiment.

[0028] FIG. 13 is a diagram illustrating an example of capturing content outside the screen size in an electronic device according to one embodiment.

[0029] FIG. 14 is a diagram illustrating an example of providing an image captured through a voice assistant application in an electronic device according to one embodiment.

[0030] FIG. 15 is a diagram illustrating an example of editing an image captured through a voice assistant application in an electronic device according to one embodiment.

[0031] FIG. 16A is a diagram illustrating an example of capturing a screen through a voice assistant application in an electronic device having various sizes according to one embodiment.

[0032] FIG. 16b is a diagram illustrating an example of capturing a screen through a voice assistant application in an electronic device having various sizes according to one embodiment.

[0033] FIG. 1 is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments.

[0034] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0035] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0036] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0037] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0038] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0039] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0040] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0041] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor for detecting a touch, or a pressure sensor for measuring the strength of a force generated by the touch.

[0042] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0043] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0044] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0045] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0046] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0047] The camera module (180) can capture still images and moving images. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0048] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0049] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0050] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0051] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0052] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0053] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0054] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0055] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0056] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0057] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0058] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0059] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0060] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0061] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0062] FIG. 2 is a block diagram illustrating a generative artificial intelligence system according to one embodiment.

[0063] Referring to FIG. 2, a generative artificial intelligence system (e.g., an intelligent server, server (108) of FIG. 1) according to one embodiment may include a user interface (210), a database (220), an application and service component (230), an AI framework (250), and a generative AI model (270).

[0064] The user interface (210) can receive a user query. The user query can be in the form of natural language, images, or videos. Additionally, context information can be transmitted along with the user query. As another example, the user query can also be a non-natural language input that does not generate natural language, such as a design request or modification. Furthermore, the query can be a mixed form of natural language, images, sounds, and context information as described above. Furthermore, the user interface (210) can output the results of the generative artificial intelligence system to the user. The output can be in the form of natural language or specific content, and can also be provided in the form of an action requested by the user.

[0065] The AI ​​framework (250) can receive a user query and coordinate and control each component necessary to carry out the user's intent. The AI ​​framework (250) may include a prompt design component (251), an application and plug-in management component (APIs / Plugins Management component) (253), and an output modification component (255).

[0066] A user query or action entered in the user interface (210) may be transmitted to a prompt design component (251). The prompt design component (251) may be used to generate prompts suitable for input into a large language model (LLM) or a large multimodal model. The prompt design component (251) may be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (251) may access a knowledge component containing user preference data, a prompt library, and prompt examples to generate prompts and transmit them to the large language model (LLM) or the large multimodal model (LMM).

[0067] The application and plugin management component (253) can communicate with external information when a request for additional information is made when user input is passed as input to the generative model. The application and plugin management component (253) establishes a channel for communication with the AI ​​Interface externally via an application programming interface (API), thereby enabling access to various data sources. Furthermore, if the application or service needs to perform an action that ultimately fulfills a user query, rather than an intermediate result, the application and plugin management component (253) can request such an action via the API. Information obtained from external sources can be passed as input to the generative model along with user input.

[0068] The output modification component (255) can fine-tune the output from the generative model. For example, the output modification component (255) can verify that content generated through a language model (LLM) or a large-scale multimodal model (LMM) is not irrelevant, does not contain biased content, or does not contain harmful content. In addition, the output modification component (255) can determine the degree to which the content matches the user's desired result and, if necessary, can proceed with additional processing. Additionally, the output modification component (255) can configure and provide hints to the user to avoid undesired output.

[0069] Generative AI models (270) generally refer to artificial intelligence neural networks that create new forms of data based on user input information. Representative models that generate images include generative adversarial networks (GANs) and variational autoencoders (VAEs). Recently, diffusion-based generative models that use VAE and transformer structures are also called generative models. In addition, language models are models trained to statistically output the most appropriate output based on input values, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, since they can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them, they are called large multimodal models (LMMs).

[0070] FIGS. 3A and 3B are diagrams illustrating an example of processing a user's screen capture request in an electronic device according to one embodiment.

[0071] Referring to FIG. 3A, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (310) on a display (e.g., the display module (150) of FIG. 1) when a user input requesting screen capture (e.g., a screen capture command) is detected. The user input may be requested through a voice assistant application and may be, for example, a speech. The voice assistant application (or voice assistant service) may collect and provide customized information to the user based on an artificial intelligence (AI) engine and voice recognition, and may perform various tasks such as schedule management, email transmission, and restaurant reservation according to the user's voice commands.

[0072] For example, a user may utter a given voice wake word (e.g., Hi Bixby) or press a button associated with a voice assistant application. The processor (120) may detect the voice wake word acquired through a microphone (e.g., the input module (150) of FIG. 1) and execute the voice assistant application. When a button (e.g., a software button or a physical button) associated with the voice assistant is selected, the processor (120) may execute the voice assistant application. When the button is selected, the processor (120) may display a keypad on the execution screen of the executed voice assistant application. According to one embodiment, when the voice assistant is called with a button, the voice assistant may operate in a text-based manner (e.g., displaying a keypad), or when the voice assistant is called with a voice, the voice assistant may operate in a voice-based manner (e.g., receiving a voice command), but the present invention may not be limited thereto.

[0073] The first user interface (310) may include a first execution screen (311) of the first application and a second execution screen (315) of the voice assistant application on which text (313) corresponding to a user input is displayed. The user input (or screen capture command) may include a user utterance saying 'screen capture' or a text input of the text 'screen capture' entered via a keypad. The processor (120) may display the first user interface (310) when a screen capture is requested via the voice assistant application while displaying the first execution screen (311) of the first application (e.g., a wayfinding application). After the screen capture is completed, the processor (120) may notify the user that the capture is completed via text (e.g., "done") or voice. The second execution screen (315) may be displayed by being partially overlaid on the first execution screen (311). The second execution screen (315) may be displayed as a foreground image in the first layer, and the first execution screen (311) may be displayed as a background image or screen in the second layer. The processor (120) may receive a screen capture command through the voice assistant application while the second execution screen (315) is overlaid and displayed in a portion of the first execution screen (311).

[0074] The second user interface (330) may capture the execution screen of the first application in response to a screen capture request via a voice assistant application. The processor (120) may perform a screen capture by storing an image of the first execution screen (311) based on the screen capture command while the second execution screen (315) is overlaid and displayed on a portion of the first execution screen (311).

[0075] According to one embodiment, the processor (120) may analyze the user intent that requested the screen capture to determine the screen to be captured. The user intent may be a correlation between contents included in two execution screens or an explicit designation of the screen to be captured. The user intent that specifies the screen to be captured may include touching (or currently focusing) the screen to be captured, or uttering (e.g., 'Capture the current screen', 'Capture the voice assistant screen'). As another example, the processor (120) may perform the correlation between the execution screen of the voice assistant application and the first application as part of the user intent analysis. Based on the analyzed user intent, the processor (120) may determine whether to capture the entire screen of the display (e.g., the first user interface (310)), the first execution screen (e.g., the second user interface (330)) or the second execution screen (315) of the voice assistant application (e.g., capture each or only one screen at a time).

[0076] The above relevance may be determined by determining whether the first content (or first text) included in the first execution screen corresponds to the second content (or second text) included in the second execution screen. The correspondence of the first content to the second content may include whether the first content matches (or is similar to) the second content within a specified range or more. For example, each of the first content and the second content may include at least one of text, an image, or a video.

[0077] When the first content and the second content are 'texts', the processor (120) can determine how much the text of the first content and the text of the second content match. For example, the processor (120) can determine whether the number of identical texts (e.g., words) between the first content and the second content is a specified number (e.g., 3) or more, or whether the proportion of identical texts among the entire texts is a specified percentage (e.g., 30%) or more. Alternatively, when the first content and the second content are 'images (or videos),' the processor (120) can analyze the images of the first content and the images of the second content to determine the identity or similarity between the two images. For example, the processor (120) can determine whether the number of identical objects (e.g., images) or objects of the same category is a specified number or more, or whether the proportion of identical images, objects of the same category, or similar images among the entire images is a specified percentage (e.g., 50%) or more. The processor (120) can similarly determine the correlation between the first content and the second content in a similar manner as described above, even when the first content and the second content include both text and images (or videos).

[0078] The processor (120) may perform screen capture by storing an image of the first execution screen and an image of the second execution screen if the first content included in the first execution screen (311) is related to the second content included in the second execution screen (315) (e.g., the first content corresponds to the second content). Capturing the first execution screen (311) and the second execution screen (315) may include both the meaning of capturing the first execution screen (311) and the second execution screen (315) separately and the meaning of capturing the first execution screen (311) and the second execution screen (315) together (e.g., like the first user interface (310)). The image of the first execution screen (311) may include the entire image of the first execution screen (311) that is not covered by the image of the second execution screen (315). The image of the second execution screen (315) may include the entire image of the second execution screen (315) that is not covered by the image of the first execution screen (311).

[0079] The processor (120) can capture the first execution screen (311) and the second execution screen (315) respectively, or capture the entire screen of the display module (160) (e.g., a screen in which the second execution screen (315) is overlaid on a portion of the first execution screen (311)) as a single image. The entire screen of the display module (160) (e.g., the first user interface (310)) can be a screen in which the second execution screen (315) is overlaid on a portion of the first execution screen (311). When a screen capture command is received through the voice assistant application while the first execution screen (311) is displayed, if the first content included in the first execution screen (311) is related to the second content included in the second execution screen (315), the processor (120) determines that the user intends to capture both the execution screens of the voice assistant application and the first application, and can capture both execution screens of the two applications.

[0080] According to one embodiment, the processor (120) may determine whether to capture the image of the first execution screen and the image of the second execution screen, respectively, or to capture the entire screen of the display module (160) as a single image, depending on the specified size of the display module (160). For example, the processor (120) may capture the first execution screen and the second execution screen, respectively, when the second execution screen is larger than or equal to the specified size of the display module (160) (e.g., 50%, 55%, 60% or more). Alternatively, the processor (120) may capture only the first execution screen when the second execution screen of the voice assistant application is smaller than the specified size of the display module (160).

[0081] Alternatively, the processor (120) may capture the first execution screen (311) if the first content included in the first execution screen is not related to the second content included in the second execution screen (e.g., the first content does not correspond to the second content). If a screen capture command is received through the voice assistant application while the first execution screen (311) is displayed, the processor (120) may determine that the user's intention is to capture the first execution screen (311) if the first content included in the first execution screen (311) is not related to the second content included in the second execution screen, and may capture the first execution screen (311). If the first content included in the first execution screen (311) is not related to the second content included in the second execution screen (315), the processor (120) may determine that the user's intention is to capture only the second execution screen (315), and may capture the second execution screen (315).

[0082] Referring to FIG. 3B, in response to a screen capture request through a voice assistant application, the processor (120) may determine whether to capture the first execution screen and the second execution screen (e.g., the third user interface (350)) as a single image, the first execution screen (e.g., the second user interface (330)), or the second execution screen (375) based on the user's last interaction. The first execution screen (351) and the second execution screen (375) may both be captured and stored with a single capture command, or only one screen (e.g., either the first execution screen (351) or the second execution screen (375)) may be stored. For example, while displaying the first execution screen (351), the processor (120) may receive a first user query (or first user input) through the voice assistant application.

[0083] Here, the first user query is not a screen capture request and may be "What time is set today?". The processor (120) may display a second execution screen of the voice assistant application including the first text (353) corresponding to the first user query on the display module (160). While displaying the first execution screen (351), if a user query is received through the voice assistant application, the processor (120) may provide the second execution screen (375) of the voice assistant application as a pop-up window or display the second execution screen (375) by overlapping it on the first execution screen (351).

[0084] Referring to the drawing symbol (370), the second execution screen (375) may be displayed as a foreground image on the first layer (371), and the first execution screen (351) may be displayed as a background image on the second layer (373). The first layer (371) may be displayed on (e.g., above) the second layer (373). The processor (120) may display the second execution screen (375) on the first execution screen (351). The second execution screen (375) may be provided as a pop-up window, so that an area covered by the pop-up window on the first execution screen (351) (e.g., a lower area of ​​the execution screen (351) of the first application) may not be visible to the user.

[0085] The processor (120) may obtain first result data for the first user query through the voice assistant application, and display a second execution screen of the voice assistant application including second text (355) corresponding to the first result data on the display module (160). In addition, the processor (120) may output a voice corresponding to the first result data to a speaker (e.g., an audio output module (155)). After displaying the second text (355), the processor (120) may receive a second user query requesting screen capture. The second user query may be a screen capture command (or request). The processor (120) may display a second execution screen of the voice assistant application including third text (357) corresponding to the second user query on the display module (160).

[0086] The third user interface (350) may include a second execution screen of a voice assistant application that includes a third text (357) corresponding to the second user query by overlapping a portion of the first execution screen (351). Since the last interaction of the user (e.g., the first user query (353)) before requesting screen capture (e.g., the second user query (357)) is detected in the voice assistant application, the processor (120) may capture the entire screen of the display module (160) (e.g., the third user interface (350)) or the second execution screen (375) of the voice assistant application.

[0087] The second execution screen (375) of the voice assistant application may be displayed on the entire screen of the display module (160) or a portion of the display module (160). The processor (120) may capture the entire screen of the display module (160) including the second execution screen (375), or a portion of the screen of the display module (160) including only the second execution screen (375) of the voice assistant application (e.g., a portion of the screen of the display module (160)). Alternatively, the processor (120) may capture the entire screen of the display module (160) including the second execution screen (375) and store the size of the second execution screen (375) (e.g., a portion of the screen of the display module (160)).

[0088] According to one embodiment, the processor (120) may determine, based on the sharing intent of the screen capture request through the voice assistant application, whether to capture the entire screen of the display module (160) (e.g., the third user interface (350)), the execution screen of the first application (e.g., the second user interface (330)), the second execution screen (375) of the voice assistant application, or the first execution screen (e.g., the second user interface (330)) and the second execution screen (375), respectively. Alternatively, if there is an intention to share a screen capture, the processor (120) may determine whether the screen to be captured contains personal information, and based on whether the personal information is included, determine whether to capture the entire screen of the display module (160) (e.g., the third user interface (350)), the execution screen of the first application (e.g., the second user interface (330)), or the second execution screen (375) of the voice assistant application. The processor (120) may store the captured screen (e.g., the entire screen of the display module (160), the execution screen of the first application, or the second execution screen (375) of the voice assistant application) in the memory (130).

[0089] An electronic device (101) according to an embodiment of the present disclosure includes a display (160), a processor (120), and a memory (130), and instructions stored in the memory, when executed by the processor, may cause the electronic device to display a first execution screen of a first application on the display, display a second execution screen of a voice assistant application overlaid on a portion of the first execution screen of the first application on the display, receive a screen capture command through the voice assistant application while the second execution screen is overlaid and displayed on a portion of the first execution screen, and store an image of the first execution screen, in which the image of the second execution screen is not overlaid, on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen, to perform a screen capture.

[0090] The second execution screen may be displayed as a foreground image in the first layer, and the first execution screen may be displayed as a background image in the second layer.

[0091] The instructions, when executed by the processor, may cause the electronic device to analyze a user intent based on the screen capture command, and, while the second execution screen is displayed as overlaid on a portion of the first execution screen, perform a screen capture by storing at least one of (1) an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen, (2) an image of the second execution screen, or (3) an image in which the image of the second execution screen is overlaid on a portion of the first execution screen based on the analyzed user intent.

[0092] The above instructions, when executed by the processor, may cause the electronic device to perform a screen capture by storing an image of the first execution screen and an image of the second execution screen overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is displayed overlaid on a portion of the first execution screen.

[0093] The instructions, when executed by the processor, may cause the electronic device to determine whether a first text included in the first execution screen is related to a second text included in the second execution screen, and if the first text is related to the second text, perform a screen capture by storing an image of the first execution screen and an image of the second execution screen, and if the first text is not related to the second text, perform a screen capture by storing an image of the first execution screen without storing the image of the second execution screen.

[0094] The above instructions, when executed by the processor, may cause the electronic device to perform a screen capture by storing an image of the first execution screen and an image of the second execution screen without overlaying the image of the second execution screen based on the screen capture command, if the second execution screen is larger than a specified size of the display.

[0095] The instructions, when executed by the processor, may cause the electronic device to determine whether the last interaction of the user before receiving the screen capture command was the first execution screen or the second execution screen, and if the last interaction of the user was the first execution screen, perform a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid in a portion of the first execution screen based on the screen capture command, and if the last interaction of the user was the second execution screen, perform a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid and an image of the second execution screen.

[0096] The above instructions, when executed by the processor, may cause the electronic device to determine whether the screen capture command includes an intent to share a screen capture, and if the screen capture command does not include a sharing intent, to perform a screen capture by storing an image of the first execution screen and an image of the second execution screen.

[0097] The above instructions, when executed by the processor, may cause the electronic device to determine whether a second text included in the second execution screen contains personal information if the screen capture command includes a sharing intent, and to perform a screen capture by storing an image of the first execution screen if the second text contains personal information.

[0098] The above instructions, when executed by the processor, may cause the electronic device to display a preview image of the captured screen in a thumbnail format after the screen capture, and to store a captured image selected by the user from the preview image.

[0099] The above instructions, when executed by the processor, may cause the electronic device to detect a user input for editing after selecting the captured image, and provide a user interface for editing the selected captured image in response to the user input for editing.

[0100] The above instructions, when executed by the processor, may cause the electronic device to store content included in the second execution screen in association with the image of the second execution screen when storing a captured image of the second execution screen.

[0101] The instructions, when executed by the processor, may cause the electronic device to, when capturing an image of the second execution screen, save the entire content included in the area displayed on the display and the area not displayed on the display, if a portion of the content included in the second execution screen extends beyond the display area of ​​the display.

[0102] The above instructions, when executed by the processor, may cause the electronic device to group and display images captured from the execution screen of the voice assistant application on the execution screen of the image storage application, display an indicator indicating that there are other images captured together with the execution screen of the voice assistant application, and, when the indicator is selected, display a representative captured image of one of the grouped images and display the other grouped images in a thumbnail format.

[0103] The instructions, when executed by the processor, may cause the electronic device to provide an editing interface for separating an image of the first execution screen and an image of the second execution screen when the captured image is an image of the second execution screen overlaid on a portion of the first execution screen.

[0104] The instructions, when executed by the processor, may cause the electronic device to receive a screen capture command through the voice assistant application while a plurality of execution screens of applications are displayed through a multi-window, and perform a screen capture by saving an image of the execution screen of the application of the currently focused window, or determine whether a third text included in the execution screen of the application of the currently focused window is related to a second text included in the execution screen of the voice assistant application, and if the third text is related to the second text, perform a screen capture by saving the image of the second execution screen and the image of the execution screen of the application of the currently focused window, and if the third text is not related to the second text, perform a screen capture by saving the image of the execution screen of the application of the currently focused window without saving the image of the second execution screen.

[0105] Figure 4 is a flowchart (400) illustrating an operating method of an electronic device according to one embodiment.

[0106] Referring to FIG. 4, in operation 401, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display an execution screen (e.g., a first execution screen) of a first application on a display (e.g., a display module (160) of FIG. 1). For example, the first application may include all applications except a voice assistant application. For example, the first application may be an Internet search application, a messenger application, or a video streaming application.

[0107] In operation 403, the processor (120) may display an execution screen (e.g., a second execution screen) of a voice assistant application on the display module (160). The voice assistant application (or voice assistant service) may collect and provide customized information to the user based on an artificial intelligence (AI) engine and voice recognition, and may perform various tasks such as schedule management, email transmission, and restaurant reservation according to the user's voice commands. For example, the user may utter a predetermined voice wake word (e.g., Hi Bixby) or press a button related to the voice assistant application. The processor (120) may detect the voice wake word acquired through a microphone (e.g., the input module (150) of FIG. 1) and execute the voice assistant application. The second execution screen may be displayed by overlapping a portion of the first execution screen. For example, the second execution screen may be displayed by overlapping a portion of the first execution screen as a pop-up window. The above second execution screen may be displayed as a foreground image in the first layer, and the above first execution screen may be displayed as a background image in the second layer.

[0108] In operation 405, the processor (120) may receive a screen capture command through a voice assistant application. The screen capture command may be requested through the voice assistant application and may include, for example, a user utterance such as 'screen capture' or 'capture the screen' or a text input such as 'screen capture' entered through a keypad. The processor (120) may receive a screen capture request from the user through the voice assistant application while displaying the first execution screen of the first application.

[0109] In operation 407, the processor (120) may analyze a user intent based on the screen capture command. The processor (120) may analyze whether the user intent is to capture an image of the first execution screen, an image of the second execution screen, or an intent to capture both images of the first execution screen and the second execution screen. The user intent may be to explicitly designate a correlation between contents included in two execution screens or a screen to be captured. The user intent to designate a screen to be captured may include touching (or currently focusing) the screen to be captured, or uttering (e.g., 'Capture the current screen', 'Capture the voice assistant screen').

[0110] According to one embodiment, one method of analyzing the user intent may determine whether the first text (or first content) included in the first execution screen is related to the second text (or second content) included in the second execution screen. As an example, while text is described, each of the first content and the second content may include at least one of text, an image, or a video.

[0111] According to an embodiment, when the first content and the second content are 'texts', the processor (120) may determine how much the text of the first content and the text of the second content match. For example, the processor (120) may determine whether the number of identical texts (e.g., words) between the first content and the second content is a specified number (e.g., 3) or more, or whether the proportion of identical texts among the entire texts is a specified percentage (e.g., 30%) or more. Alternatively, when the first content and the second content are 'images (or videos),' the processor (120) may analyze the images of the first content and the images of the second content to determine the identity or similarity between the two images. For example, the processor (120) may determine whether the number of identical images is a specified number or more, or whether the proportion of identical or similar images among the entire images is a specified percentage (e.g., 50%) or more. The processor (120) can similarly determine the correlation between the first content and the second content in a similar manner as described above, even when the first content and the second content include both text and images (or videos).

[0112] According to one embodiment, another method of analyzing the user intent may include allowing the user to explicitly specify an execution screen to be captured. For example, the processor (120) may specify a screen capture command such as "Capture the first execution screen" or "Capture the first application screen."

[0113] In operation 409, the processor (120) may capture the screen and save it as an image based on the analyzed user intent. When a screen capture request is made through the voice assistant application while the execution screen of the first application is displayed, the processor (120) may perform a correlation analysis between the voice assistant application and the execution screen of the first application as part of the user intent analysis. The correlation may be determining whether the first content (or first text) included in the first execution screen of the first application corresponds to the second content (or second text) included in the second execution screen of the voice assistant application.

[0114] If the second content is related to the first content, the processor (120) determines that the user intends to capture both the voice assistant application and the execution screen of the first application, and may capture the execution screens of the two applications. For example, the processor (120) may capture each of the first execution screen and the first execution screen, or may capture the entire screen of a display (e.g., the display module (160) of FIG. 1) as a single image. Since the entire screen of the display module (160) displays the second execution screen of the voice assistant application on the first execution screen of the first application, the first execution screen of the first application and a portion of the second execution screen of the voice assistant application may overlap.

[0115] According to one embodiment, the processor (120) may capture the execution screen of the first application if the first content does not correspond to the second content. If a screen capture is requested through the voice assistant application while the execution screen of the first application is displayed, and if there is no correlation between the voice assistant application and the execution screen of the first application, the processor (120) may determine that the user intends to capture the execution screen of the first application and may capture the execution screen of the first application.

[0116] According to one embodiment, the processor (120) may determine a screen to be captured (e.g., an execution screen of the voice assistant application or an execution screen of the first application) based on the user's last interaction in response to a screen capture request through the voice assistant application. Alternatively, the processor (120) may determine a screen to be captured based on the user's intention to share the screen capture in response to a screen capture request through the voice assistant application. Alternatively, if there is an intention to share the screen capture, the processor (120) may determine whether the screen to be captured contains personal information and determine the screen to be captured based on whether or not the personal information is included.

[0117] FIG. 5A is a diagram illustrating an example of capturing an execution screen of a voice assistant application in an electronic device according to one embodiment.

[0118] Referring to FIG. 5A, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (510) including an execution screen of a voice assistant application on a display (e.g., the display module (160) of FIG. 1). The first user interface (510) is an execution screen of the voice assistant application and may include first result data (513) corresponding to a first user query (511). The first user query (511) may correspond to a voice command obtained as a user input, for example, through a microphone (e.g., the input module (150) of FIG. 1). The processor (120) may display a text corresponding to the first user query (511) on the voice assistant application. The first result data (513) may include the output of a generative artificial intelligence system in response to a first user query (511). The first result data (513) may be in natural language or specific content format, and may also be provided in the form of an action requested by the user. The processor (120) may display text and / or images corresponding to the first result data (513) in the voice assistant application.

[0119] The processor (120) may display a second user interface (520) on the display module (160) in response to a user input selecting URL information (515) included in the first result data (513). The second user interface (520) may not display the execution screen of a voice assistant application, but may include the execution screen of an Internet search application. Since the first user interface ((510)) has a lot of content to display, it may appear that the voice assistant screen is displayed in full screen and then switched to the Internet, but even when displayed as a pop-up, the pop-up screen may disappear when URL information (515) is selected.

[0120] The execution screen of the Internet search application may include accommodation information located in a specific region (e.g., Fukuoka). While displaying the execution screen of the Internet search application, the processor (120) may receive a user input to call a voice assistant application. The user input may be uttering a predetermined voice call word or selecting a button associated with the voice assistant application.

[0121] While the processor (120) displays the execution screen of the Internet search application, if the user input is detected, the processor (120) may display a third user interface (530) including an indicator (533) indicating that the voice assistant application is called on the display module (160). The third user interface (530) may include an indicator (533) indicating that the voice assistant application is called on the execution screen (531) of the Internet search application. While displaying the third user interface (530), the processor (120) may receive a second user query (541) requesting screen capture.

[0122] The processor (120) may display a fourth user interface (540) on the display module (160) in response to a second user query (541). The fourth user interface (540) is an execution screen of a voice assistant application and may include text corresponding to the second user query (541) (e.g., “Capture me”). The execution screen of the voice assistant application may include text corresponding to each of the first user query, the first result data, or the second user query (541). The processor (120) may store conversation information of the voice assistant application in a memory (e.g., the memory (130) of FIG. 1) for a specified time (e.g., 10 minutes, 1 hour, 1 day) immediately before (or before) detecting the second user query (541). If a second user query (541) is detected within the above-mentioned time, the processor (120) can display the stored conversation information (543) along with text corresponding to the second user query on the voice assistant application.

[0123] The processor (120) can capture the execution screen of the Internet search application based on the second user query (541). The processor (120) can determine the screen to be captured based on the last interaction of the user immediately before receiving the second user query (541) requesting screen capture. The last interaction of the user immediately before receiving the second user query (541) can be detected (or generated) on the execution screen of the Internet search application, such as the second user interface (520). In this case, the processor (120) can capture only the execution screen of the Internet search application without capturing the execution screen of the voice assistant application displayed on the display module (160) in response to the second user query (541). The processor (120) can store the captured image (e.g., the execution screen of the Internet search application) in a memory (e.g., the memory (130) of FIG. 1).

[0124] FIG. 5b is a diagram illustrating an example of capturing the entire screen of a display in an electronic device according to one embodiment.

[0125] Referring to FIG. 5B, when a screen capture is requested through a voice assistant application while the processor (120) displays an execution screen (551) of a first application (e.g., an Internet search application), the processor (120) may display a fifth user interface (550) on the display module (160). The fifth user interface (550) may include an execution screen (553) of a voice assistant application on the execution screen (551) of the first application. After providing first result data (556) for a first user query (555), the processor (120) may receive a second user query (557) requesting a screen capture. For example, a user may be viewing a web page (e.g., displaying an execution screen (551) of the first application), and may ask the voice assistant application what 'EPA' is because the word 'EPA' is unfamiliar. A user input asking what 'EPA' is may correspond to the first user query (555).

[0126] After the processor (120) provides the first result data (556) for the first user query (555), if the processor (120) subsequently receives the second user query (557), it can determine that the user's last interaction was detected in the voice assistant application. In this case, the processor (120) can determine whether the first content included in the execution screen (553) of the voice assistant application corresponds to the second content included in the execution screen (551) of the first application. The correspondence of the first content to the second content may include whether the first content matches the second content by a specified range or more. For example, the first content and the second content may each include at least one of text, an image, or a video. For example, if the first content and the second content are 'text', the processor (120) can determine how much the text of the first content matches the text of the second content. For example, the processor (120) can determine whether there are a specified number (e.g., 3) or more of identical texts (e.g., words) between the first content and the second content, or whether the proportion of identical texts among the entire text is a specified proportion (e.g., 30%) or more.

[0127] The processor (120) may capture the entire screen of the display module (160) in response to the screen capture request when the first content included in the execution screen (553) of the voice assistant application and the second content included in the execution screen (551) of the first application correspond (e.g., the first content matches the second content by a specified range or more). The processor (120) may capture the entire screen of the display module (160), such as the sixth user interface (560). The processor (120) may store the captured image (e.g., the sixth user interface (560)) in the memory (130).

[0128] FIG. 5c is a diagram illustrating an example of capturing an execution screen of a first application or an execution screen of a voice assistant application in an electronic device according to one embodiment.

[0129] Referring to FIG. 5C, if the first content included in the execution screen (553) of the voice assistant application corresponds to the second content included in the execution screen (551) of the first application, the processor (120) may capture the execution screen of the first application or the execution screen of the voice assistant application in response to the screen capture request. For example, the seventh user interface (570) may include the execution screen of the first application. In addition, the eighth user interface (580) may include the execution screen of the voice assistant application. The processor (120) may store the captured image (e.g., the seventh user interface (570) or the eighth user interface (580)) in the memory (130).

[0130] FIG. 5D is a diagram illustrating an example of capturing the entire screen of a display in an electronic device according to one embodiment.

[0131] Referring to FIG. 5D, if the first content included in the execution screen (593) of the voice assistant application corresponds to the second content included in the execution screen (591) of the first application (e.g., an Internet search application), the processor (120) may capture the entire screen of the display module (160) in response to a user's screen capture request through the voice assistant application. The ninth user interface (590) may include the entire screen of the display module (160). Alternatively, if the first content included in the execution screen (593) of the voice assistant application corresponds to the second content included in the execution screen (591) of the first application, the processor (120) may capture the execution screen (593) of the voice assistant application and the execution screen (591) of the first application, respectively. The processor (120) can store the captured image (e.g., the execution screen (593) of the voice assistant application or the execution screen (591) of the first application) in the memory (130).

[0132] According to one embodiment, the processor (120) may capture the execution screen (593) of the voice assistant application and the execution screen (591) of the first application, respectively, when the execution screen (593) of the voice assistant application is larger than a specified size of the display module (160) (e.g., larger than 50%, 55%, 60%, etc.). Alternatively, the processor (120) may capture only the execution screen (591) of the first application, when the execution screen (593) of the voice assistant application is smaller than a specified size of the display module (160).

[0133] FIG. 6 is a flowchart (600) illustrating a method for capturing a screen based on a user's last interaction in an electronic device according to one embodiment. FIG. 6 may be performed alone as a method for capturing a screen, or may be performed prior to operation 401 of FIG. 4 or when 'Yes' is determined in operation 403 of FIG. 4.

[0134] Referring to FIG. 6, in operation 601, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1) according to an embodiment may identify a last user interaction. The last user interaction may refer to a user input immediately before requesting a screen capture via a voice assistant application. The user input may include an input of touching a display (e.g., display module (160) of FIG. 1), an input of pressing a physical button, or a voice command (e.g., a user query).

[0135] In operation 603, the processor (120) may determine whether the identified last interaction was detected in the voice assistant application. The user may request a screen capture while the execution screen of the first application is displayed, or may request a screen capture while the execution screen of the voice assistant application is displayed. For example, if a screen capture is requested while the execution screen of the first application is displayed, the user's last interaction may be detected in the execution screen of the first application. Alternatively, if a screen capture is requested while the execution screen of the voice assistant application is displayed, the user's last interaction may be detected in the execution screen of the voice assistant application.

[0136] The processor (120) may perform operation 605 if the identified last interaction is detected in the voice assistant application, and may perform operation 607 if the identified last interaction is not detected in the voice assistant application.

[0137] If the identified last interaction is detected in the voice assistant application, in operation 605, the processor (120) may capture the execution screens of the voice assistant application and the first application. The processor (120) may store the captured image in a memory (e.g., the memory (130) of FIG. 1). The captured image may be the execution screens of the voice assistant application and the first application.

[0138] If the identified last interaction is not detected in the voice assistant application, in operation 607, the processor (120) may capture the execution screen of the first application. The processor (120) may store the captured image, the execution screen of the first application, in the memory (130). If the identified last interaction is not detected in the voice assistant application, the identified last interaction may be detected in the first application. In this case, the processor (120) may not capture the execution screen of the voice assistant application, but may capture only the execution screen of the first application.

[0139] FIG. 7A is a diagram illustrating an example of capturing a screen based on a user's last interaction in an electronic device according to one embodiment.

[0140] Referring to FIG. 7A, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (710) on a display (e.g., a display module (160) of FIG. 1) when a screen capture is requested through a voice assistant application while displaying an execution screen (711) of a first application (e.g., an Internet search application). The first user interface (710) may include an execution screen (713) of a voice assistant application on the execution screen (711) of the first application. After providing first result data (717) for a first user query (715), the processor (120) may receive a second user query (719) requesting a screen capture. For example, a user may be viewing a web page (e.g., displaying the execution screen (711) of the first application) and, wondering about the location of 'Israel', may ask the voice assistant application where the 'location of Israel' is. The user input asking about the 'location of Israel' may correspond to the first user query (715). The execution screen (713) of the voice assistant application may include text corresponding to the first user query (715), text corresponding to the first result data (717), and an image.

[0141] After the processor (120) provides the first result data (717) for the first user query (715), if the processor (120) subsequently receives the second user query (719), it can determine that the user's last interaction was detected in the voice assistant application. In this case, the processor (120) can determine whether the first content included in the execution screen (713) of the voice assistant application corresponds to the second content included in the execution screen (711) of the first application. The correspondence of the first content to the second content may include whether the first content matches the second content by a specified range or more. For example, the first content and the second content may each include at least one of text, an image, or a video. For example, if the first content and the second content are 'text', the processor (120) can determine how much the text of the first content matches the text of the second content. For example, the processor (120) can determine whether there are a specified number (e.g., 3) or more of identical texts (e.g., words) between the first content and the second content, or whether the proportion of identical texts among the entire text is a specified proportion (e.g., 30%) or more.

[0142] When the first content included in the execution screen (713) of the voice assistant application and the second content included in the execution screen (711) of the first application correspond (e.g., the first content matches the second content by a specified range or more), the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application in response to the screen capture request. For example, the second user interface (720) may include the execution screen of the first application. In addition, the third user interface (730) may include the execution screen of the voice assistant application. The processor (120) may store the captured image (e.g., the second user interface (720) or the third user interface (730)) in a memory (e.g., the memory (130) of FIG. 1).

[0143] FIG. 7b is a diagram illustrating an example of capturing a screen based on an association between a voice assistant application and a first application in an electronic device according to one embodiment.

[0144] Referring to FIG. 7B, the processor (120) may display a fourth user interface (740) including an execution screen of a first application (e.g., a travel application) on the display module (160). The fourth user interface (740) is an execution screen of the travel application and may include information introducing a specific region (e.g., Busan). While displaying the execution screen of the travel application, the processor (120) may receive a first user inquiry from the user through a voice assistant application. While viewing the introduction information of a specific region, the user may be curious about the weather information of the specific region and may inquire about 'the weather in Haeundae next week' through the voice assistant application.

[0145] A user input asking about 'the weather in Haeundae next week' may correspond to a first user query (751). The fifth user interface (750) is an execution screen of the voice assistant application, and may include text corresponding to the first user query (751), text and an image corresponding to the first result data (753) corresponding to the first user query (751). After providing the first result data (753) for the first user query (751), the processor (120) may then receive a second user query (755) requesting a screen capture. The fifth user interface (750) may further include text corresponding to the second user query (755).

[0146] The processor (120) may determine a screen to capture based on the user's last interaction immediately before receiving the second user query (755). The user's last interaction immediately before receiving the second user query (755) may be detected (or generated) on the execution screen of the voice assistant application, such as the fifth user interface (750). In this case, the processor (120) may determine whether the first content included in the execution screen of the voice assistant application (e.g., the fifth user interface (750)) corresponds to the second content included in the execution screen of the travel application (e.g., the fourth user interface (740)). The correspondence of the first content to the second content may include whether the first content matches (or is similar to) the second content within a specified range or more.

[0147] The processor (120) may determine that although Haeundae and Busan are not the same words, Haeundae is geographically included in Busan and that there is a correlation between the regional introduction information and the weather information. If there is a correlation between the execution screen of the voice assistant application and the execution screen of the travel application, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the travel application. The sixth user interface (770) may display a preview image of the captured screen in thumbnail format on the execution screen of the voice assistant application. The first thumbnail image (771) included in the sixth user interface (770) may be the execution screen of the voice assistant application, and the second thumbnail image (773) may be the execution screen of the travel application.

[0148] A user can select a screen to be captured through a thumbnail image. For example, when a screen capture command is received, the processor (120) may display the entire screen, the execution screen of the voice assistant application, and the execution screen of the travel application as thumbnail images, respectively, and save only the thumbnail images selected by the user as capture screens (e.g., by placing a check mark on the thumbnail image). In addition, when a current screen capture is requested, a menu item (775) may be displayed, such as the sixth user interface (770). The menu item (775) may include a menu for editing the captured image or a scroll capture menu. The menu item (775) may be displayed when a screen capture command is received and may disappear after a certain period of time (e.g., by removing the display).

[0149] FIG. 7c is a diagram illustrating an example of capturing an execution screen of a voice assistant application and an execution screen of a first application in an electronic device according to one embodiment.

[0150] Referring to FIG. 7c, if the execution screen of the voice assistant application and the execution screen of the travel application are related, the processor (120) may store the execution screen of the voice assistant application and the execution screen of the travel application in the memory (130). The seventh user interface (780) may include the execution screen of the travel application. The eighth user interface (790) may include the execution screen of the voice assistant application.

[0151] FIG. 8 is a flowchart (800) illustrating a method for capturing a screen based on an intention to share a screen capture in an electronic device according to an embodiment. FIG. 8 may be performed alone as a method for capturing a screen, or may be performed when 'Yes' is determined in operation 403 of FIG. 4 or 'Yes' is determined in operation 603 of FIG. 6.

[0152] Referring to FIG. 8, in operation 801, a processor (e.g., the processor (120) of FIG. 1) of an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may determine an intention to share a screen capture. When a request for screen capture is received from a user through a voice assistant application, the processor (120) may determine an intention to share the screen capture. The user may simply utter, "Capture the screen," but may also utter, "Capture the screen and send it to Mark," to share the captured screen with the other party (e.g., someone other than the user).

[0153] If there is an intention to share the screen capture, the processor (120) may perform operation 805, and if there is no intention to share the screen capture, the processor (120) may perform operation 803.

[0154] If there is no intention to share the screen capture, in operation 803, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. If a screen capture is requested while the execution screen of the voice assistant application is displayed, the first content included in the voice assistant application corresponds to the second content included in the first application, and there is no intention to share the screen capture, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. The processor (120) may store the captured image in a memory (e.g., the memory (130) of FIG. 1). The captured image may be the execution screen of the voice assistant application and the first application.

[0155] If there is an intention to share the above screen capture, in operation 805, the processor (120) may determine whether the voice assistant application contains personal information. The personal information may include private information related to the user, such as the user's weight, height, date of birth, address, face, or contact information. If the voice assistant application contains personal information, the processor (120) may perform operation 807, and if the first content included in the voice assistant application does not contain personal information, the processor (120) may perform operation 803.

[0156] If the voice assistant application contains personal information, in operation 807, the processor (120) may capture the execution screen of the first application. If the voice assistant application contains personal information, the processor (120) may not capture the execution screen of the voice assistant application, but may capture only the execution screen of the first application, in order to protect the personal information. According to one embodiment, the processor (120) may capture the execution screen of the voice assistant application with only the personal information portion processed to be invisible, and share the captured execution screen of the voice assistant application. For example, the processor (120) may process the personal information portion by masking it with another layer or replacing it with a random number. The processor (120) may share the captured image with the other party (e.g., transmit it to the other party's electronic device). The captured image may be the execution screen of the first application. The processor (120) may store the captured image in the memory (130).

[0157] If there is an intention to share the screen capture and the voice assistant application does not contain personal information, in operation 803, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. If a screen capture is requested while the execution screen of the voice assistant application is displayed, the first content included in the voice assistant application corresponds to the second content included in the first application, there is an intention to share the screen capture, and the voice assistant application does not contain personal information, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. The processor (120) may share the captured image with the other party (e.g., transmit it to the other party's electronic device). The captured image may be the execution screen of the voice assistant application and the first application. The processor (120) may store the captured image in the memory (130).

[0158] FIG. 9A is a diagram illustrating an example of capturing a screen based on an intention to share a screen capture in an electronic device according to one embodiment.

[0159] Referring to FIG. 9A, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (910) including an execution screen of a voice assistant application on a execution screen of a first application (e.g., a shopping application) on a display (e.g., a display module (160) of FIG. 1). While displaying the execution screen of the shopping application, the processor (120) may receive a first user query and a second user query from a user through the voice assistant application. While looking at sneakers, the user may be curious about the soccer match schedule and may inquire about 'when the soccer match is' through the voice assistant application. The user input of 'inquiring about the soccer match schedule' may correspond to the first user query. The first user interface (910) may include text corresponding to a first user query, text (or an image) corresponding to first result data corresponding to the first user query, and a second user query. The second user query may be a request to capture a screen and share the captured screen with another user.

[0160] The processor (120) can determine whether the screen capture includes a sharing intent, and if the screen capture includes a sharing intent, can determine whether the execution screen of the voice assistant application includes personal information. If the execution screen of the voice assistant application does not include personal information, the processor (120) can capture the execution screen of the voice assistant application and the execution screen of the shopping application. The second user interface (920) can include a preview image of the captured screen in thumbnail format on the execution screen of the voice assistant application. The first thumbnail image (921) included in the second user interface (920) can be the execution screen of the voice assistant application, and the second thumbnail image (922) can be the execution screen of the shopping application. In addition, when a current screen capture is requested, a menu item (923) can be displayed as in the second user interface (920). The menu item (923) may include a menu for editing the captured image or a scroll capture menu. The menu item (923) may be displayed when a screen capture command is received and then disappear after a certain period of time (e.g., removed from display). The processor (120) may share the captured image with the other party (e.g., transmit it to the other party's electronic device). The captured image may be an execution screen of the voice assistant application and the first application.

[0161] If the execution screen of the voice assistant application includes personal information, the processor (120) may capture the execution screen of the shopping application. The third user interface (930) may include the execution screen of the shopping application. The processor (120) may share the captured image with the other party (e.g., transmit it to the other party's electronic device). The captured image may be the execution screen of the first application.

[0162] FIG. 9b is a diagram illustrating an example of capturing a screen by specifying a screen to be captured in an electronic device according to one embodiment.

[0163] Referring to FIG. 9B, the processor (120) may be configured by the user to capture the execution screen of the voice assistant application or the execution screen of the first application. The processor (120) may capture only the execution screen of the application selected by the user (e.g., by touch) or may capture only the execution screen of the currently focused application. The processor (120) may determine the screen to be captured by judging the user's intention for requesting capture through the voice assistant application. The processor (120) may determine the screen to be captured by interpreting and judging inferable language, such as referential pronouns or relative pronouns, through user input (e.g., voice utterance or typing command) through the voice assistant application.

[0164] When a request is received from a user to capture the execution screen of a voice assistant application (e.g., capture this screen now), the processor (120) may capture the execution screen of the voice assistant application. The fifth user interface (950) may include the execution screen of the voice assistant application.

[0165] Alternatively, if a request is received from the user to capture the execution screen of the first application (e.g., app screen capture, previous screen capture, capture after uttering the app name), the processor (120) may capture the execution screen of the first application. The sixth user interface (960) may include the execution screen of the first application.

[0166] Alternatively, if a request is received from the user to capture the execution screen of the voice assistant application and the first application (e.g., capture the voice assistant app and the first app screen together, capture both screens), the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. The seventh user interface (970) may include the execution screen of the voice assistant application and the execution screen of the first application.

[0167] FIG. 10 is a flowchart (1000) illustrating a method for capturing a screen by identifying the intention of a user who requested a screen capture in an electronic device according to one embodiment.

[0168] Referring to FIG. 10, in operation 1001, a processor (e.g., processor (120) of FIG. 1) of an electronic device (e.g., electronic device (101) of FIG. 1) according to an embodiment may detect a user input for screen capture through a voice assistant application. While displaying an execution screen of a first application, the processor (120) may receive a screen capture request from a user through the voice assistant application. The user input may be requested through the voice assistant application and may include, for example, a user utterance saying "screen capture."

[0169] The above voice assistant application collects and provides customized information to the user based on an artificial intelligence (AI) engine and voice recognition, and can perform various tasks such as schedule management, email transmission, and restaurant reservation based on the user's voice commands. For example, the user can utter a predetermined voice wake word or press a button related to the voice assistant application. The processor (120) can detect the voice wake word acquired through a microphone (e.g., the input module (150) of FIG. 1) and execute the voice assistant application. The processor (120) can display text corresponding to the user input for screen capture on the execution screen of the voice assistant application.

[0170] Although the drawing illustrates that actions 1003, 1005, 1007, and 1009 are performed in that order, the order of the actions may be changed. For example, action 1005 may be performed first, followed by actions 1003, 1007, and 1009 in that order. Alternatively, some actions may not be performed.

[0171] In operation 1003, the processor (120) may analyze a user intent. The processor (120) may analyze whether the user intent included in the detected user input is to capture an image of the execution screen of the first application, to capture an image of the execution screen of the voice assistant application, or to capture images of both the execution screen of the first application and the execution screen of the voice assistant application. The user intent may be a correlation between contents included in two execution screens or an explicit designation of a screen to be captured. The user intent for designating a screen to be captured may include touching (or currently focusing) the screen to be captured or uttering (e.g., 'Capture the current screen', 'Capture the voice assistant screen').

[0172] According to one embodiment, the processor (120) may determine, as an example of analyzing user intent, whether the first content (or first text) included in the first application corresponds to the second content (or second text) included in the voice assistant application. The correspondence of the first content to the second content may include whether the first content matches the second content by a specified range or more. For example, each of the first content and the second content may include at least one of text, an image, or a video. For example, when the first content and the second content are 'texts', the processor (120) may determine how much the text of the first content matches the text of the second content.

[0173] In operation 1005, the processor (120) may determine whether the last interaction was detected in the voice assistant application. The processor (120) may determine whether the last user interaction immediately before the user input for the screen capture was detected in the voice assistant application or on the execution screen of the first application. The last user interaction may refer to a user input immediately before requesting a screen capture through the voice assistant application. The user input may include an input by touching a display (e.g., a display module (160) of FIG. 1), an input by pressing a physical button, or a voice command (e.g., a user query).

[0174] A user may request a screen capture while displaying the execution screen of a first application, or may request a screen capture while displaying the execution screen of a voice assistant application. For example, if a screen capture is requested while displaying the execution screen of the first application, the user's last interaction may be detected on the execution screen of the first application. Alternatively, if a screen capture is requested while displaying the execution screen of the voice assistant application, the user's last interaction may be detected on the execution screen of the voice assistant application. If the identified last interaction is detected on the voice assistant application, the processor (120) may perform operation 1007, and if the identified last interaction is not detected on the voice assistant application, the processor (120) may perform operation 1013.

[0175] If the last interaction identified above is detected by the voice assistant application, in operation 1007, the processor (120) can determine the intention to share the screen capture. If the processor (120) receives a request for screen capture from the user through the voice assistant application, the processor (120) can determine the intention to share the screen capture. The user may simply utter, "Capture the screen," but may also utter, "Capture the screen and send it to OO," to share the captured screen with the other party (e.g., someone other than the user).

[0176] If there is an intention to share the screen capture, the processor (120) may perform operation 1009, and if there is no intention to share the screen capture, the processor (120) may perform operation 1013.

[0177] If there is an intention to share the above screen capture, in operation 1009, the processor (120) may determine whether the voice assistant application contains personal information. The personal information may include private information related to the user, such as the user's weight, height, date of birth, address, face, or contact information. If the voice assistant application contains personal information, the processor (120) may perform operation 1013, and if the voice assistant application does not contain personal information, the processor (120) may perform operation 1011.

[0178] If the voice assistant application does not contain personal information, in operation 1011, the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. If the last interaction of the user is detected in the voice assistant application, the first content corresponds to the second content, and there is no intention to share the screen capture (e.g., 'No' in operation 1007), the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application.

[0179] If the last interaction of the user is detected in the voice assistant application, the first content corresponds to the second content, there is an intention to share the screen capture, and the voice assistant application does not contain personal information (e.g., 'No' in operation 1009), the processor (120) may capture the execution screen of the voice assistant application and the execution screen of the first application. The processor (120) may share the captured image with the other party (e.g., transmit it to the other party's electronic device). The captured image may be the execution screen of the voice assistant application and the execution screen of the first application. The processor (120) may store the captured image in a memory (e.g., the memory (130) of FIG. 1).

[0180] In operation 1013, the processor (120) may capture the execution screen of the first application. For example, if the last interaction of the user is detected in the first application (e.g., 'No' in operation 1005), the processor (120) may capture the execution screen of the first application. If the last interaction of the user is detected in the voice assistant application and the first content does not correspond to the second content (e.g., 'No' in operation 1005), the processor (120) may capture the execution screen of the first application. If the last interaction of the user is detected in the voice assistant application and the first content corresponds to the second content and there is no intention to share the screen capture (e.g., 'No' in operation 1007), the processor (120) may capture the execution screen of the first application. The processor (120) may capture the execution screen of the first application when the user's last interaction is detected in the voice assistant application, the first content corresponds to the second content, there is an intention to share a screen capture, and the voice assistant application includes personal information (e.g., 'Yes' in operation 1009).

[0181] FIG. 11 is a diagram illustrating an example of selectively storing a captured screen in an electronic device according to one embodiment.

[0182] Referring to FIG. 11, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (1110) on a display (e.g., a display module (160) of FIG. 1) in response to a screen capture request through a voice assistant application. The first user interface (1110) may include an execution screen of a voice assistant application (1113) on an execution screen (1111) of a first application. In addition, the first user interface (1110) may include a preview image of the captured screen in thumbnail format. The first thumbnail image (1101) included in the first user interface (1110) may be an execution screen of the voice assistant application, the second thumbnail image (1103) may be an execution screen of the first application, and the third thumbnail image (1105) may be an entire screen of the display module (e.g., a screen in which the execution screen of the voice assistant application is displayed on the execution screen of the first application). In addition, the first user interface (1110) may further include a menu item for editing the captured image.

[0183] A user can select at least one image (e.g., a captured image or an image for capturing) to save or edit. The second user interface (1130) may indicate that the first thumbnail image (1101) and the second thumbnail image (1103) are selected from among the first thumbnail image (1101), the second thumbnail image (1103), and the third thumbnail image (1105). The user can change the selected thumbnail (e.g., select (check box check) → deselect (check box uncheck), or deselect → select). The selected image may be displayed to be distinct from the unselected image. For example, the first thumbnail image (1101) and the second thumbnail image (1103), which are the selected images, may be marked with a check mark to indicate that they are selected. The third user interface (1150) may indicate that the second thumbnail image (1103) has been selected from among the first thumbnail image (1101), the second thumbnail image (1103), and the third thumbnail image (1105). The processor (120) may store or edit the selected thumbnail image in a memory (e.g., the memory (130) of FIG. 1) according to a user's request.

[0184] FIG. 12 is a diagram illustrating an example of providing a screen captured image through a voice assistant application in an electronic device according to one embodiment.

[0185] Referring to FIG. 12, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (1210) including an execution screen of a gallery application on a display (e.g., a display module (160) of FIG. 1). The gallery application may provide images or videos stored in a memory (e.g., a memory (130) of FIG. 1) of the electronic device (101). The processor (120) may group (e.g., bundle images) images captured (or stored) through a voice assistant application and provide them to the gallery application. When the processor (120) receives a selection from a user among the images stored through the voice assistant application in the first user interface (1210), the processor (120) may display a second user interface (1230) on the display module (160).

[0186] The second user interface (1230) may include a selected image (1231) and an indicator (1233) related to the selected image (1231). The indicator (1233) may indicate the number of images (e.g., 2) stored (or captured) with the selected image (1231) through the voice assistant application. When a detailed view is requested from a user in the second user interface (1230), the processor (120) may display a third user interface (1250) on the display module (160). The detailed view request from the user may be a scroll input (1251) of touching and dragging an image displayed on the display module (160). The third user interface (1250) may include an image list (1253) related to the selected image (1231).

[0187] When the user continues to input an upward scroll in the third user interface (1250), the processor (120) may display a fourth user interface (1270) on the display module (160). The fourth user interface (1270) includes detailed information (or content) corresponding to the selected image (1231), and the detailed information may include text (1271) and a related image (1273). According to one embodiment, when storing an image captured from the execution screen of the voice assistant application, the processor (120) may store detailed information of the execution screen of the voice assistant application in association with the captured image. That is, the memory (130) may store a captured image (e.g., the execution screen of the voice assistant application) and detailed information associated with the captured image. The detailed information may include at least one of text, an image, or a video.

[0188] FIG. 13 is a diagram illustrating an example of capturing content outside the screen size in an electronic device according to one embodiment.

[0189] Referring to FIG. 13, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (1310) on a display (e.g., a display module (160) of FIG. 1). The first user interface (1310) illustrates an example in which a screen capture is requested through a voice assistant application. The processor (120) may capture the execution screen of the voice assistant application in response to the screen capture through the voice assistant application. In this case, if the content included in the execution screen of the voice assistant application goes beyond the display module (160), the processor (120) may capture not only the content displayed on the display module (160), but also the content not displayed on the display module (160).

[0190] The second user interface (1330) is depicted as being larger in both the upper and lower dimensions than the first user interface (1310). The size of the display module (160) corresponds to the size of the first user interface (1310), and the second user interface (1330) may be depicted to indicate that a portion of the content for which a screen capture has been requested is outside the display module (160) to aid in understanding the invention.

[0191] The processor (120) may not only store the execution screen of the voice assistant application displayed on the display module (160), but may also capture the entire content included in the voice assistant application that requested the screen capture and store it in a memory (e.g., the memory (130) of FIG. 1). The captured content may be the same as the third user interface (1350). The third user interface (1350) may include at least one of text, an image, or a video as the captured content. The captured content may be stored as detailed information of the captured image, and detailed information not displayed on the display module (160) may be further displayed according to the user's scroll input on the captured image.

[0192] FIG. 14 is a diagram illustrating an example of providing an image captured through a voice assistant application in an electronic device according to one embodiment.

[0193] Referring to FIG. 14, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (1410) on a display (e.g., a display module (160) of FIG. 1). The first user interface (1410) may be a running screen of a gallery application, and may include an image (1411) captured (or stored) through a voice assistant application and an indicator (1413) related to the image (1411). The running screen of the gallery application may include a plurality of stored background images (e.g., a first running screen of a first application) and assistant images (e.g., a second running screen of a voice assistant application) or an integrated full image (e.g., a full screen in which a second running screen is overlaid on a first running screen), and may be grouped and stored. Although the drawing shows two images grouped together, an image overlaid on the second execution screen may also be added to the first execution screen. The indicator (1413) may indicate the number of images (e.g., 2) grouped together with the image (1411) captured through the voice assistant application. The processor (120) may group the images captured through the voice assistant application and indicate the number of grouped images with the indicator (1413).

[0194] When the processor (120) detects a user input for selecting an indicator (1413) in the first user interface (1410), it may display a second user interface (1430) on the display module (160). The second user interface (1430) may display grouped images (1431, 1433) together with the image (1411) captured through the voice assistant application in a thumbnail format. The grouped images may include images (e.g., an image of a second execution screen, or an image in which the second execution screen is overlaid on the first execution screen) that are stored (or captured) together when the image (1411) is stored (or captured). For example, the second user interface (1430) may include the grouped images, the first thumbnail image (1431) or the second thumbnail image (1433), in a thumbnail format.

[0195] When a second thumbnail image (1433) is selected from a second user interface (1430), the processor (120) may display a third user interface (1450) on the display module (160). The third user interface (1450) may display the second thumbnail image (1433) and images (1451, 1453) grouped with the second thumbnail image (1433) in a thumbnail format. For example, the third user interface (1450) may include a third thumbnail image (1451) or a fourth thumbnail image (1453), which are images grouped with the second thumbnail image (1433), in a thumbnail format.

[0196] FIG. 15 is a diagram illustrating an example of editing an image captured through a voice assistant application in an electronic device according to one embodiment.

[0197] Referring to FIG. 15, a processor (e.g., a processor (120) of FIG. 1) of an electronic device (e.g., an electronic device (101) of FIG. 1) according to an embodiment may display a first user interface (1510) on a display (e.g., a display module (160) of FIG. 1). The first user interface (1510) may include an image captured through a voice assistant application and may include images grouped together with the captured image. The grouped images are displayed in a thumbnail format and may include, for example, a first thumbnail image (1511) and a second thumbnail image (1513).

[0198] When the edit menu (1515) is selected in the first user interface (1510), the processor (120) can display the second user interface (1530) on the display module (160). The second user interface (1530) illustrates an example of editing a grouped image through the edit tool (1531). The second user interface (1530) may display a portion of the first thumbnail image (1511) and a portion of the second thumbnail image (1513) based on the edit tool (1531). The user can edit the first thumbnail image (1511) or the second thumbnail image (1513) by moving the edit tool (1531). For example, if the first thumbnail image (1511) includes the execution screen of the voice assistant application on top of the execution screen of the first application (e.g., an image of the entire screen of the display module (160)), the user can edit it to include only the execution screen of the first application or to include only the execution screen of the voice assistant application.

[0199] FIG. 16A is a diagram illustrating an example of capturing a screen through a voice assistant application in an electronic device having various sizes according to one embodiment.

[0200] Referring to FIG. 16A, an electronic device (e.g., the electronic device (101) of FIG. 1) according to an embodiment may have various screen sizes (or display sizes). If the electronic device (101) is a tablet PC, execution screens of multiple applications may be displayed in a multi-window manner, such as in the first user interface (1610). The first user interface (1610) may include execution screens of the first applications (1611), execution screens of the second applications (1613), and execution screens of the third applications (1615). Alternatively, if the electronic device (101) is a computer, execution screens of multiple applications may be displayed in a multi-window manner, such as in the second user interface (1620). Alternatively, if the electronic device (101) is a foldable electronic device (or flexible electronic device), execution screens of multiple applications may be displayed in a multi-window manner, such as in the third user interface (1630). Depending on the type of electronic device (101), it can be seen that the screen sizes are different, such as the first user interface (1610), the second user interface (1620), and the third user interface (1630). Even when the screen sizes are different, the electronic device (101) can implement (or apply) the invention described through FIGS. 2 to 15.

[0201] FIG. 16b is a diagram illustrating an example of capturing a screen through a voice assistant application in an electronic device having various sizes according to one embodiment.

[0202] Referring to FIG. 16B, when the electronic device (101) is a foldable electronic device, execution screens of multiple applications can be displayed on a display (e.g., the display module (160) of FIG. 1) through a multi-window. The fourth user interface (1650) may include execution screens (1651) of first applications, execution screens (1653) of second applications, and execution screens (1655) of third applications. The fourth user interface (1650) may be an example in which the last interaction of the user is detected on the execution screens (1653) of the second applications, and the execution screen of the third application (1655) is an execution screen of a voice assistant application.

[0203] When a screen capture is requested through a voice assistant application while the processor (120) of the electronic device (101) displays the fourth user interface (1650), the processor can capture the execution screens of the second applications and the execution screens of the third application. The fifth user interface (1660) can display the execution screens of the second applications, and the sixth user interface (1670) can display the execution screens of the third applications. In response to the screen capture through the voice assistant application, the processor (120) can capture an image corresponding to the fifth user interface (1660) and an image corresponding to the sixth user interface (1670), and store the captured images in a memory (e.g., the memory (130) of FIG. 1).

[0204] According to an embodiment of the present disclosure, an operating method of an electronic device (101) may include an operation of displaying a first execution screen of a first application on a display (160) of the electronic device, an operation of displaying a second execution screen of a voice assistant application overlaid on a portion of the first execution screen of the first application on the display, an operation of receiving a screen capture command through the voice assistant application while the second execution screen is overlaid and displayed on a portion of the first execution screen, and an operation of performing a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen.

[0205] The second execution screen may be displayed as a foreground image in the first layer, and the first execution screen may be displayed as a background image in the second layer. The method may further include an operation of analyzing a user intent based on the screen capture command. The operation may include an operation of performing a screen capture by storing at least one of (1) an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen, (2) an image of the second execution screen, or (3) an image in which the image of the second execution screen is overlaid on a portion of the first execution screen, while the second execution screen is overlaid and displayed on a portion of the first execution screen.

[0206] The method may include an operation of determining whether a first text included in the first execution screen is related to a second text included in the second execution screen, an operation of performing a screen capture by storing an image of the first execution screen and an image of the second execution screen if the first text is related to the second text, and an operation of performing a screen capture by storing an image of the first execution screen without storing the image of the second execution screen if the first text is not related to the second text.

[0207] The method may further include an operation of determining whether the last interaction of the user before receiving the screen capture command is the first execution screen or the second execution screen, an operation of performing a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen based on the screen capture command if the last interaction of the user is the first execution screen, and an operation of performing a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid and an image of the second execution screen if the last interaction of the user is the second execution screen.

[0208] The various embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to facilitate easy explanation of the technical content of the present invention and aid understanding thereof, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be interpreted to include all modifications or variations derived based on the technical concept of the present invention, in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), display (160); Processor (120); and The instructions stored in the memory (130), when executed by the processor, cause the electronic device to: Display the first execution screen of the first application on the above display, Displaying a second execution screen of a voice assistant application on the display, overlaying a portion of the first execution screen of the first application; While the second execution screen is displayed as an overlay on a portion of the first execution screen, a screen capture command is received through the voice assistant application, An electronic device that performs a screen capture by storing an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen.

2. In paragraph 1, An electronic device, wherein the second execution screen is displayed as a foreground image in the first layer, and the first execution screen is displayed as a background image in the second layer.

3. In the first paragraph, when the instructions are executed by the processor, the electronic device, Analyze user intent based on the above screen capture command, An electronic device that performs a screen capture by storing at least one of (1) an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen, (2) an image of the second execution screen, or (3) an image of the first execution screen in which the image of the second execution screen is overlaid on a portion of the first execution screen, based on the analyzed user intention, while the second execution screen is overlaid and displayed on a portion of the first execution screen.

4. In the first paragraph, when the instructions are executed by the processor, the electronic device, An electronic device that performs a screen capture by saving an image of the first execution screen and an image of the second execution screen overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is displayed as an overlay on a portion of the first execution screen.

5. In the first paragraph, when the instructions are executed by the processor, the electronic device, Determine whether the first text included in the first execution screen is related to the second text included in the second execution screen, If the first text is related to the second text, a screen capture is performed by saving an image of the first execution screen and an image of the second execution screen, An electronic device that performs screen capture by saving an image of the first execution screen without saving an image of the second execution screen if the first text is not related to the second text.

6. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: An electronic device that performs screen capture by storing an image of the first execution screen and an image of the second execution screen without overlaying the image of the second execution screen based on the screen capture command when the second execution screen is larger than a specified size of the display.

7. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: Before receiving the above screen capture command, determine whether the user's last interaction is the first execution screen or the second execution screen, If the last interaction of the user is the first execution screen, a screen capture is performed by saving an image of the first execution screen in which the image of the second execution screen is not overlaid on a part of the first execution screen based on the screen capture command, An electronic device that performs a screen capture by saving an image of the first execution screen and an image of the second execution screen without overlaying the image of the second execution screen when the last interaction of the user is the second execution screen.

8. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: Determine whether the above screen capture command includes the intent to share a screen capture, If the above screen capture command does not include a sharing intent, the screen capture is performed by saving the image of the first execution screen and the image of the second execution screen, If the above screen capture command includes a sharing intent, determine whether the second text included in the second execution screen includes personal information, An electronic device that performs screen capture by saving an image of the first execution screen when the second text contains personal information.

9. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: After capturing the above screen, a preview image of the captured screen is displayed in thumbnail format. Save the captured image selected by the user from the above preview image, After selecting the above captured image, detect user input for editing, An electronic device providing a user interface for editing said selected captured image in response to user input for said editing.

10. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: An electronic device that, when saving a captured image of the second execution screen, stores content included in the second execution screen in association with the image of the second execution screen.

11. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: An electronic device that, when capturing an image of the second execution screen, if a part of the content included in the second execution screen goes beyond the display area of ​​the display, saves the entire content included in the area displayed on the display and the area not displayed on the display.

12. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: In the execution screen of the image storage application, the captured image of the execution screen of the voice assistant application is grouped and displayed, Display an indicator indicating that there is another image captured along with the running screen of the above voice assistant application, An electronic device that, when the above indicator is selected, displays a representative captured image of one of the grouped images and displays the other grouped images in thumbnail format.

13. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: An electronic device that provides an editing interface for separating the image of the first execution screen and the image of the second execution screen when the captured image is an image of the second execution screen overlaid on a portion of the first execution screen.

14. In the first paragraph, the instructions, when executed by the processor, cause the electronic device to: When multiple application execution screens are displayed through multi-window, a screen capture command is received through the voice assistant application, Perform a screen capture by saving an image of the running screen of the application in the currently focused window, or Determine whether the third text included in the execution screen of the application of the currently focused window is related to the second text included in the execution screen of the voice assistant application, If the third text is related to the second text, a screen capture is performed by saving an image of the second execution screen and an image of the execution screen of the application of the currently focused window, An electronic device that performs screen capture by saving an image of the execution screen of the application of the currently focused window without saving an image of the second execution screen if the third text is not related to the second text.

15. In the operating method of an electronic device (101), An action of displaying a first execution screen of a first application on a display of the electronic device; An action of displaying a second execution screen of a voice assistant application on the display, the second execution screen being overlaid on a portion of the first execution screen of the first application; An action of receiving a screen capture command through the voice assistant application while the second execution screen is overlaid and displayed on a part of the first execution screen; and A method including an action of performing a screen capture by saving an image of the first execution screen in which the image of the second execution screen is not overlaid on a portion of the first execution screen based on the screen capture command while the second execution screen is overlaid and displayed on a portion of the first execution screen.

Images