Electronic device for providing executable function to selection area of screen, operating method thereof, and recording medium

Abstract

Disclosed are an electronic device for providing a specialized function to a selection area of a screen, and an operating method thereof. The electronic device comprises: one or more processors; and memory including one or more storage media for storing instructions, wherein the instructions, when executed by the one or more processors individually or collectively, may control the electronic device to perform the following operations. The electronic device may receive an execution command for a selection mode from a user. The electronic device may analyze a screen of an application being executed in the electronic device, in response to receiving the execution command. The electronic device may receive a selection command for the analyzed screen from the user. The electronic device may display, on a screen, a selection area determined on the basis of the selection command. The electronic device may suggest, to the user, some of a plurality of functions executable for an object on the basis of the attribute of the object present in the selection area and the type of the selection command.

Description

An electronic device providing an executable function to a selected area of ​​a screen, a method of operating the same, and a recording medium

[0001] Embodiments of the present disclosure relate to an electronic device providing an executable function to a selected area of ​​a screen, a method of operating the same, and a recording medium.

[0002] Conventional "Circle to Search" technology performs a search function by recognizing information drawn on a touchscreen through a user interface on a touchscreen-equipped device. Because this "Circle to Search" technology is limited to search functions, the user experience (UX) it can provide is limited.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] According to one embodiment, an electronic device may include a memory including one or more processors and one or more storage media storing instructions. The instructions, when individually or collectively executed by the one or more processors, may control the electronic device to perform the following operations. The electronic device may receive an execution command for a selection mode from a user. In response to receiving the execution command, the electronic device may analyze a screen of an application running on the electronic device. The electronic device may receive a selection command for the analyzed screen from the user. The electronic device may display a selection area determined based on the selection command on the screen. The electronic device may suggest to the user some of a plurality of functions executable for an object based on attributes of an object existing within the selection area and the type of the selection command.

[0005] According to one embodiment, a method of operating an electronic device may include an operation of entering a screen selection mode for providing a user with a specialized function in response to a control command received through a user interface. The method of operating the electronic device may include an operation of receiving an execution command for the selection mode from a user. The method of operating the electronic device may include an operation of analyzing a screen of an application running on the electronic device in response to receiving the execution command. The method of operating the electronic device may include an operation of receiving a selection command for the analyzed screen from the user. The method of operating the electronic device may include an operation of displaying a selection area determined based on the selection command on a screen. The method of operating the electronic device may include an operation of suggesting to the user some of a plurality of functions executable for an object based on attributes of an object existing within the selection area and the type of the selection command.

[0006] According to one embodiment, instructions stored on a non-transitory computer-readable recording medium, when executed by one or more processors, can cause the electronic device to perform operations of a method of operating the electronic device.

[0007] In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

[0008] FIG. 1A is a block diagram of an electronic device within a network environment according to various embodiments.

[0009] FIG. 1b is an example of a generative artificial intelligence system according to one embodiment.

[0010] FIG. 2 is a diagram illustrating a method for providing an executable function to a selected area of ​​a screen according to one embodiment.

[0011] FIG. 3 is a drawing for explaining a method of entering a selection mode according to one embodiment.

[0012] FIG. 4 is a drawing for explaining a first selection principle in a selection mode according to one embodiment.

[0013] FIG. 5 is a diagram for explaining a second selection principle in a selection mode according to one embodiment.

[0014] FIG. 6 is a diagram for explaining a third selection principle in a selection mode according to one embodiment.

[0015] FIG. 7 is a diagram for explaining multi-input in selection mode according to one embodiment.

[0016] FIGS. 8 to 12 are drawings for explaining a method of displaying a selection area determined based on a selection command according to one embodiment.

[0017] Fig. 13 is a drawing for explaining a method for correcting a selection area according to one embodiment.

[0018] FIGS. 14 to 19 are diagrams for explaining a method of proposing an executable function determined based on the type of an object's properties and selection command according to one embodiment.

[0019] FIGS. 20 to 22 are drawings for explaining a method of tuning a selection area according to one embodiment.

[0020] FIGS. 23 and 24 are diagrams for explaining a method of tuning information included in a selection area according to one embodiment.

[0021] FIG. 25 is a diagram for explaining a method of entering a selection mode supporting multi-input according to one embodiment.

[0022] FIG. 26 is a diagram of a screen write function that provides different levels of functionality based on the type of a selection command in a selection mode for a selected functional tool among a plurality of functional tools according to one embodiment.

[0023] FIG. 27 is a diagram illustrating a method for supporting continuous operation in a multi-input supporting moa calculation mode according to one embodiment.

[0024] Hereinafter, embodiments will be described in detail with reference to the attached drawings. However, the present disclosure may be implemented in various different forms and is not limited to the embodiments described herein. In the description with reference to the attached drawings, identical components will be assigned the same reference numerals regardless of the drawing numbers, and redundant descriptions thereof will be omitted.

[0025] FIG. 1A is a block diagram of an electronic device (101) within a network environment (100) according to various embodiments. Referring to FIG. 1A, in the network environment (100), the electronic device (101) may communicate with the electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (104) or the server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0026] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). The processor (120) may also be implemented as a system on chip (SoC) or an integrated circuit (IC) that performs processing. The processor (120) may include one or more processors, and the operations of the electronic device (101) described in the present disclosure may be performed by a single processor or by a combination of multiple processors. When the operations of the electronic device (101) are performed by a combination of multiple processors, any one processor included in the combination of processors may perform some of the operations of the electronic device (101). For example, the processor (120) may correspond to multiple processors that collectively perform a plurality of operations by dividing them among the processors.

[0027] According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0028] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0029] The memory (130) can store various data used by at least one component (e.g., the processor (120) or the sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., the program (140)) and input data or output data for commands related thereto. The memory (130) can include a volatile memory (132) or a non-volatile memory (134). The memory (130) can store at least one instruction executable by the processor (120). The memory (130) can include one or more memories, and instructions for controlling the processor (120) to perform operations of the electronic device (101) described in the present disclosure can be stored in one memory or can be divided and stored in multiple memories.

[0030] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0031] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0032] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0033] The display module (160) can visually provide information to an external device (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector, and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch. The display module (160) may be implemented with an illustrative foldable structure and / or a rollable structure. For example, the size of the display screen of the display module (160) may be reduced when folded, and may be expanded when unfolded.

[0034] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0035] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0036] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0037] The connection terminal (178) may include a connector that allows the electronic device (101) to be physically connected to an external electronic device (e.g., the electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0038] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0039] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0040] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0041] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0042] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a Wi-Fi communication module, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0043] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0044] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0045] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0046] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0047] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0048] Electronic devices according to various embodiments disclosed in this disclosure may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this disclosure are not limited to the aforementioned devices.

[0049] The various embodiments of the present disclosure and the terminology used therein are not intended to limit the technical features described in the present disclosure to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In the present disclosure, each of the phrases "A or B," "at least one of A and B," "at least one of A or B," "A, B, or C," "at least one of A, B, and C," and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among the phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0050] The term "module" used in various embodiments of the present disclosure may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0051] Various embodiments of the present disclosure may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101) of FIG. 1A). For example, a processor (e.g., a processor (120)) of the machine (e.g., the electronic device (101)) may call at least one command among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0052] According to one embodiment, the method according to various embodiments disclosed in the present disclosure may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

[0053] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and arranged in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

[0054] According to one embodiment, the electronic device (101) may include a memory (130) including one or more processors (120) and one or more storage media storing instructions. The instructions, when individually or collectively executed by the one or more processors (120), may cause the electronic device (101) to receive an execution command for a selection mode from a user, analyze a screen of an application running on the electronic device (101) in response to receiving the execution command, receive a selection command for the analyzed screen from the user, display a selection area determined based on the selection command on the screen, and suggest to the user some of a plurality of functions executable for an object based on attributes of an object existing within the selection area and the type of the selection command.

[0055] According to one embodiment, the instructions, when individually or collectively executed by one or more processors (120), may cause the electronic device (101) to perform paragraph-by-paragraph analysis of text included on the screen and / or object-by-object analysis of images.

[0056] According to one embodiment, the commands, when individually or collectively executed by one or more processors (120), may cause the electronic device (101) to, when a tap type selection command is received, select an image of an object unit or text of a paragraph unit corresponding to a tap position, and display a rectangular area including an image of the selected object unit or a rectangular area including text of the selected paragraph unit as a selection area on the screen.

[0057] According to one embodiment, the commands, when individually or collectively executed by one or more processors (120), may cause the electronic device (101) to, when a draw type selection command is received, display a rectangular area including an area drawn by a draw type selection command as a selection area on the screen.

[0058] According to one embodiment, the instructions, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to identify priorities of each of a plurality of functions executable for an object based on properties of the object and the type of the selection command, and to display execution buttons for a predetermined number of functions in order of highest identified priorities.

[0059] According to one embodiment, the instructions, when individually or collectively executed by one or more processors (120), may cause the electronic device (101) to perform the function of an execution button selected by a selection command when a selection command for the displayed execution button is received from a user.

[0060] According to one embodiment, the commands, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to output a sound asking whether to perform the function of the execution button selected by the selection command.

[0061] According to one embodiment, the commands, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to display a rectangular area of ​​a preset size as a selection area on the screen if no object exists within the screen area corresponding to the selection command.

[0062] According to one embodiment, the commands, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to re-receive a selection command for the analyzed screen from the user if a selection command for an area other than the displayed execution button is received from the user.

[0063] According to one embodiment, the instructions, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to, if an object present within a selection area is text, divide the text into words, identify some missing letters in the word-divided text, and correct the selection area to include some missing letters.

[0064] According to one embodiment, the instructions, when executed individually or collectively by one or more processors (120), may cause the electronic device (101) to, when an object existing within a selection area is an image, divide the image into object units, identify an area not included in the selection area in the image divided into object units, and correct the selection area so that the not included area is included.

[0065]

[0066] FIG. 1b is an example of a generative artificial intelligence system according to one embodiment.

[0067] A User Query / Response Interface (110b) can receive user input. The user input can be any type of input, such as natural language, image, audio, and / or video. Furthermore, context information can also be transmitted when the user input is transmitted. The context information can include various side information related to the time at which the user input is input into the artificial intelligence system (100b). For example, the context information can include information about the application currently being used by the user or information about the user's location. Furthermore, the user input can be a mixed type of input that includes the aforementioned natural language, image, audio, video, and / or context information. Furthermore, the user input can include non-natural language input, such as selecting a menu.

[0068] The user query / response interface (110b) can provide the user with output from the generative artificial intelligence system. The output may include a natural language-based response and / or specific content. The output may also include an action requested by the user.

[0069] The AI ​​framework (120b) can receive user input. Based on the user input (e.g., the user's query), the AI ​​framework (120b) can coordinate and control one or more components necessary to perform an action corresponding to the user's intent.

[0070] User input received from the user query / response interface (110b) can be transmitted to a prompt design component (121b). The prompt design component (121b) can be used to generate a prompt suitable as input to a generative model (e.g., a large language model (LLM) and / or a large multimodal model (LMM)) based on the user input.

[0071] The prompt design component (121b) may be an AI component that utilizes a machine learning algorithm or a neural network. The prompt design component (121b) may generate improved prompts through learning over time. The prompt design component (121b) may access a knowledge repository (130b) to generate prompts based on user input. The knowledge repository (130b) may include user preference data, a prompt library, and / or prompt examples. The prompt design component (123b) may provide the generated prompts to a generative model (e.g., an LLM and / or an LMM).

[0072] The APIs / Plugins management component (123b) can communicate with external information sources based on requests for additional information when user input is transmitted to the generative model.

[0073] The APIs / Plugins management component (123b) can establish a communication channel for external communication of the system (100b) via APIs. The APIs / Plugins management component (123b) can enable access to various data sources via the communication channel. The acquired information can be used to generate prompts by the prompt design component (121b) along with user input, or can be used as input for the generative model (150b).

[0074] The APIs / Plugins management component (123b) can request a final action via an API when the final action in response to user input, rather than an intermediate action, must be performed by an application or service.

[0075] The refiner component (125b) can fine-tune the output of the generative model (150b). For example, the refiner component (125b) can determine the relevance (e.g., a score) between the output of the generative model (e.g., content) and the user input. For example, the refiner component (125b) can determine whether the output contains biased information (e.g., selective information). For example, the refiner component (125b) can determine whether the output contains harmful information (e.g., violent content or profanity).

[0076] The refining component (125b) can determine the degree of matching (e.g., a score) between the output of the generative model (150b) and the user input (e.g., the intent of the user input). If the refining component (125b) determines that the output of the generative model (125b) does not correspond to the user input, the refining component (125b) can modify the output so that it corresponds to the user input.

[0077] The refinement component (125b) can provide hints (e.g., hints for generating prompts) to the user so that the user can obtain information that matches the user's intention from the generative model (150b).

[0078] A generative model (150b) may refer to an artificial intelligence neural network that generates new data (e.g., text, images, audio, or video) based on user input (e.g., user utterances). The generative model (150b) may include an image generation model and / or a language generation model.

[0079] Image generation models may include generative adversarial networks (GANs) and / or variational autoencoders (VAEs). An example of an image generation model is a diffusion-based generative model with a VAE and transformer architecture.

[0080] A language generation model (e.g., ChatGPT) can be a model trained to generate statistically most appropriate output based on input. A language generation model can include an LMM. An LMM can identify various types of input, such as text, images, audio (e.g., speech), and / or video, and generate new data corresponding to the input.

[0081]

[0082] FIG. 2 is a diagram illustrating a method for providing an executable function to a selected area of ​​a screen according to one embodiment. In one embodiment, at least one of the operations of FIG. 2 may be performed simultaneously or in parallel with other operations, and the order between the operations may be changed. In addition, at least one of the operations may be omitted, and other operations may be additionally performed. The operations illustrated in FIG. 2 may be performed by a processor (e.g., processor 120 of FIG. 1A) of an electronic device (e.g., electronic device 101 of FIG. 1A).

[0083] In operation (210), the processor may receive an execution command for a selection mode from a user. At this time, the execution command for the selection mode may be received through at least one user interface. For example, the user interface may include a touch user interface based on gesture input on a touch screen, a voice user interface based on voice input, a gesture interface based on motion recognition using a camera or motion sensor, and a graphical user interface that uses a mouse, keyboard, or trackball as an input method. However, the types of such user interfaces are merely examples and are not limited to the above examples.

[0084] In operation (220), the processor may analyze the screen of an application running on the electronic device (101) in response to receiving an execution command for a selection mode. More specifically, the processor may analyze the screen of the application to identify the type of content contained within the screen. For example, the processor may analyze the screen of the application to identify graphic elements, such as images, text, and videos, contained within the screen.

[0085] More specifically, if the screen contains text, the processor can analyze the text at the paragraph level during screen analysis. Alternatively, if the screen contains images, the processor can analyze the images at the object level during screen analysis. Alternatively, if the screen contains both text and images, the processor can analyze the text at the paragraph level and the images at the object level during screen analysis.

[0086] In one embodiment, the processor may perform analysis on the entire screen of an application running on the electronic device (101). For example, if the screen size of the running application exceeds the size of the display included in the electronic device (101), the processor may also perform analysis on the screen of the application that is not displayed through the display.

[0087] In one embodiment, the processor may perform analysis on a screen area selected by the user from the entire screen of an application running on the electronic device (101) or a screen area preset for each application. For example, if the user selects a portion of the screen to obtain more detailed information on specific content running through the application, the processor may perform a more precise and detailed analysis on the selected portion of the screen.

[0088] In operation (230), the processor may receive a selection command for the analyzed screen from the user. More specifically, the processor may receive a tap-type selection command or a draw-type selection command performed by the user on the analyzed screen. According to one embodiment, the processor may receive a tap input in which the user briefly touches the screen using a finger or a stylus on the analyzed screen as a selection command. In this case, the processor may identify text in paragraph units or images in object units existing on the analyzed screen in response to such a tap input.

[0089] In one embodiment, the processor may receive a tap-and-hold input, in which the user's finger or stylus is used to touch the analyzed screen for a predetermined period of time, as a selection command. In response to this tap-and-hold input, the processor may identify a primary subject within an image of an object unit present on the analyzed screen.

[0090] For example, if a tap-and-hold input is identified for an image of an object unit existing on the analyzed screen, the processor can identify the main subject within the image of the object unit by performing an image cutout function that removes the background and separates the main subject.

[0091] In one embodiment, the processor may receive drawing inputs continuously received via a user's finger or stylus on the analyzed screen as selection commands. In response to such drawing inputs, the processor may identify a drawing area displayed on the analyzed screen.

[0092] In operation (240), the processor may display a selection area determined based on a selection command on the screen. According to one embodiment, the processor may determine a minimum rectangular area surrounding a text in a paragraph unit or an image in an object unit identified through a tap input as the selection area and display the determined selection area on the screen. Alternatively, the processor may determine an internal area of ​​a main subject within an image in an object unit identified through a tap-and-hold input as the selection area and display the determined selection area on the screen.

[0093] According to one embodiment, the processor may display a selection area determined based on the shape of the drawing area corresponding to the drawing input on the screen. For example, in a screen containing text, if a drawing area in the form of a movement path or a drawing area in the form of a shape is identified in response to continuously received drawing input, the processor may determine a rectangular area containing the text included in the identified drawing area in line units as the selection area and display it on the screen.

[0094] According to one embodiment, for a screen including an image, if a drawing area in the form of a movement path is identified in response to continuously received drawing inputs, the processor may determine a minimum rectangular area surrounding the identified drawing area as a selection area and display the area on the screen. Alternatively, for a screen including an image, if a drawing area in the form of a shape or a free-form drawing area including an inner area is identified in response to continuously received drawing inputs, the processor may determine an inner area of ​​the identified drawing area as a selection area and display the area on the screen.

[0095] Meanwhile, if there is no object within the screen area corresponding to the selection command, the processor can display a rectangular area of ​​a preset size as a selection area on the screen.

[0096] In operation (250), the processor may identify at least some of a plurality of functions executable for an object based on the properties of the object present in the selection area and the type of the selection command. The processor may identify the priority of each of the identified functions, display execution buttons for a preset number of functions based on the identified priorities, and display additional buttons for the remaining functions. In this case, the priority of each of the plurality of functions executable for the object may be determined by applying different weights depending on the importance or technical completeness of the functions.

[0097]

[0098]

[0099] Table 1 above shows examples of executable functions corresponding to object properties. However, these examples are merely examples and are not limited to the examples above.

[0100] According to one embodiment, the processor may display execution buttons for a preset number of functions, in descending order of priority, based on the priorities of the functions identified in this manner. The execution buttons may be displayed in an area adjacent to the selection area displayed on the screen; however, the display area of ​​these execution buttons is merely an example and is not limited to the above example.

[0101] According to one embodiment, when a selection command for a displayed execution button is received from a user, the processor may perform an action for the function of the selected execution button. In this case, before performing the action for the function of the selected execution button, the processor may output a sound asking whether to perform the function of the selected execution button.

[0102] Meanwhile, if the processor receives a selection command from the user for an area other than the displayed execution button, it may return to operation (230) and re-receive a selection command for the analyzed screen from the user.

[0103] According to one embodiment, a method of operating an electronic device (101) may include an operation of receiving an execution command for a selection mode from a user. The method of operating may include an operation of analyzing a screen of an application running on the electronic device (101) in response to receiving the execution command. The method of operating may include an operation of receiving a selection command for the analyzed screen from the user. The method of operating may include an operation of displaying a selection area determined based on the selection command on a screen. The method of operating may include an operation of suggesting to the user some of a plurality of functions executable for an object based on attributes of an object existing within the selection area and the type of the selection command.

[0104] The analyzing action may include performing paragraph-by-paragraph analysis on text contained on the screen and / or object-by-object analysis on images.

[0105] The displaying action may include an action of selecting an image of an object unit or text of a paragraph unit corresponding to a tap position when a selection command of the tap type is received. The displaying action may include an action of displaying a rectangular area including an image of the selected object unit or a rectangular area including text of the selected paragraph unit as a selection area on the screen.

[0106] The displaying action may include an action of displaying a rectangular area including an area drawn by a draw type selection command as a selection area on the screen when a draw type selection command is received.

[0107] The proposed action may include identifying the priority of each of a plurality of executable functions for the object based on the object's properties and the type of the selection command. The proposed action may include displaying execution buttons for a predetermined number of functions in order of the highest identified priority.

[0108] The operating method of the electronic device (101) may further include an operation of performing a function of the execution button selected by the selection command when a selection command for the execution button displayed above is received from the user. The operating method may further include an operation of outputting a sound inquiring whether to perform the function of the execution button selected by the selection command.

[0109] The displaying action may include displaying a rectangular area of ​​a preset size as a selection area on the screen when no object exists within the screen area corresponding to the selection command.

[0110] The method of operating the electronic device (101) may further include an operation of re-receiving a selection command for the analyzed screen from the user when a selection command for an area other than the displayed execution button is received from the user.

[0111]

[0112] FIG. 3 is a diagram illustrating a method for entering a selection mode according to one embodiment. The operations illustrated in FIG. 3 may be performed by a processor (e.g., processor (120) of FIG. 1a) of an electronic device (e.g., electronic device (101) of FIG. 1a).

[0113] Referring to FIG. 3, the processor may enter a selection mode through an interface element or stylus input present in a portion of the screen. According to one embodiment, when a drag-and-drop (or drag) function is executed on an edge panel present in a portion of the screen, as shown in the drawing (310), the processor may display at least one functional tool, as shown in the drawing (320).

[0114] For example, the processor may display functional tools related to sketch transformation (321), smart select (322), translation (323), and interpretation (324) on the screen, such as the drawing (320). The functional tool related to sketch transformation (321) may provide a function that allows a user to add or modify a sketch to an image or graphic selected by the user on the screen, and the functional tool related to smart select (322) may provide a function that allows a user to select text, an image, or a specific area on the screen. In addition, the functional tool related to translation (323) may provide a function that translates text selected by the user into another language in real time, and the functional tool related to interpretation (324) may provide a function that translates voice data selected by the user in real time and outputs it in another language.

[0115] However, the types and number of functional tools displayed on the screen are only examples and are not limited to the above examples.

[0116] When one of the functional tools displayed on the screen is selected, the processor may enter a selection mode for the selected functional tool, as shown in drawing (330). For example, drawing (330) shows an example in which a functional tool corresponding to smart select is selected among the functional tools and enters the selection mode.

[0117] According to one embodiment, when a stylus input (341) (e.g., air command) using a stylus is detected as in the drawing (340) and one of the multiple functional tools presented as in the drawing (350) is selected (e.g., smart select), the processor may enter a selection mode corresponding to the smart select as in the drawing (330). Meanwhile, referring to the drawing (340), a functional tool related to addition (351) may be displayed on the screen. Such a functional tool related to addition (351) may provide a function that allows a user to insert a new functional tool into the screen.

[0118] However, the method of entering the selection mode is only one example and is not limited to the above example.

[0119]

[0120] FIG. 4 is a diagram illustrating a first selection principle in a selection mode according to one embodiment. The operations illustrated in FIG. 4 may be performed by a processor (e.g., processor (120) of FIG. 1A) of an electronic device (e.g., electronic device (101) of FIG. 1A).

[0121] Referring to FIG. 4, a tab-type selection command can support easy selection of an object existing on the screen. According to one embodiment, the processor can analyze the screen of an application running on the electronic device (101) in response to entering the selection mode as shown in the drawing (410). At this time, if the screen contains text, the processor can analyze the text in paragraph units when analyzing the screen. Alternatively, if the screen contains an image, the processor can analyze the image in object units when analyzing the screen. For example, if the processor analyzes the image in object units, the processor can identify the entire image as a single upper object and identify each foreground object separated from the entire image as a lower object.

[0122] According to one embodiment, when a tap input (411) is detected for a sub-object among a plurality of objects existing on the screen, the processor may determine a minimum rectangular area surrounding the sub-object as a selection area (421) with priority over the upper object, as shown in the drawing (420). When displaying the selection area (421) determined in this way on the screen, the processor may provide a graphical user interface (GUI) (e.g., a handler) for resizing an area (e.g., a vertex, a corner) of the selection area (421).

[0123] In addition, the processor may select at least one function that can be provided for an object within the selection area (421) and display an execution button on the screen for executing the selected function. For example, the processor may display execution buttons related to AI drawing (423), Make sticker (424), and a toolbar (425) on the screen, as in the drawing (420). The execution button related to AI drawing (423) may provide a function for automatically generating a drawing using artificial intelligence or for correcting and completing a sketch drawn by a user. The execution button related to Make sticker (424) may provide a function for converting an image displayed on the screen into a sticker form. The toolbar (425) may include an execution button for providing at least one additional function (e.g., copy, share, save).

[0124] However, the types and number of execution buttons displayed on the screen are only an example and are not limited to the above example.

[0125] According to one embodiment, when a tap input (412) is detected for an upper object among a plurality of objects present on the screen, the processor may determine a minimum rectangular area surrounding the upper object as a selection area (431), as shown in the drawing (430). Similarly, when displaying the selection area (431) determined in this way on the screen, the processor may provide a graphical user interface for resizing a portion of the selection area (431).

[0126] According to one embodiment, when a tap input (422) is additionally detected for a selection area (421) determined in response to a lower-level object, the processor may expand the selection area to a selection area (431) of a higher-level object that includes the lower-level object. When the same tap input (422) is additionally detected while the selection area is expanded in this manner, the processor may also reduce the selection area to a selection area (421) determined in response to a previous lower-level object.

[0127] Meanwhile, if a tap input (422) is additionally detected for a selection area (421) determined in response to a lower object, instead of expanding the selection area to a selection area (431) corresponding to the upper object, the processor may additionally display on the screen the selection areas of lower objects adjacent to the selection area (421). If such tap inputs (422) are additionally detected continuously, the processor may add selection areas to the screen in the order of lower objects adjacent to the selection area (421) in the order of the shortest distance. The selection areas of adjacent foreground objects may also be continuously activated and displayed in order of proximity.

[0128]

[0129] FIG. 5 is a diagram illustrating a second selection principle in a selection mode according to one embodiment. The operations illustrated in FIG. 5 may be performed by a processor (e.g., processor (120) of FIG. 1a) of an electronic device (e.g., electronic device (101) of FIG. 1a).

[0130] Referring to FIG. 5, a drawing type selection command can support specific selections reflecting the user's intent. According to one embodiment, the processor can analyze the screen of an application running on the electronic device (101) in response to entering selection mode, as shown in FIG. 510.

[0131] According to one embodiment, when a drawing area is detected through a stylus input (511, 512), the processor may determine a minimum rectangular area surrounding the drawing area, such as a drawing (520, 530), as a selection area (521, 531). When displaying the selection area (521, 531) determined in this manner on a screen, the processor may provide a graphical user interface for resizing a portion of the selection area (521, 531).

[0132] Additionally, the processor may select at least one function that can be provided for an object within the selection area (521, 531) and display an execution button on the screen for executing the selected function. For example, the processor may display execution buttons related to AI drawing (522), Make sticker (523), and a toolbar (524) on the screen as shown in the drawing (520). These execution buttons may provide the same function as the execution buttons provided in FIG. 4 (e.g., AI drawing (423), Make sticker (424), and toolbar (425)).

[0133]

[0134] FIG. 6 is a diagram illustrating a third selection principle in a selection mode according to one embodiment. The operations illustrated in FIG. 6 may be performed by a processor (e.g., processor (120) of FIG. 1A) of an electronic device (e.g., electronic device (101) of FIG. 1A).

[0135] Referring to FIG. 6, a selection command of a tap type or a draw type can provide consistent selection logic for images and / or text existing on the screen. According to one embodiment, the processor can analyze the screen of an application running on the electronic device (101) in response to entering a selection mode, such as in FIG. (610) and FIG. (640).

[0136] According to one embodiment, when a tap input (611) is detected for an object that is an image of one of a plurality of objects present on the screen, the processor may determine a minimum rectangular area surrounding the image of the object unit as a selection area and display the same on the screen, as in the drawing (620). Conversely, when a drawing area is detected through a stylus input (612), the processor may determine a minimum rectangular area surrounding the drawing area as a selection area and display the same on the screen, as in the drawing (630).

[0137] According to one embodiment, when a tap input (641) is detected for an object that is text among a plurality of objects present on the screen, the processor may determine a minimum rectangular area surrounding the text in sentence units as a selection area and display the same on the screen, as in the drawing (650). Conversely, when a drawing area is detected through a stylus input (642, 643), the processor may determine a rectangular area including the text included in the drawing area in line units as a selection area and display the same on the screen, as in the drawing (660).

[0138] In addition, the processor may select at least one function that can be provided for an object within the selection area, and display an execution button on the screen for executing the selected function. For example, the processor may display execution buttons related to AI drawing (651), Convert unit (652), add to note (653), and a toolbar (654) on the screen, as in the drawing (620). The execution button related to Convert unit (652) may provide a function that allows a user to automatically convert a selected number or unit value into another unit. The execution button related to Add to note (653) may provide a function that allows a user to save selected text, images, or other content to a note application. The remaining execution buttons related to AI drawing (651) and the toolbar (654) may provide the same function as the execution buttons provided in FIG. 4 (e.g., AI drawing (423) and the toolbar (425)).

[0139]

[0140] FIG. 7 is a diagram illustrating multi-input in a selection mode according to one embodiment. The operations illustrated in FIG. 7 may be performed by a processor (e.g., processor (120) of FIG. 1A) of an electronic device (e.g., electronic device (101) of FIG. 1A).

[0141] The above-described FIGS. 4 to 6 provide a method for displaying a selection area in a selection mode for a single input. In contrast, FIG. 7 may provide a method for displaying a selection area in a screen selection mode for multiple inputs. According to one embodiment, the processor may analyze the screen of an application running on the electronic device (101) in response to entering the selection mode, as shown in the drawing (710).

[0142] According to one embodiment, the processor may provide a function button (e.g., a pin button) (711) in an area of ​​the screen to perform a multi-input operation. More specifically, when a tap input (e.g., 712, 713) or a stylus input for two or more objects among a plurality of objects present on the screen is continuously detected while the function button (711) is activated, the processor may sequentially display on the screen selected areas (721, 722) determined in response to the objects, as shown in the drawing (720).

[0143] However, the form of the function button (711) provided to perform such multi-input operation is only one example and is not limited to the above example.

[0144] Additionally, the processor may select at least one function that can be provided for an object within the selection area (721, 722) and display an execution button on the screen for executing the selected function. For example, the processor may display execution buttons related to AI drawing (723), Make sticker (724), and a toolbar (725) on the screen as shown in the drawing (720). These execution buttons may provide the same function as the execution buttons provided in FIG. 4 (e.g., AI drawing (423), Make sticker (424), and toolbar (425)).

[0145]

[0146] FIGS. 8 to 12 are diagrams for explaining a method of displaying a selection area determined based on a selection command according to one embodiment. The operations illustrated in FIGS. 8 to 12 may be performed by a processor (e.g., the processor (120) of FIG. 1A) of an electronic device (e.g., the electronic device (101) of FIG. 1A).

[0147] According to one embodiment, when a tap input is detected for an object that is text existing on the screen while the selection mode is entered as shown in drawing (810), the processor may select a paragraph of text corresponding to the tap input as shown in drawing (820). The processor may determine a minimum rectangular area surrounding the text of the selected paragraph as a selection area and display it on the screen.

[0148] According to one embodiment, when a tap input for an object, which is an image existing on the screen, is detected while entering the selection mode as shown in drawing (830), the processor may select an image of an object unit corresponding to the tap input as shown in drawing (840). The processor may determine a minimum rectangular area surrounding the image of the selected object unit as a selection area and display it on the screen.

[0149] According to one embodiment, when a drawing input for the same line of an object that is text existing on the screen is detected while entering a selection mode as in drawing (910), the processor may select the text in units of words included in the corresponding line as in drawing (920). The processor may determine the minimum rectangular area surrounding the text in units of words selected as a selection area and display it on the screen.

[0150] According to one embodiment, when a tap-and-hold input is detected for an object that is an image on the screen while the selection mode is entered as shown in drawing (930), the processor may select a main subject from which the background has been removed in the image in response to the tap-and-hold input as shown in drawing (940). The processor may determine an internal area of ​​the main subject selected as such as a selection area and display it on the screen.

[0151] According to one embodiment, when a drawing input for an object that is text existing on the screen is detected while entering a selection mode as in drawing (1010), the processor may select text corresponding to the drawing input line by line as in drawing (1020). The processor may determine a minimum rectangular area surrounding the text of the selected line by line as a selection area and display it on the screen.

[0152] According to one embodiment, when a drawing input for an object that is an image existing on the screen is detected while entering a selection mode as in drawing (1030), the processor can identify a drawing area corresponding to the drawing input as in drawing (1040). The processor can determine a minimum rectangular area surrounding the drawing area identified as such as a selection area and display it on the screen.

[0153] According to one embodiment, when a drawing input in the form of a shape for an object that is text existing on the screen is detected while entering a selection mode as in drawing (1110), the processor can select text corresponding to the drawing input line by line as in drawing (1120). The processor can determine the minimum rectangular area surrounding the text of the selected line by line as a selection area and display it on the screen.

[0154] According to one embodiment, when a drawing input in the form of a shape for an object that is an image existing on the screen is detected while entering a selection mode as in drawing (1130), the processor can identify a shape corresponding to the drawing input as in drawing (1140). The processor can determine an internal area of ​​the shape identified in this way as a selection area and display it on the screen.

[0155] According to one embodiment, when drawing inputs for text objects and images existing on the screen are detected simultaneously in a selection mode as in drawing (1210), the processor can identify drawing areas of text and images in units of lines corresponding to the drawing inputs as in drawing (1220). The processor can determine the minimum rectangular area surrounding the drawing areas of the text and images in units of lines identified in this way as a selection area and display it on the screen.

[0156] According to one embodiment, when a free-form drawing input for an object that is an image existing on the screen is detected while the selection mode is entered as in drawing (1230), the processor may determine a free-form drawing area corresponding to the drawing input as a selection area and display it on the screen as in drawing (1240). Alternatively, the processor may identify a main subject from which the background has been removed by performing clipping based on the free-form drawing area corresponding to the drawing input. The processor may also determine an internal area of ​​the main subject identified in this way as a selection area and display it on the screen.

[0157]

[0158] Fig. 13 is a diagram for explaining a method for correcting a selection area according to one embodiment. The operations illustrated in Fig. 13 may be performed by a processor (e.g., the processor (120) of Fig. 1a) of an electronic device (e.g., the electronic device (101) of Fig. 1a).

[0159] According to one embodiment, when a drawing input is detected for the same line of an object that is text existing on the screen while entering a selection mode as in drawing (1310), the processor may select the text of a word unit included in the corresponding line as in drawing (1320). The processor may determine and display the minimum rectangular area surrounding the text of the selected word unit as the selection area. However, when some missing letters are identified as a result of analysis of the text within the selection area as in drawing (1320), the processor may correct the selection area so that some missing letters are included as in drawing (1330).

[0160] According to one embodiment, when a drawing input for an object that is an image existing on the screen is detected while entering a selection mode as in drawing (1340), the processor can identify a drawing area corresponding to the drawing input as in drawing (1350). The processor can determine a minimum rectangular area surrounding the drawing area identified as such as a selection area and display it on the screen. However, when an area not included in the selection area among the object unit images is identified as a result of analyzing the image within the selection area as in drawing (1350), the processor can correct the selection area so that the not included area is included as in drawing (1360).

[0161]

[0162] FIGS. 14 to 19 are diagrams for explaining a method for proposing an executable function determined based on the properties of an object and the type of a selection command according to one embodiment. The operations illustrated in FIGS. 14 to 19 may be performed by a processor (e.g., the processor (120) of FIG. 1A) of an electronic device (e.g., the electronic device (101) of FIG. 1A).

[0163] According to one embodiment, the processor may suggest at least one executable function to the user based on the properties of an object within a selection area displayed on the screen and the type of the selection command. More specifically, the processor may identify the priority of each of a plurality of executable functions for the object based on the properties of the object and the type of the selection command. Based on the priorities of the functions thus identified, the processor may display execution buttons for a preset number of functions in order of highest priority.

[0164] According to one embodiment, when an object that is an image exists within a selection area, such as a drawing (1410), the processor may select and display on the screen execution buttons for a preset number of functions with the highest priority among executable functions corresponding to the object that is an image (e.g., AI Drawing (1411) and Make sticker (1412)). In this case, the selected execution buttons may be displayed in an area adjacent to the selection area, but the display area of ​​such execution buttons is only one example and is not limited to the above example.

[0165] The processor may perform an action (e.g., activating the AI ​​Drawing panel) corresponding to the selected execution button when any one of the execution buttons of the proposed functions (e.g., AI Drawing (1411)) is selected, as in the drawing (1420).

[0166] For example, if an image of sunglasses is drawn using a user interface on an activated AI Drawing panel, the processor may receive an image of a dog wearing sunglasses from an AI model connected to the AI ​​Drawing panel and output it on the activated AI Drawing panel. Such operations of the AI ​​model may be performed directly on-device on the electronic device (101) or on an external server.

[0167] According to one embodiment, when a text object exists within a selection area, as shown in the drawing (1510), the processor may select and display on the screen execution buttons for a preset number of functions with the highest priority among executable functions corresponding to the text object. However, the processor may classify object properties through analysis of the text object within the selection area, and may preferentially suggest executable functions corresponding to the classified object properties.

[0168] For example, if the analysis results for an object that is text within a selection area identify that the object property of the text is song lyrics, the processor may preferentially suggest an executable function corresponding to the song lyrics. More specifically, the processor may preferentially suggest to the user an execution button (e.g., “Play music (1511)”) for a function related to song lyrics among the executable functions corresponding to the text object, as shown in the drawing (1510).

[0169] When the suggested execution button is selected by the user, the processor can play the sound source corresponding to the song lyrics by performing an action corresponding to the selected execution button (e.g., executing a music app) as shown in the drawing (1520).

[0170] According to one embodiment, when an image object and a text object exist simultaneously within a selection area, as in the drawing (1610), the processor may select and display on the screen execution buttons for a preset number of functions with the highest priority among executable functions corresponding to the image object and the text object.

[0171] However, the processor can classify object properties through analysis of objects that are images and text objects within the selection area, and can preferentially provide executable functions corresponding to the classified object properties.

[0172] For example, if the analysis results for a text object within a selection area identify the text as a foreign language describing an image within the selection area, the processor may preferentially suggest an executable function corresponding to the identified foreign language. More specifically, the processor may preferentially suggest to the user an execution button (e.g., Translate (1611)) for a function related to the foreign language among the executable functions corresponding to the text object, as shown in the drawing (1610).

[0173] When the suggested execution button is selected by the user, the processor can overlay the translation of the foreign language by performing an action corresponding to the selected execution button (e.g., translating an overlay image), as shown in the drawing (1620).

[0174] Additionally, the processor may display on the screen an execution button for a function that allows selection of text (e.g., text extraction (1612)) if the text object within the selection area exists in the form of an image.

[0175] According to one embodiment, when a text object exists within a selection area, as shown in the drawing (1710), the processor may select and display on the screen execution buttons for a preset number of functions with the highest priority among executable functions corresponding to the text object. However, the processor may classify object properties through analysis of the text object within the selection area, and may preferentially provide executable functions corresponding to the classified object properties.

[0176] For example, if the analysis result for a text object within a selection area indicates that the text is longer than a certain length, the processor may preferentially suggest an executable function corresponding to the text longer than the certain length. More specifically, the processor may preferentially suggest to the user an execution button (e.g., Summarize (1711)) for a function related to text longer than the certain length among the executable functions corresponding to the text object, as shown in the drawing (1710).

[0177] When the suggested execution button is selected by the user, the processor can provide the user with a summary of the text longer than a certain length by performing an action (e.g., summary) corresponding to the selected execution button as shown in the drawing (1720). At this time, the processor can provide the summary, which is the result of the execution of the execution button, through a panel or sheet (e.g., Bottom sheet (1721)) that appears as a slide at the bottom of the screen. However, the method of displaying the result of the execution of the execution button is only one example and is not limited to the above example.

[0178] According to one embodiment, when a text object exists within a selection area, as shown in the drawing (1810), the processor may select and display on the screen execution buttons for a preset number of functions with the highest priority among executable functions corresponding to the text object. However, the processor may classify object properties through analysis of the text object within the selection area, and may preferentially provide executable functions corresponding to the classified object properties.

[0179] For example, if the analysis results for a text object within a selection area reveal that the text is a place name indicating a specific location, the processor may preferentially provide an executable function corresponding to the place name indicating the specific location. More specifically, the processor may preferentially provide the user with an execution button (e.g., Map (1811)) for a function related to the place name indicating the specific location among the executable functions corresponding to the text object, as shown in the drawing (1810).

[0180] When the suggested execution button is selected by the user, the processor can provide the user with a result of searching for the location of a specific place on a map by performing an action corresponding to the selected execution button (e.g., opening a map) as shown in the drawing (1820).

[0181] According to one embodiment, the drawing (1910) shows the result of displaying a selection area for text less than a certain length through stylus input. At this time, if the text included in the selection area is a foreign language less than the certain length, the processor may preferentially suggest to the user an execution button (e.g., Translate (1911)) of a function related to a foreign language among executable functions corresponding to the text object, and display the result of executing the execution button.

[0182] In contrast, the drawing (1920) shows the result of a selection area being displayed for text longer than a certain length through stylus input. At this time, if the text included in the selection area is a foreign language longer than a certain length, the processor can preferentially suggest to the user an execution button (e.g., Summarize (1921)) related to text longer than a certain length among the executable functions corresponding to the text object, and display the result of the execution of the execution button.

[0183] That is, the processor can distinguish and select the types of executable functions to suggest to the user based on the type of the various selection commands (e.g., short foreign language text or long foreign language text), even if the object properties within the selection area are similar (e.g., foreign language text), thereby providing the user with an optimal user experience.

[0184] While the above example provides a method for proposing executable functions based on object attributes and the type of selection command, this method for proposing executable functions is merely an example and is not limited to the above example. For example, the processor may prioritize proposing the user's most frequently used function or the function most recently used over other functions.

[0185]

[0186] FIGS. 20 to 22 are diagrams for explaining a method of tuning a selection area according to one embodiment. The operations illustrated in FIGS. 20 to 22 may be performed by a processor (e.g., the processor (120) of FIG. 1A) of an electronic device (e.g., the electronic device (101) of FIG. 1A).

[0187] According to one embodiment, when a drawing input for an object that is text existing on the screen is detected while entering a selection mode, as in drawing (2010), the processor may select text corresponding to the drawing input line by line, as in drawing (2020). The processor may determine a minimum rectangular area surrounding the text of the selected line by line as a selection area and display it on the screen.

[0188] In response to inputting text within the selected area as indicated above into the AI ​​model, the processor can analyze the text line by line, dividing it into sentences. Based on the analysis results, the processor can correct the selected area by deleting sentence elements (2031) determined to be unnecessary by the AI ​​model and adding sentence elements (2032) determined to be missing.

[0189] According to one embodiment, when a drawing input for an object that is text existing on the screen is detected while entering a selection mode as in drawing (2110), the processor can select text corresponding to the drawing input in units of lines as in drawing (2120). The processor can determine a minimum rectangular area surrounding the text of the selected line unit as a selection area (2121) and display it on the screen.

[0190] According to one embodiment, in response to inputting text within the selection area (2121) displayed as such into the AI ​​model, the processor may analyze whether content (2122) related to the text within the selection area (2121) exists in an unexposed screen of the running application. If the processor determines, based on the analysis result of the AI ​​model, that content (2122) related to the text within the selection area (2121) exists in an unexposed screen of the running application, the processor may correct the selection area (2121) so that the related content (2122) existing in the unexposed screen is included within the selection area (2121).

[0191] According to one embodiment, when a drawing input for the same line of an object that is text existing on the screen is detected while entering a selection mode as in drawing (2210), the processor may determine a minimum rectangular area surrounding the text in word units included in the corresponding line as a selection area and display it on the screen as in drawing (2220).

[0192] In response to inputting text within the selection area displayed in this manner into the AI ​​model, the processor can analyze whether there is content related to the text within the selection area on the screen of the running application.

[0193] If content related to the text within the selection area exists on the screen of the running application, the processor can change the properties of the text corresponding to the related content and display it, as shown in drawing (2230). In the example of FIG. 22, content related to the text within the selection area is highlighted, but the method of changing the properties of such related content is merely an example and is not limited to the above example.

[0194] When text with changed properties is selected through a user interface, the processor can additionally determine a minimum rectangular area surrounding the selected text as a selection area and display it on the screen, as shown in the drawing (2240).

[0195]

[0196] FIGS. 23 and 24 are diagrams illustrating a method for tuning information included in a selection area according to one embodiment. The operations illustrated in FIGS. 23 and 24 may be performed by a processor (e.g., the processor (120) of FIG. 1A) of an electronic device (e.g., the electronic device (101) of FIG. 1A).

[0197] According to one embodiment, when a drawing input for an object that is text and an object that is an image existing on the screen are detected simultaneously in a state where selection mode has been entered as in drawing (2310), the processor can activate and display a selection area corresponding to the drawing input as in drawing (2320).

[0198] If an execution button corresponding to a document editing function is suggested and selected among the functions that can be suggested for the information within the displayed selection area, the processor can edit the information within the selection area in a similar form by considering the arrangement of text and images within the selection area as well as the font and size of the text in response to the AI ​​model input.

[0199] According to one embodiment, when a drawing input for an object that is text existing on the screen is detected while entering a selection mode as in drawing (2410), the processor can determine a selection area corresponding to the drawing input as in drawing (2420) and display it on the screen.

[0200] If an execution button corresponding to an information sharing function among the suggested functions for the text within the displayed selection area is suggested and selected, the processor can identify components that are determined to be unnecessary based on the sharing target and sharing purpose in response to inputting the text within the selection area into the AI ​​model.

[0201] The processor can provide the user with an optimal user experience by deleting components deemed unnecessary, as shown in the drawing (2430), and performing paragraph division processing at an appropriate location according to the content.

[0202]

[0203] FIG. 25 is a diagram illustrating a method for entering a selection mode supporting multi-input according to one embodiment. The operations illustrated in FIG. 25 may be performed by a processor (e.g., processor (120) of FIG. 1a) of an electronic device (e.g., electronic device (101) of FIG. 1a).

[0204] The embodiment of FIG. 7 provides a configuration for performing multi-input via function buttons present in one area of ​​the screen. Conversely, the processor may provide a method for entering a selection mode that supports multi-input by default without a separate function button.

[0205] Referring to FIG. 25, the processor may enter a selection mode that supports multi-input through an interface element or stylus input existing in an area of ​​the screen. According to one embodiment, when a drag-and-drop (or drag) function is executed on an edge panel existing in an area of ​​the screen, such as in the drawing (2510), and one of the tools supporting multi-input presented as in the drawing (2520) (e.g., AI select (2521), Translate (2522), Summarize (2523), AI drawing (2524)) is selected, the processor may enter a selection mode that supports multi-input.

[0206] According to one embodiment, when a stylus input (2531) (e.g., an air command) using a stylus is detected as in the drawing (2530) and one of the tools supporting multiple inputs presented as in the drawing (2540) is selected, the processor may enter a selection mode supporting multiple inputs.

[0207] However, the method of entering the selection mode that supports such multi-input is only one example and is not limited to the above example.

[0208]

[0209] FIG. 26 is a diagram illustrating a screen write function that provides different levels of functionality based on the type of a selection command in a selection mode for a selected functional tool among a plurality of functional tools according to one embodiment. The operations illustrated in FIG. 26 may be performed by a processor (e.g., the processor (120) of FIG. 1A) of an electronic device (e.g., the electronic device (101) of FIG. 1A).

[0210] Referring to FIG. 26, a processor may provide a method for providing different levels of functionality depending on the type of selection command for text when a summary mode supporting multi-input is selected among multiple functional tools and the processor enters a selection mode. However, the type of selected functional tool is merely an example and is not limited to the above example.

[0211] According to one embodiment, the processor may determine a different level of summary depending on the speed of text selection when entering a selection mode by selecting a summarizing mode that supports multiple inputs, as shown in the drawing (2610). For example, if the speed of text selection is slow according to a text selection command for individual lines, as shown in the upper image of the drawing (2610), the processor may summarize the selected text at a long level. Conversely, if the speed of text selection is fast according to a text selection command using a single line, as shown in the lower image of the drawing (2610), the processor may summarize the selected text at a short level.

[0212] According to one embodiment, the processor may determine the degree of summary differently depending on the number of selections of text when entering the selection mode by selecting a summarizing mode that supports multiple inputs, as shown in the drawing (2620). For example, as shown in the upper image of the drawing (2620), the processor may summarize the selected text longer as the number of selections of text in the selection area increases. Conversely, as shown in the lower image of the drawing (2620), the processor may summarize the selected text shorter as the number of selections of text in the selection area decreases.

[0213] According to one embodiment, when the processor enters the selection mode with a summarize mode that supports multiple inputs selected and text is underlined, the processor may summarize the selected text by emphasizing the font of the specific underlined text. Alternatively, when text is underlined in the selection area, the processor may determine the summary length of the selected text differently depending on the thickness of the underline. For example, the processor may summarize the selected text longer as the underline thickness increases, and summarize the selected text shorter as the underline thickness decreases.

[0214] According to one embodiment, when the processor enters the selection mode by selecting a summary mode that supports multiple inputs, such as in the drawing (2630), and some words of the selected text have symbols drawn on them, the processor may determine a different summary form for the selected text. For example, when some words of the selected text have shapes drawn on them, such as stars, caution signs, and numbers, as in the drawing (2630), the processor may summarize the selected text by adding bullets and emphasizing the font. At this time, if there is a topic related to some words with shapes drawn on them in the selected text, the processor may also summarize the text of the topic at length and in detail.

[0215] According to one embodiment, when the processor enters the selection mode by selecting a summarize mode that supports multi-input, such as the drawing (2640), and when characters are drawn in an adjacent area of ​​the selected text or highlighting (e.g., circles) are drawn for some words in the selected text, the processor may summarize the selected text using the characters drawn in an adjacent area of ​​the selected text or some highlighted words in the selected text as keywords.

[0216] Meanwhile, the processor may summarize the selected text longer the longer the user stays on the screen in the summary mode before selecting the text, and may summarize the selected text shorter the shorter the user stays on the screen.

[0217] However, the method of summarizing selected texts with different functional levels depending on the type of selection command in the summarize mode that supports such multi-input is only one example and is not limited to the above example.

[0218]

[0219] FIG. 27 is a diagram illustrating a method for supporting continuous operations in a multi-input, multi-calculation mode according to one embodiment. The operations illustrated in FIG. 27 may be performed by a processor (e.g., processor (120) of FIG. 1A) of an electronic device (e.g., electronic device (101) of FIG. 1A).

[0220] In selection modes that require multiple inputs, it may be difficult for users to remember exactly what they selected previously, and especially when multiple inputs are required sequentially, the order of the multiple inputs may become confusing.

[0221] In one embodiment, when the processor detects a stylus input (e.g., a hovering input) while in a multi-input supporting moa calculation mode, the processor may display previous selections to provide information about what the user previously selected or what multi-input the user should select in the current sequence.

[0222] For example, if a text (2711) of a specific content is selected in the previous order, such as in the drawing (2710), the processor can display the text (2711) of the specific content selected in the previous order in an area adjacent to the hovering area when the stylus (2721) is hovered before selecting the multi-input of the current order, such as in the drawing (2720).

[0223] Supporting continuous operations in a multi-input calculation mode like this can be particularly useful when multiple inputs need to be selected over time, or when multiple selections are required, such as through app switching.

[0224]

[0225] The embodiments of the present invention disclosed in this specification and drawings are merely specific examples presented to easily explain the technical contents according to the embodiments of the present invention and to help understand the embodiments of the present invention, and are not intended to limit the scope of the embodiments of the present invention. Therefore, the scope of the embodiments of the present invention should be interpreted as including all changes or modified forms derived based on the technical idea of ​​the embodiments of the present invention in addition to the embodiments disclosed herein.

Claims

1. In an electronic device (101), one or more processors (120); and A memory (130) comprising one or more storage media for storing commands. Including, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: Receives a command to execute a selection mode from the user, In response to receiving the above execution command, analyze the screen of the application running on the electronic device (101), Receive a selection command for the analyzed screen from the user, Displaying the selection area determined based on the above selection command on the screen, To suggest to the user some of a plurality of functions executable for the object based on the properties of the object existing in the selection area and the type of the selection command. Electronic device (101).

2. In paragraph 1, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: To perform paragraph-by-paragraph analysis on text included in the above screen and / or object-by-object analysis on images, Electronic device (101).

3. In any one of paragraphs 1 and 2, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: When a tap type selection command is received, an image of an object unit or text of a paragraph unit corresponding to the tap position is selected, To display a rectangular area containing an image of the selected object unit or a rectangular area containing text of the selected paragraph unit as a selection area on the screen. Electronic device (101).

4. In any one of paragraphs 1 to 3, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: When a selection command of the draw type is received, a rectangular area including the area drawn by the selection command of the draw type is displayed as a selection area on the screen. Electronic device (101).

5. In any one of paragraphs 1 to 4, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: Identifying the priority of each of a plurality of functions executable for the object based on the properties of the object and the type of the selection command, To display execution buttons for a preset number of functions in the order of highest priority identified above, Electronic device (101).

6. In any one of paragraphs 1 to 5, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: When a selection command for the above-mentioned execution button is received from the above-mentioned user, the function of the execution button selected by the selection command is performed. Electronic device (101).

7. In any one of paragraphs 1 to 6, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: To output a sound asking whether to perform the function of the execution button selected by the above selection command, Electronic device (101).

8. In any one of paragraphs 1 to 7, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: If there is no object within the screen area corresponding to the above selection command, a rectangular area of ​​a preset size is displayed as a selection area on the screen. Electronic device (101).

9. In any one of paragraphs 1 to 8, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: If a selection command for an area other than the above-mentioned execution button is received from the user, a selection command for the analyzed screen is re-received from the user. Electronic device (101).

10. In any one of paragraphs 1 to 9, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: If the object existing within the above selection area is text, the text is divided into word units, Identify some missing spellings in the text divided into the above word units, Correct the above selection to include some of the missing spellings. Electronic device (101).

11. In any one of paragraphs 1 to 10, The above commands, when individually or collectively executed by the one or more processors (120), cause the electronic device (101) to: If the object existing within the above selection area is an image, the image is divided into object units, Identifying an area not included in the selection area in the image divided into the above object units, Correcting the selection area so that the above-mentioned non-included area is included; Electronic device (101).

12. In the operating method of the electronic device (101), An action that receives an execution command for a selection mode from the user; An action of analyzing the screen of an application running on the electronic device (101) in response to receiving the above execution command; An action of receiving a selection command for the analyzed screen from the user; An operation of displaying a selection area determined based on the above selection command on the screen; and An action of suggesting to the user some of a plurality of functions executable for the object based on the properties of the object existing in the selection area and the type of the selection command. A method of operation including:

13. In paragraph 12, The above analyzing action is, An action to perform paragraph-by-paragraph analysis on text included in the above screen and / or object-by-object analysis on images. A method of operation including:

14. In any one of paragraphs 12 to 13, The actions indicated above are: When a selection command of the tap type is received, an action of selecting an image of an object unit or text of a paragraph unit corresponding to the tap position; and An action of displaying a rectangular area containing an image of the selected object unit or a rectangular area containing text of the selected paragraph unit as a selection area on the screen. A method of operation including:

15. In any one of paragraphs 12 to 14, The actions indicated above are: When a draw type selection command is received, an action of displaying a rectangular area including an area drawn by the draw type selection command as a selection area on the screen A method of operation including:

Images