Electronic device and method for providing virtual keyboard or recommended words on basis of one or more elements displayed on screen, and computer-readable storage medium

Abstract

An electronic device according to an embodiment comprises: a display; at least one processor including a processing circuit; and a memory including one or more storage media storing instructions. The instructions may, when executed individually or collectively by the at least one processor, enable the electronic device to: display, on the display, a screen generated on the basis of data elements arranged in a hierarchical structure; receive an input for a text input portion in the displayed screen; identify, on the basis of the input, a first data element corresponding to the text input portion; identify a second data element related to the first data element; determine, on the basis of the second data element, a type of text to be input through the text input portion; and display, on the display, a virtual keyboard together with the text input portion of the screen according to the determined type of text.

Description

Electronic device, method, and computer-readable storage medium for providing a virtual keyboard or suggested words based on one or more elements displayed on a screen

[0001] The present disclosure relates to an electronic device, a method, and a computer-readable storage medium for providing a virtual keyboard or recommended words based on one or more elements displayed on a screen.

[0002] Artificial intelligence (AI) is a technology for simulating human (or biological) neural activity, such as perception and / or inference. It can be implemented by hardware, software, or any combination thereof designed to perform computations for simulating neural activity. Based on AI, an electronic device can be provided that analyzes the hierarchy for displaying the screen and provides a virtual keyboard or suggested words.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above is applicable as prior art related to the present disclosure.

[0004] An electronic device is disclosed. According to one embodiment, the electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display on the display a screen generated based on data elements arranged in a hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input for a text input portion within the displayed screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the input, a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a second data element associated with the first data element among the data elements arranged in the hierarchy. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine a type of text to be input through the text input portion based on the second data element. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a virtual keyboard on the display together with the text input portion of the screen according to the determined type of text.

[0005] An electronic device is disclosed. According to one embodiment, the electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a screen generated based on data elements arranged in a hierarchical structure on the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input for a text input portion within the displayed screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the input, a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a second data element associated with the first data element among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine one or more recommended words associated with one or more contents provided by the second data element.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a virtual keyboard including the one or more suggested words on the display together with the text input portion of the screen.

[0006] A method of an electronic device is disclosed. According to one embodiment, the method of the electronic device may include an operation of displaying a screen generated based on data elements arranged in a hierarchical structure on a display of the electronic device. The method may include an operation of receiving an input for a text input portion within the displayed screen. The method may include an operation of identifying, based on the input, a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure. The method may include an operation of identifying a second data element related to the first data element among the data elements arranged in the hierarchical structure. The method may include an operation of determining a type of text to be input through the text input portion based on the second data element. The method may include an operation of displaying a virtual keyboard on the display together with the text input portion of the screen, based on the determined type of the text.

[0007] A computer-readable storage medium is disclosed. In one embodiment, a non-transitory computer-readable storage medium storing one or more programs may include instructions that, when executed by at least one processor of an electronic device having a display, cause a screen generated based on data elements arranged in a hierarchical structure to be displayed on the display. The one or more programs may include instructions that, when executed by the at least one processor, cause an input to be received for a text input portion within the displayed screen. The one or more programs may include instructions that, when executed by the at least one processor, cause a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure to be identified based on the input. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to identify a second data element related to the first data element among the data elements arranged in the hierarchy. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to determine a type of text to be input through the text input portion based on the second data element. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to display a virtual keyboard on the display together with the text input portion of the screen according to the determined type of text.

[0008] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0009] FIG. 2 illustrates an exemplary electronic device for analyzing a hierarchical structure for displaying a screen.

[0010] Figures 3a and 3b illustrate a hierarchical structure for displaying a screen of an exemplary electronic device.

[0011] Figures 4a and 4b illustrate the operation of an exemplary electronic device.

[0012] Figures 4c and 4d illustrate the operation of an exemplary electronic device.

[0013] Figure 5 illustrates the operation of an exemplary electronic device.

[0014] Figure 6 illustrates the operation of an exemplary electronic device.

[0015] Figure 7 is a flow chart showing the operation of an exemplary electronic device.

[0016] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0017] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0018] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or an auxiliary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (121). For example, when the electronic device (101) includes the main processor (121) and the auxiliary processor (123), the auxiliary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a given function. The auxiliary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0019] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0020] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0021] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0022] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0023] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0024] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. According to one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0025] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0026] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0027] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0028] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0029] The haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. According to one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0030] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0031] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

[0032] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0033] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0034] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0035] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas, for example, by the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0036] According to various embodiments, the antenna module (197) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0037] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0038] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0039] FIG. 2 illustrates an exemplary electronic device for analyzing a hierarchical structure for displaying a screen.

[0040] Referring to FIG. 2, an electronic device (101) may be communicatively connected to an electronic device (102). The electronic device (102) may include a server of a service provider (e.g., server (108) of FIG. 1). The server may include a computer designed to provide server functionality, such as a workstation. The embodiment is not limited thereto, and the server may include a combination of a personal computer and a software application installed on the personal computer. The server may include a group (or cluster) of example workstations and / or personal computers. The server may be configured by a service provider to execute a software application for providing a service over a network, such as the Internet.

[0041] According to one embodiment, the electronic device (101) may include a smartphone (101-1), a smartpad, and / or a tablet PC. The electronic device (101) may include a personal computer, such as a desktop (101-2) and a laptop. The embodiment is not limited thereto, and the electronic device (101) may include a smart accessory, such as a smartwatch and / or a head-mounted device (HMD), in which an application provided with a software keyboard (230) is executable within the electronic device (101).

[0042] According to one embodiment, the electronic device (102) may be configured to receive a request from the electronic device (101) to analyze a hierarchy structure for displaying a screen of the electronic device (101). The electronic device (102) may perform an operation for displaying a virtual keyboard and / or keywords related to the user input based on a user request obtained from the electronic device (101) and / or a user input entered by a community service.

[0043] According to one embodiment, a hierarchical analysis model (210) may be provided to the electronic device (102) as a neural network for analyzing the hierarchical structure, for executing a function corresponding to a request from the electronic device (101) for analyzing the hierarchical structure generated to display the screen of the electronic device (101). The electronic device (102) may be configured to generate a hierarchical structure different from the hierarchical structure or to search for a plurality of elements within the hierarchical structure, based on a user input for hierarchical analysis received by the electronic device (101), using the hierarchical analysis model (210). For example, the electronic device (102) may regenerate the hierarchical structure using the hierarchical analysis model (210) based on an input for hierarchical analysis received by the electronic device (101). Within this document, when an element is referred to as regenerating a hierarchy for displaying a screen, it may mean generating an additional hierarchy (e.g., hierarchy (302) of FIG. 3b) different from the hierarchy by rearranging data elements within the hierarchy for displaying the screen (e.g., hierarchy (301) of FIG. 3a) or changing the location between data elements within the hierarchy.

[0044] A data element within a hierarchical structure (e.g., hierarchical structure (301) of FIG. 3a, hierarchical structure (302) of FIG. 3b) may be referenced as any one of a node constituting part of the hierarchical structure, a document object model (DOM) element, and a DOM node, but is not limited thereto.

[0045] For example, the hierarchical analysis model (210) may be trained based on the weights of a plurality of elements displayed on the screen of the electronic device (101). The electronic device (101) may be configured to regenerate a hierarchical structure based on a text input portion (e.g., the text input portion (320) of FIG. 3A) for receiving a user input among the plurality of elements displayed on the screen of the electronic device (101) using the hierarchical analysis model (210) or to search for elements displayed around the text input portion. The hierarchical analysis model (210) may be trained based on the distance between a data element corresponding to the text input portion (320) and each of the data elements corresponding to elements around the text input portion (320) within the hierarchical structure for displaying the screen. The electronic device (101) may be configured to detect a corresponding element (e.g., a corresponding element (340) of FIG. 3B) related to the text input portion (320) based on the relevance of each of the searched data elements to the text input portion (320) using the hierarchical analysis model (210). The electronic device (101) may be configured to identify a data type representing the corresponding element (340) or identify information represented by the corresponding element based on the corresponding element (340) related to the text input portion (320).

[0046] For example, the processor (120) may be configured to identify a corresponding element (340) associated with a text input portion (320) within the screen by using unique values ​​of each of a plurality of preset elements for displaying the screen, or to identify the corresponding element (340) associated with the text input portion (320) by using a pattern of the layout of the screen according to a user input for each of the plurality of elements. The unique values ​​may be provided as a hash table from a database of the electronic device (101) and / or the electronic device (102), but are not limited thereto.

[0047] Although the hierarchical analysis model (210) is illustrated and described as being provided through an electronic device (102) distinct from the electronic device (101), it is not limited thereto. The hierarchical analysis model (210) may be provided to the electronic device (101). For example, the hierarchical analysis model (210) may be driven based on a plurality of parameters contained in a memory of the electronic device (101) (e.g., the memory (130) of FIG. 1).

[0048] According to one embodiment, if the hierarchical analysis model (210) fails to detect another data element related to one data element in the hierarchical structure, the hierarchical analysis model (210) may be retrained using data additionally input to the hierarchical analysis model (210). Training the hierarchical analysis model (210) using the additional data may be referred to as transfer learning, but is not limited thereto. For example, the hierarchical analysis model (210) may be configured to correct errors in data values ​​output from the hierarchical analysis model (210) by being trained through federated learning of a plurality of trained models including the hierarchical analysis model (210).

[0049] Hereinafter, with reference to FIGS. 3A to 6, an operation of an electronic device (101) using a hierarchical analysis model (210) to provide a virtual keyboard (e.g., a virtual keyboard (330) of FIG. 3A) and / or one or more recommended words related to a text input portion (320) for user input is described. The operations illustrated and described in FIGS. 3A and 3B may be performed by a processor of the electronic device (101) (e.g., the processor (120) of FIG. 1).

[0050] Figures 3a and 3b illustrate a hierarchical structure for displaying a screen of an exemplary electronic device.

[0051] Referring to FIGS. 3A and 3B , an exemplary screen (300) including a plurality of elements (330) may be displayed through a display (310) of an electronic device (101). The operations of FIGS. 3A and 3B may be performed by the electronic device (101) and / or the processor (120) of the electronic device (101) of FIG. 1 .

[0052] Referring to FIG. 3A, the processor (120) may be configured to generate a hierarchical structure (301) to display a screen (300) including a plurality of elements (330). For example, the hierarchical structure (301) may include a data element (351) corresponding to a screen (300) displayed through a display (310). The hierarchical structure (301) may include a plurality of data elements for displaying a plurality of elements (330) within the screen (300), with the data element (351) as a root data element (root node). For example, the processor (120) may be configured to display a screen (300) generated based on data elements arranged within the hierarchical structure (301). The screen (300) may include a plurality of elements (330) corresponding to each of the plurality of data elements.

[0053] According to one embodiment, the processor (120) may be configured to receive a first user input through a text input portion (320) configured to receive a user input among the plurality of elements (330) while the screen (300) including the plurality of elements (330) is displayed through the display (310). For example, the processor (120) may be configured to identify a first user input (e.g., a touch input to the text input portion (320) within the screen (300)) to the text input portion (320) for entering text, numbers, and / or emoticon objects displayed within the screen (300). The processor (120) may be configured to execute a function for entering text and / or numbers within the text input portion (320), for example, based on the first user input to the text input portion (320).

[0054] According to one embodiment, the processor (120) may be configured to display, through the display (310), a screen (300) generated based on data elements arranged in the hierarchical structure (301). The processor (120) may be configured to identify, based on a first user input to the text input portion (320), a data element (353b) corresponding to the text input portion (320) among the data elements arranged in the hierarchical structure (301). For example, the processor (120) may control the display (310) so that a plurality of elements (330) corresponding to each of the data elements in the hierarchical structure (301) are displayed on the screen (300). The processor (120) may be configured to identify a data element (353b) within the hierarchical structure (301) corresponding to the text input portion (320) based on a first user input to the text input portion (320) among the plurality of elements (330).

[0055] According to one embodiment, the processor (120) may be configured to identify a data element (353c) related to a data element (353b) corresponding to a text input portion (320) among data elements arranged within the hierarchy (301) and / or the hierarchy (302). For example, the processor (120) may be configured to identify the data element (353c) based on a progressive search of the data elements arranged within the hierarchy (301) for a hierarchical depth from the data element (353b) corresponding to the text input portion (320) using a trained hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2). The corresponding element (340) corresponding to the data element (353c) may be an element having the highest relevance to the text input portion (320) among a plurality of elements (330) displayed on the screen (300).

[0056] According to one embodiment, the processor (120) may be configured to identify a data element (353c) associated with a data element (353b) corresponding to a text input portion (320) using a trained hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2). Each distance between the text input portion (320) and other elements may be identified using x, y coordinate values ​​of a display area of ​​the display (310) corresponding to each of the one or more elements. The processor (120) may be configured to repeatedly detect one or more elements from the text input portion (320) by varying the hierarchical depth from the data element (353b) within the hierarchical structure (302) until a corresponding element (340) associated with the text input portion (320) is detected. Training of the hierarchical analysis model (210) may be referred to as, but is not limited to, machine learning.

[0057] For example, referring to FIG. 3A, the processor (120) may be configured to generate a hierarchical structure (301) for displaying a screen (300) including a plurality of elements (330) through the display (310) before a first user input is input through the text input portion (320). The hierarchical structure (301) may include a data element (351) corresponding to the screen (300), a data element (352a) corresponding to an area (300a) of the screen (300) as a lower data element of the data element (351), and a data element (353a) corresponding to an area (300b) within the area (300a) of the screen (300) as a lower data element of the data element (352a). The hierarchical structure (301) may include a data element (352b) corresponding to a virtual keyboard (360) displayed on the screen (300) as a lower data element of the data element (351) for the screen (300). The hierarchical structure (301) may include a data element (353b) corresponding to a text input section (320) displayed in the area (300a) of the screen (300) as a lower data element of the data element (352a) for the area (300a) of the screen (300), and a data element (353c) corresponding to a corresponding element (340) related to the text input section (320) displayed in the area (300a). The hierarchical structure (301) may include, as a lower data element of the data element (353a) for the area (300b) within the area (300a), a data element (354a) corresponding to a visual object (331) displayed within the area (300b), and a data element (354b) corresponding to text (332) displayed within the area (300b). However, the above-described embodiment is exemplary and is not limited thereto.The processor (120) may be configured to create a hierarchical structure (301) to display a plurality of elements (320, 331, 332, 340) in areas (300a, 300b) of a screen (300) displayed through a display (310) before a user input through a text input portion (320) is identified.

[0058] For example, referring to FIG. 3b, the processor (120) may, in response to a first user input received through the text input portion (320), regenerate a hierarchical structure using a trained hierarchical analysis model (210) based on weights of a plurality of elements (330). For example, referring to FIGS. 3a and 3b sequentially, the processor (120) may, based on receiving a first user input to the text input portion (320), regenerate a hierarchical structure (302) from the hierarchical structure (301) using a data element (353b) corresponding to the text input portion (320) as a root data element. For example, the processor (120) may generate an additional hierarchy (302) including data elements within the hierarchy (301) by modifying the hierarchy (301) based on a data element (353b) corresponding to the text input portion (320) based on a first user input to the text input portion (320). The data element (353b) corresponding to the text input portion (320) within the hierarchy (302) may be set as a root data element (root node) of the hierarchy (302). The processor (120) may be configured to identify a data element (353c) corresponding to a corresponding element (340) associated with the text input portion (320) using the regenerated hierarchy (302).

[0059] For example, referring to FIG. 3b, the processor (120) may be configured to determine whether a data element (e.g., data element (353a)) having a first hierarchical depth from a data element (353b) corresponding to a text input portion (320) among the data elements in the hierarchical structure (302) is related to the data element (353b). The processor (120) may be configured to identify the data element as a data element for determining a type of text to be input through the text input portion (320) based on determining that the data element among the data elements in the hierarchical structure (302) is related to the data element (353b). The processor (120) may be configured to determine, based on determining that the data element is not related to the data element (353b), whether another data element (e.g., data element (352a), data element (354a), or data element (354b)) having a second hierarchical depth from the data element (353b) is related to the data element (353b). The second hierarchical depth may be greater than the first hierarchical depth.

[0060] For example, the processor (120) may be configured to first detect an area (300b) corresponding to a data element (353a) within a first hierarchical depth from a data element (353b) corresponding to a text input portion (320) within a hierarchical structure (301) generated before a first user input is received. The processor (120) may be configured to detect components corresponding to lower data elements of the data element (353a) to identify whether a corresponding element (340) related to the text input portion (320) is detected among the components. For example, the processor (120) may detect a visual object (331) displayed within the area (300b) and corresponding to the data element (354a), and a text (332) corresponding to the data element (354b), to identify whether the visual object (331) and the text (332) are related to the text input portion (320). The processor (120) may be configured to detect an area (300a) corresponding to a data element (352a) within a second hierarchical depth greater than the first hierarchical depth from the data element (353a) as a higher data element of the data element (353b) corresponding to the text input portion (320) within the hierarchical structure (301) by increasing the hierarchical depth based on identifying that the visual object (331) and the text (332) are not related to the text input portion (320). For example, the processor (120) may identify whether the corresponding element (340) is related to the text input portion (320) by detecting a corresponding element (340) displayed within the area (300a) and corresponding to the data element (353c). The processor (120) may be configured to stop detecting a plurality of elements (330) displayed on the screen (300) based on detecting a corresponding element (340) associated with the text input portion (320).

[0061] For example, the processor (120) may be configured to detect elements corresponding to upper data elements (e.g., data element (352a), data element (351)) of the data element (353b) by increasing the hierarchical depth from the data elements (e.g., data element (353a)) closest to the data element (353b) corresponding to the text input portion (320) based on a first user input through the text input portion (320) using the hierarchical analysis model (210). The processor (120) may be configured to efficiently detect a corresponding element (340) related to the text input portion (320) by generating a hierarchical structure (302) for displaying a screen (300) in which the data element (353b) corresponding to the text input portion (320) is set as a root data element based on the first user input through the text input portion (320).

[0062] According to one embodiment, the processor (120) may determine the type of text to be input through the text input portion (320) based on identifying a data element (353c) related to a data element (353b) corresponding to the text input portion (320). For example, the type of text to be input into the text input portion (320) may include, but is not limited to, at least one of a character, a number, and a combination thereof. The processor (120) may be configured to display a virtual keyboard (360) on the display (310) together with the text input portion (320) of the screen (300) according to the determined type of text. The processor (120) displaying a virtual keyboard (360) having various layouts according to the determined type of text will be described below with reference to FIG. 4A.

[0063] According to one embodiment, the processor (120) may be configured to display the screen (300) on the display (310) by executing a first software application that generates the hierarchy (301) and / or the hierarchy (302). The processor (120) may be configured to execute a second software application (e.g., a keyboard application) to display a virtual keyboard (360) superimposed on the screen (300) based on a first user input to the text input portion (320). The processor (120) may be configured to identify, based on the execution of the second software application, a data element (353c) associated with a data element (353b) corresponding to the text input portion (320) among data elements arranged in the hierarchy (302) generated by the first software application. For example, the processor (120) may be configured to generate a hierarchical structure (302) in which a data element (353b) corresponding to the text input portion (320) is set as a root data element based on a first user input to the text input portion (320) while a second software application for displaying a virtual keyboard (360) on the display (310) is being executed. The processor (120) may be configured to use a hierarchical analysis model (210) to incrementally detect data elements related to the data element (353b) in the hierarchical structure (302) using a hierarchical depth. Detecting data elements related to the data element (353b) within the hierarchical structure (302) may be performed using and / or by the second software application, but is not limited thereto.

[0064] According to one embodiment, referring to FIG. 3B, the processor (120) may be configured to identify, based on identifying a corresponding element (340) associated with a text input portion (320) detected through the hierarchical structure (302), whether a data type representing the corresponding element (340) corresponds to text. The processor (120) may be configured to obtain information included in the text (e.g., a meaning of the text or a meaning of one or more words included in the text) based on identifying, based on identifying, a corresponding element (340) associated with a text input portion (320) detected through the hierarchical structure (302). For example, the processor (120) may be configured to identify, based on identifying, based on identifying, a corresponding element (340) associated with a text input portion (320) detected through the hierarchical structure (302), whether a data type representing the corresponding element (340) corresponds to an image. The processor (120) may be configured to obtain information included in the image (e.g., one or more visual objects included in the image or information represented by the visual objects) based on identifying that a data type representing the corresponding element (340) corresponds to text. The processor (120) may be configured to display a virtual keyboard (360) and / or one or more recommended words associated with the corresponding element (340) and / or the text input portion (320) using the data type representing the corresponding element (340) and / or the information represented by the corresponding element (340).

[0065] According to one embodiment, the processor (120) may be configured to identify a second data element based on a relevance between the first data element and the second data element determined by the hierarchical analysis model (210), using the hierarchical analysis model (210). The relevance may be output from the hierarchical analysis model (210) based on a referential relationship between the first data element and the second data element.

[0066] The reference relationship between the first data element and the second data element may be referred to as a reference relationship and / or association between the first data element and the second data element, in which the first data element and the second data element partially share data related to each other, and the degree of association between the first data element and the second data element may be determined based on the reference relationship, but is not limited thereto.

[0067] For example, the processor (120) may be configured to identify the data element (353b) and the data element (353c) related to the data element (353b) using the hierarchical analysis model (210) based on the reference relationship between the data element (353b) corresponding to the text input portion (320) and other data elements other than the data element (353b). The processor (120) may be configured to efficiently detect the corresponding element (340) related to the text input portion (320) using the relevance output from the hierarchical analysis model (210) based on the reference relationship.

[0068] According to one embodiment, the processor (120) may be configured to determine a type of a virtual keyboard (360) corresponding to the text input portion (320) by using a data type representing a corresponding element (340) associated with the text input portion (320) detected through the hierarchy (302) and / or information represented by the corresponding element (340). The processor (120) may be configured to control the display (310) to display the virtual keyboard (360) having a layout corresponding to the determined type.

[0069] For example, referring to FIG. 3B, the processor (120) may identify that the data type of the corresponding element (340) associated with the text input portion (320) searched through the hierarchy (302) is an image. Based on the identification that the data type of the corresponding element (340) is an image, the processor (120) may be configured to display a virtual keyboard (360) corresponding to a first software keyboard (361) including one or more alphabets (365) for inputting text associated with the corresponding element (340), a plurality of numbers (366) for visual objects within the corresponding element (340), and one or more emoticon objects (367) for expressing the visual objects within the corresponding element (340), using the data type of the corresponding element (340) and / or information indicated by the corresponding element (340). The operation of the processor (120) for displaying a virtual keyboard (360) having a layout related to the text input portion (320) using the corresponding element (340) is described with reference to FIGS. 4A to 4D. The operation of the processor (120) for displaying one or more recommended words related to the text input portion (320) using the corresponding element (340) is described with reference to FIGS. 5 and 6.

[0070] According to the above-described embodiment, the processor (120) of the electronic device (101) may be configured to efficiently detect a corresponding element (340) related to the text input portion (320) by searching for a data element (353c) related to the data element (353b) using a hierarchical depth from the data element (353b) corresponding to the text input portion (320). The processor (120) may be configured to identify a data type representing the corresponding element (340) and / or information represented by the corresponding element (340), thereby providing a virtual keyboard (360) having a layout related to the text input portion (320) or displaying one or more recommended words related to the text input portion (320) through a display (310).

[0071] Figures 4a and 4b illustrate the operation of an exemplary electronic device. Figures 4c and 4d illustrate the operation of an exemplary electronic device.

[0072] Referring to FIGS. 4A, 4B, 4C, and 4D, exemplary screens (411, 412, 413, 414) including a plurality of elements (330) may be displayed through the display (310) of the electronic device (101). The operations of FIGS. 4A to 4D may be performed by the electronic device (101) and / or the processor (120) of the electronic device (101) of FIG. 1.

[0073] Referring to FIGS. 4A and 4B, the processor (120) may be configured to generate a hierarchical structure for displaying the plurality of elements (330) through the display (310) based on the text input portion (320) using a hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2) based on a first user input to the text input portion (320) among the plurality of elements (330).

[0074] For example, the processor (120) may be configured to first detect, based on a first user input (e.g., a user's touch input to the text input portion (320)) entered into the text input portion (320), an area (412a) within the screen (412) corresponding to a data element closest to the data element corresponding to the text input portion (320) and / or elements within the area (412a). For example, based on the first user input through the text input portion (320), the processor (120) may be configured to detect a corresponding element (340) within the area (412a) and a visual object (431) for authentication of text and / or numbers entered through the text input portion (320). The processor (120) may be configured to obtain a data type representing the corresponding element (340) and information represented by the corresponding element (340) based on detecting a corresponding element (340) related to the text input portion (320).

[0075] According to one embodiment, the processor (120) may be configured to determine the type of text to be input through the text input portion (320) based on a data element associated with a data element corresponding to the text input portion (320). The processor (120) may be configured to identify a virtual keyboard (360) corresponding to the type of text among virtual keyboards available within the electronic device (101). The processor (120) may be configured to display the virtual keyboard (360) corresponding to the type of text together with the text input portion (320) within a screen displayed through the display (310). The virtual keyboard (360) corresponding to the type of text may be associated with the type of text by comparing it with one or more remaining virtual keyboards among the virtual keyboards available within the electronic device (101). For example, the processor (120) may control the display (310) to display a virtual keyboard (360) having a layout related to the text input portion (320) by using a data type representing a corresponding element (340) related to the text input portion (320) and / or information represented by the corresponding element (340).

[0076] For example, the processor (120) may be configured to determine, based on detecting a corresponding element (340) associated with the text input portion (320), that the type of text to be input through the text input portion (320) is a number. The processor may be configured to identify that the data type of the corresponding element (340) corresponds to a number. For example, the processor (120) may be configured, based on detecting a corresponding element (340) associated with the text input portion (320), to identify that information indicated by the corresponding element (340) is a number. The processor (120) may control the display (310) to display a virtual keyboard (360) corresponding to a second software keyboard (362) having a number-type layout and including a plurality of numbers (366), unlike the first software keyboard (361) of FIGS. 3A and 3B , based on identifying that the information indicated by the corresponding element (340) is a number. The second software keyboard (362) may be associated with the type of text determined by the number to be input into the text input section (320) compared to the first software keyboard (361). However, it is not limited thereto.

[0077] For example, the processor (120) may include text input portions (421, 422, 423) distinct from the text input portion (320). The processor (120) may be configured to regenerate a hierarchy based on at least one of the text input portions (421, 422, 423) based on a user input entered through at least one of the text input portions (421, 422, 423). For example, the processor (120) may be configured to detect an area (412b) within the screen (412) that is closest to the text input portions (421, 422, 423) and / or elements within the area (412b) based on a user input entered through at least one of the text input portions (421, 422, 423). For example, the processor (120) may be configured to detect elements (441) associated with text input portions (421, 422, 423) within the area (412b), visual objects (432) for authenticating text and / or numbers entered into the text input portions (421, 422, 423), and visual objects (450) for transmitting information entered into the screen (412) to an external electronic device (e.g., the electronic device (102) of FIG. 1). The processor (120) may be configured to identify that the type of text to be entered into the text input portions (421, 422, 423) is a number, based on detecting the elements (441) associated with the text input portions (421, 422, 423). The above processor (120) can obtain the data type of the above element (441) and / or information indicated by the above element (441).The processor (120) may control the display (310) to display a virtual keyboard (360) corresponding to a second software keyboard (362) having a number type layout and including a plurality of numbers (366), unlike the first software keyboard (361) of FIGS. 3A and 3B, based on determining that the type of text to be input into the text input section (320) is a number. However, the above-described embodiments are exemplary and are not limited thereto.

[0078] For example, referring to FIGS. 4C and 4D , the processor (120) may be configured to identify that the type of text to be input into the text input portion (320) is a signature based on detecting a corresponding element (340) associated with the text input portion (320). Based on identifying that the type of text is a signature, the processor (120) may control the display (310) to display a virtual keyboard (360) corresponding to a board (363) configured to receive user input via a user's gesture and / or a stylus pen (not shown) having a board-type layout, unlike FIGS. 3A , 3B , 4A, and 4B . For example, the board (363) may provide visual objects (461, 462) for displaying user input within the board (363), and a visual object (463) for deleting the user input entered within the board (363). For example, the board (363) may be configured to display a visual object (470) corresponding to the user input entered within the board (363) based on the visual objects (461, 462). However, the present invention is not limited thereto. The processor (120) of the electronic device (101) may detect a corresponding element (340) related to a text input portion (320) using a hierarchical analysis model (210), and provide a virtual keyboard (360) having various layouts related to the text input portion (320) based on the data type of the corresponding element (340) and / or information indicated by the corresponding element (340).

[0079] Hereinafter, with reference to FIGS. 5 and 6, operations of the processor (120) of the electronic device (101) for providing one or more recommended words (540) and / or one or more visual objects (530) related to the text input portion (320) using a corresponding element (340) related to the text input portion (320) are described.

[0080] Figure 5 illustrates the operation of an exemplary electronic device.

[0081] Referring to FIG. 5, an exemplary screen (500) including a plurality of elements (330) may be displayed through a display (310) of an electronic device (101). The operations of FIG. 5 may be performed by the electronic device (101) and / or the processor (120) of the electronic device (101) of FIG. 1.

[0082] Referring to FIG. 5, the processor (120) may be configured to generate a hierarchical structure (e.g., hierarchical structure (302) of FIG. 3b) for displaying the plurality of elements (330) through the display (310) based on the text input portion (320) based on a first user input to the text input portion (320) among the plurality of elements (330). The processor (120) may be configured to detect the plurality of elements (330) displayed within the screen (500) while gradually increasing the hierarchical depth from the data element corresponding to the text input portion (320) until a data element (e.g., data element (353c) of FIG. 3a) corresponding to a corresponding element (340) related to the text input portion (320) is detected based on the data element (e.g., data element (353b) of FIG. 3a) corresponding to the text input portion (320). The processor (120) may be configured to perform an operation of detecting the data element corresponding to the corresponding element (340) using a hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2). The operations of detecting the data element corresponding to the corresponding element (340) may be substantially the same as or similar to the operations illustrated and described in FIGS. 3A and 3B.

[0083] According to one embodiment, the processor (120) may be configured to determine one or more recommended words (e.g., recommended words (541, 542)) related to one or more contents (e.g., images, videos, etc.) provided by a data element (e.g., data element (353c) of FIG. 3A) associated with a data element (e.g., data element (353b) of FIG. 3A) corresponding to a text input portion (320). The processor (120) may be configured to display a virtual keyboard (360) including the one or more recommended words on the display (310) together with the text input portion (320) of the screen (500).

[0084] According to one embodiment, the processor (120) may be configured to identify a data type of a data element (e.g., data element (353c) of FIG. 3A) associated with a data element (e.g., data element (353b) of FIG. 3A) corresponding to a text input portion (320). The processor (120) may be configured to determine at least one subject area (510) of an image corresponding to the data element (353c) based on the data type identified as an image. The processor (120) may be configured to display, within an area where a virtual keyboard (360) is displayed, one or more visual objects (e.g., visual objects (531, 532, 533)) each representing one or more recommended words (e.g., recommended words (541, 542)) associated with the at least one subject area (510). For example, at least one subject area (510) may be an area surrounding at least one visual object (520) included in an image corresponding to the corresponding element (340). For example, at least one subject area (510) may include a subject area (511) surrounding a visual object (521) in the image, a subject area (512) surrounding a visual object (522), and a subject area (523) surrounding a visual object (523).

[0085] According to one embodiment, the processor (120) may be configured to obtain another image corresponding to the displayed image by changing the size of the image provided from the data element (353c) related to the data element (353b) to a preset size related to the image analysis model. The processor (120) may be configured to detect at least one subject area (510) of the image by executing the image analysis model using the other image. The processor (120) may be configured to identify at least one visual object (520) (e.g., visual objects (521, 522, 523)) included in the at least one subject area (510) by using the image analysis model, thereby displaying recommended words (541, 542) and / or visual objects (531, 532, 533) corresponding to the at least one visual object (520) through the virtual keyboard (360).

[0086] For example, the processor (120) may be configured to identify a size of an image corresponding to the corresponding element (340) displayed in the screen (500) based on identifying that a data type representing the corresponding element (340) corresponds to an image. The processor (120) may be configured to change the size of the image to a specified size based on identifying that the size of the image exceeds a specified size. The processor (120) may be configured to detect at least one visual object (520) included in the image having a size less than or equal to the specified size using an image analysis model. The image analysis model may be provided through the electronic device (102) as a neural network, such as the hierarchical analysis model (210) of FIG. 2, or may be driven based on a plurality of parameters included in a memory (e.g., the memory (130) of FIG. 1) within the electronic device (101).

[0087] For example, the processor (120) may be configured to identify whether the image has a size exceeding a specified size processable by an image analysis model based on identifying that the data type of the corresponding element (340) associated with the text input portion (320) is an image. The processor (120) may be configured to input each image data value of elements whose sizes are changed (e.g., visual objects (521, 522, 523) of FIG. 5) within the corresponding element (340) corresponding to the image into the image analysis model, without changing the display of the corresponding element (340) within the screen, based on identifying the image having the size exceeding the specified size. For example, the processor (120) may be configured to input the image into the image analysis model based on identifying an image corresponding to the corresponding element (340) having a size less than or equal to a specified size processable by the image analysis model. The processor (120) may be configured to detect at least one visual object (520) included in the image using the image analysis model. For example, the processor (120) may be configured to receive information related to visual objects (521, 522, 523) included in the image, output from the image analysis model.

[0088] According to one embodiment, the processor (120) may classify at least one visual object (520) based on detecting at least one visual object (520) included in a corresponding element (340) using an image analysis model. The processor (120) may be configured to control the display (310) to display one or more recommended words (540) related to the at least one visual object (520) based on the classified at least one visual object (520).

[0089] For example, the processor (120) may be configured to identify that the data type of the corresponding element (340) associated with the text input portion (320) displayed in the area (500a) is an image. The processor (120) may be configured to detect visual objects (521, 522, 523) included in the corresponding element (340) using an image analysis model. The processor (120) may classify the visual objects (521, 522, 523) by classifying each of the visual objects (521, 522, 523) to obtain information represented by each of the visual objects (521, 522, 523). The above processor (120) may be configured to display one or more recommended words (540) related to the visual objects (521, 522, 523) through the display (310) using information indicated by each of the visual objects (521, 522, 523).

[0090] For example, the processor (120) may be configured to display a keyword (541) related to the visual object (521) through the virtual keyboard (360) based on classifying the visual object (521) among at least one visual object (520). For example, the processor (120) may be configured to display a keyword (542) related to the visual object (523) through the virtual keyboard (360) based on classifying the visual object (523) among at least one visual object (520). However, the above-described embodiments are exemplary and are not limited thereto.

[0091] According to one embodiment, the processor (120) may be configured to display, on the display (310), a virtual keyboard (360) including one or more executable objects (550), each of which represents one or more recommended words (540) in the text input portion (320). For example, the processor (120) may receive a second user input for an executable object (551) among the one or more executable objects (550). Based on the second user input, the processor (120) may be configured to display the recommended word (541) represented by the executable object (541) in the text input portion (320) on the display (310). For example, the processor (120) may receive a second user input for an executable object (552) among the one or more executable objects (550). Based on the second user input, the processor (120) may be configured to display the recommended word (542) indicated by the executable object (552) in the text input portion (320) on the display (310). However, the above-mentioned embodiment is exemplary and is not limited thereto.

[0092] According to one embodiment, the processor (120) may be configured to control the display (310) to display one or more visual objects (530) related to the at least one visual object (520) based on the classified at least one visual object (520). For example, the processor (120) may be configured to display emoticon objects (531, 532) related to the visual object (521) through the virtual keyboard (360) based on classifying the visual object (521) among the at least one visual object (520). For example, the processor (120) may be configured to display an emoticon object (534) related to the visual object (522) through the virtual keyboard (360) based on classifying the visual object (522) among the at least one visual object (520). For example, the processor (120) may be configured to display an emoticon object (533) related to the visual object (523) through the virtual keyboard (360) based on classifying the visual object (523) among at least one visual object (520). However, the above-described embodiments are exemplary and are not limited thereto.

[0093] Figure 6 illustrates the operation of an exemplary electronic device.

[0094] Referring to FIG. 6, an exemplary screen (600) including a plurality of elements (330) may be displayed through a display (310) of an electronic device (101). The operations of FIG. 6 may be performed by the electronic device (101) and / or the processor (120) of the electronic device (101) of FIG. 1.

[0095] Referring to FIG. 6, the processor (120) may be configured to generate a hierarchical structure (e.g., hierarchical structure (302) of FIG. 3b) for displaying the plurality of elements (330) through the display (310) based on the text input portion (320) based on a first user input to the text input portion (320) among the plurality of elements (330). The processor (120) may be configured to detect the plurality of elements (330) displayed within the screen (500) while gradually increasing the hierarchical depth from the data element corresponding to the text input portion (320) until a data element (e.g., data element (353c) of FIG. 3a) corresponding to a corresponding element (340) related to the text input portion (320) is detected based on the data element (e.g., data element (353b) of FIG. 3a) corresponding to the text input portion (320). The processor (120) may be configured to perform an operation of detecting the data element corresponding to the corresponding element (340) using a hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2). The operations of detecting the data element corresponding to the corresponding element (340) may be substantially the same as or similar to the operations illustrated and described in FIGS. 3A and 3B.

[0096] According to one embodiment, the processor (120) may be configured to determine a type of text to be input into the text input portion (320) based on detecting a corresponding element (340) displayed within an area (600a) of the screen (600) associated with the text input portion (320). The processor (120) may control the display (310) to display a virtual keyboard (360) corresponding to a first software keyboard (361) associated with the text input portion (320) and having a layout of a character type, based on identifying that the type of text to be input into the text input portion (320) is a character. The first software keyboard (361) may include, for example, at least one alphabet (365) for text input and a plurality of numbers (366) for number input, but is not limited thereto.

[0097] According to one embodiment, the processor (120) may be configured to display, through the display (310), one or more contents provided from a data element corresponding to a corresponding element (340) associated with a text input portion (320). For example, the processor (120) may control the display (310) so that a video provided from a data element corresponding to the corresponding element (340) is displayed within the area (600a).

[0098] According to one embodiment, the processor (120) may be configured to determine one or more recommended words (540) associated with one or more contents provided from a data element corresponding to the corresponding element (340). The processor (120) may control the display (310) to display a virtual keyboard (360) including the one or more recommended words (540) on the screen together with the text input portion (320).

[0099] According to one embodiment, the processor (120) may be configured to receive a second user input (e.g., a user's touch input to a virtual keyboard (360)) for inputting at least one character (621) associated with the text input portion (320). In response to the second user input, the processor (120) may be configured to display, through the display (310), one or more recommended words (540) including the at least one character (621) and associated with the text input portion (320) using a data type representing the corresponding element (340) and / or information represented by the corresponding element (340).

[0100] For example, although not shown, the processor (120) may be configured to display, through the display (310), one or more other recommended words that are associated with the corresponding element (340) and are distinct from the one or more recommended words (540) before a second user input is received to the text input portion (320). The processor (120) may be configured to determine one or more recommended words (540) using the at least one character (621) and / or the corresponding element (340) based on receiving a second user input for input of at least one character (621) through the text input portion (320). The processor (120) may be configured to display a virtual keyboard (360) including one or more recommended words (540) including at least one character (621) on the display (310) together with a text input portion (320) including at least one character (621). Upon displaying the one or more recommended words (540), the display of some of the one or more other recommended words on the virtual keyboard (360) may be terminated.

[0101] For example, the processor (120) may be configured to identify that the data type of the corresponding element (340) associated with the text input portion (320) displayed in the area (600a) is text. The processor (120) may be configured to obtain information indicated by the detected corresponding element (340) (e.g., the meaning of the text corresponding to the corresponding element (340) or the meaning of one or more words included in the corresponding element (340). The processor (120) may be configured to use the information to display one or more recommended words (540) associated with the visual objects (521, 522, 523) through the display (310).

[0102] For example, the processor (120) may use the information indicated by the corresponding element (340) to provide recommended words (641, 642) related to the corresponding element (340) and the text input portion (320) and including at least one character (621) through the virtual keyboard (360). However, the above-described embodiment is exemplary and is not limited thereto.

[0103] For example, the processor (120) may be configured to display, through the display (310), one or more recommended words (540) including the at least one character (621) and / or one or more visual objects (530) associated with the corresponding element (340), based on receiving a second user input for inputting at least one character (621) associated with the text input portion (320). For example, the processor (120) may be configured to display, through the virtual keyboard (360), emoticon objects (631, 632, 633, 634) associated with the corresponding element (340), based on identifying information represented by the corresponding element (340). However, the above-described embodiments are exemplary and are not limited thereto.

[0104] Figure 7 is a flow chart showing the operation of an exemplary electronic device.

[0105] The operations of FIG. 7 may be performed by the electronic device (101) and / or the processor (120) of the electronic device (101) of FIG. 1.

[0106] Referring to FIG. 7, in operation (701), the processor (120) may be configured to receive a first user input through a text input portion (e.g., a text input portion (320) of FIG. 3A) configured to receive a user input within a screen displayed through a display (e.g., a display (310) of FIG. 3A). For example, the processor (120) may be configured to identify a user input related to the input element (320) based on a user's touch input to an area within the screen where the text input portion (320) is displayed.

[0107] In operation (703), the processor (120) may be configured to incrementally search data elements within a hierarchy (e.g., hierarchy (301) and / or hierarchy (302) of FIG. 3A) based on identifying a first data element (e.g., data element (353b) of FIG. 3A) corresponding to the text input portion (320) with respect to a hierarchy depth from the first data element (353b). For example, the processor (120) may be configured to generate a hierarchy (e.g., hierarchy (302) of FIG. 3B) in which the first data element (353b) corresponding to the text input portion (320) is set as a root data element, based on a user input to the text input portion (320). The processor (120) may be configured to detect a second data element (e.g., data element (353c) of FIG. 3a) having a relatively high correlation with the first data element (353b) among data elements corresponding to each of a plurality of elements (e.g., a plurality of elements (330) of FIG. 3a) within the hierarchical structure (302), for example, by using a hierarchical analysis model (e.g., hierarchical analysis model (210) of FIG. 2a).

[0108] In operation (705), the processor (120) may be configured to identify whether a corresponding element (340) associated with the text input portion (320) has been detected. For example, with reference to FIG. 3b, the processor (120) may be configured to detect elements corresponding to data elements closest to the data element (353b) corresponding to the text input portion (320) within the hierarchy (302) generated based on the user input. The processor (120) may be configured to identify whether a corresponding element (340) having a relevance higher than a specified relevance with the text input portion (320) has been detected based on the relevance with the text input portion (320). If the processor (120) does not detect the corresponding element (340) (e.g., 705-No), the processor (120) may perform operation (707).

[0109] In operation (707), the processor (120) may increase a hierarchical depth for searching the plurality of elements (330). For example, referring to FIG. 3B, the processor (120) may detect an area (300a) corresponding to a data element (353a) located within a first hierarchical depth from a data element (353b) corresponding to a text input portion (320). Based on identifying that a corresponding element (340) is not detected within the area (300b), the processor (120) may increase the hierarchical depth to detect an area (300a) corresponding to a data element (352a) located within a second hierarchical depth greater than the first hierarchical depth from the data element (353b) corresponding to the text input portion (320).

[0110] In operation (709), the processor (120) may be configured to determine a type of text to be input into the text input portion (320) based on detecting a corresponding element (340) associated with the text input portion (320). For example, the processor (120) may be configured to determine a type of text (e.g., a number, a character) to be input into the text input portion (320) based on the corresponding element (340) and / or a second data element (353c) corresponding to the corresponding element (340).

[0111] In operation (711), the processor (120) may be configured to identify a data type representing the corresponding element (340) and information represented by the corresponding element (340) based on detecting the corresponding element (340). For example, the processor (120) may be configured to obtain a data type of the corresponding element (340) and information contained within the corresponding element (340) based on detecting the corresponding element (340). For example, the processor (120) may be configured to identify whether the data type representing the corresponding element (340) is an image. For example, the processor (120) may be configured to identify that the data type represented by the corresponding element (340) is text or a number based on identifying that the type representing the corresponding element (340) is not an image. If the processor (120) fails to identify that the data type is an image (e.g., 711-No), it may perform operation (713).

[0112] In operation (713), the processor (120) may be configured to display a virtual keyboard (e.g., the virtual keyboard (360) of FIG. 3A) having a layout related to the text input portion (320) based on the type of text determined based on the corresponding element (340). For example, the processor (120) may be configured to display a first software keyboard (e.g., the first software keyboard (361) of FIG. 3A) having a character layout based on determining that the type of text to be input into the text input portion (320) is a character through the corresponding element (340). For example, the processor (120) may be configured to display a second software keyboard (e.g., the second software keyboard (362) of FIG. 4B) having a number layout based on determining that the type of text to be input into the text input portion (320) is a number through the corresponding element (340).

[0113] In operation (715), the processor (120) may be configured to identify whether the size of the image is less than a specified size based on the identification that the data type of the corresponding element (340) is an image. For example, the processor (120) may be configured to identify whether the image has a size exceeding a specified size that can be processed by an image analysis model. If the processor (120) identifies the image as having a size exceeding the specified size (e.g., 715-Yes), the processor (120) may perform operation (717).

[0114] In operation (717), the processor (120) may be configured to change the size of the image to the specified size based on identifying that the size of the image corresponding to the corresponding element (340) exceeds a specified size (or based on identifying that the size of the image is not less than the specified size). For example, the processor (120) may be configured to change the size of the image to the specified size that is processable through an image analysis model based on identifying that the size of the image exceeds a specified size.

[0115] In operation (719), the processor (120) may be configured to detect at least one visual object (e.g., at least one visual object (520) of FIG. 5) included within a corresponding element ((340). For example, the processor (120) may be configured to identify at least one visual object (520) included within an image corresponding to the corresponding element (340) output through an image analysis model.

[0116] In operation (721), the processor (120) may be configured to classify at least one visual object (520) included in the corresponding element (340). For example, the processor (120) may be configured to obtain information about each of the at least one visual object (520) based on identifying the at least one visual object (520) included in the corresponding element (340). The processor (120) may be configured to classify the at least one visual object (520) using the obtained information about the at least one visual object (520).

[0117] In operation (723), the processor (120) may be configured to display one or more recommended words (e.g., one or more recommended words (540) of FIG. 5) related to the at least one visual object (520) together with the text input portion (320) using an image including the at least one visual object (520), based on the classified at least one visual object (520). For example, when referring to FIG. 5 together, the processor (120) may provide recommended words (541, 542) related to the at least one visual object (520) based on the at least one visual object (520) included in the corresponding element (340).

[0118] According to the above-described embodiment, an electronic device (e.g., the electronic device (101) of FIG. 1) may include a display (e.g., the display (310) of FIG. 3A), at least one processor including a processing circuit (e.g., the processor (120) of FIG. 1), and a memory (e.g., the memory (130) of FIG. 1) including one or more storage media for storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display on the display a screen generated based on data elements arranged in a hierarchical structure (e.g., the hierarchical structure (301) of FIG. 3A). The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input for a text input portion (e.g., the text input portion (320) of FIG. 3A) within the displayed screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the input, a first data element (e.g., data element 353b of FIG. 3A) corresponding to the text input portion among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the input, a second data element (e.g., data element 353c of FIG. 3A) related to the first data element among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine, based on the second data element, a type of text to be input through the text input portion.The above instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a virtual keyboard (e.g., the virtual keyboard (360) of FIG. 3A) on the display together with the text input portion of the screen according to the determined type of text.

[0119] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the second data element based on incrementally searching the data elements with respect to a hierarchical depth from the first data element.

[0120] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine whether a data element having a first hierarchical depth from the first data element among the data elements is related to the first data element, identify the data element as the second data element based on determining that the data element is related to the first data element, and identify the second data element based on determining that the data element is not related to the first data element, by determining whether another data element having a second hierarchical depth from the first data element is related to the first data element. The second hierarchical depth may be greater than the first hierarchical depth.

[0121] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the second data element based on a relationship between the first data element and the second data element determined by a hierarchical analysis model (e.g., the hierarchical analysis model (210) of FIG. 2).

[0122] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to, based on the input, create an additional hierarchical structure including the data elements by modifying the hierarchical structure based on the first data element, and the first data element in the additional hierarchical structure may be set as a root data element. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the second data element using the additional hierarchical structure.

[0123] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a virtual keyboard corresponding to the type of text among virtual keyboards available within the electronic device. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the virtual keyboard corresponding to the type of text on the display together with the text input portion of the screen. The virtual keyboard among the virtual keyboards may be associated with the type of text by comparing it with one or more other virtual keyboards.

[0124] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the screen on the display by executing a first software application that creates the hierarchy of the data elements. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute a second software application to superimpose the virtual keyboard on the screen based on the input. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the execution of the second software application, the second data element among the data elements arranged in the hierarchy created by the first software application.

[0125] In one embodiment, an electronic device may include a display, at least one processor including a processing circuit, and a memory including one or more storage media storing instructions. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display on the display a screen generated based on data elements arranged in a hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input for a text input portion within the displayed screen. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify, based on the input, a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a second data element associated with the first data element among the data elements arranged in the hierarchical structure. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine one or more recommended words (e.g., one or more recommended words (540) of FIG. 5) associated with one or more contents provided by the second data element.The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display a virtual keyboard including the one or more suggested words on the display together with the text input portion of the screen.

[0126] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the second data element based on incrementally searching the data elements with respect to a hierarchical depth from the first data element.

[0127] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine whether a data element having a first hierarchical depth from the first data element among the data elements is related to the first data element, identify the data element as the second data element based on determining that the data element is related to the first data element, and identify the second data element based on determining that the data element is not related to the first data element, by determining whether another data element having a second hierarchical depth from the first data element is related to the first data element. The second hierarchical depth may be greater than the first hierarchical depth.

[0128] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify the second data element using a hierarchical analysis model.

[0129] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify at least one character (e.g., at least one character 621 of FIG. 6) entered into the text input portion via the virtual keyboard while the virtual keyboard including the one or more suggested words is displayed. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine one or more other suggested words using the at least one character and the one or more contents. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the virtual keyboard including the one or more other suggested words on the display together with the text input portion including the at least one character on the screen. Upon displaying the one or more other suggested words, displaying some of the one or more suggested words on the virtual keyboard may be terminated.

[0130] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display on the display the virtual keyboard, which includes one or more executable objects (e.g., one or more executable objects (550) of FIG. 5) each representing one or more suggested words in the text input portion. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive another input for an executable object of the one or more executable objects. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, in the text input portion on the display, a suggested word represented by the executable object, based on the another input.

[0131] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to identify a data type of the second data element. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to determine, based on the data type identified in the image, at least one subject region (e.g., at least one subject region (510) of FIG. 5 ) of an image corresponding to the second data element, and to display, within an area where the virtual keyboard is displayed, one or more visual objects (e.g., one or more visual objects (510) of FIG. 5 ), each of which represents one or more suggested words associated with the at least one subject region.

[0132] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to obtain another image corresponding to the displayed image by resizing the image to a preset size associated with an image analysis model based on the data type identified as the image, and to execute the image analysis model using the another image to detect the at least one subject area.

[0133] According to one embodiment, a method of an electronic device may include an operation of displaying a screen generated based on data elements arranged in a hierarchical structure on a display of the electronic device. The method may include an operation of receiving an input for a text input portion within the displayed screen. The method may include an operation of identifying, based on the input, a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure. The method may include an operation of identifying a second data element related to the first data element among the data elements arranged in the hierarchical structure. The method may include an operation of determining a type of text to be input through the text input portion based on the second data element. The method may include an operation of displaying a virtual keyboard on the display together with the text input portion of the screen according to the determined type of text.

[0134] For example, the method may further include identifying the second data element based on incrementally searching the data elements with respect to a hierarchical depth from the first data element.

[0135] For example, the method may further include determining whether a data element having a first hierarchical depth from the first data element among the data elements is related to the first data element, identifying the data element as the second data element based on determining that the data element is related to the first data element, and identifying the second data element based on determining that the data element is not related to the first data element, determining whether another data element having a second hierarchical depth from the first data element is related to the first data element. The second hierarchical depth may be greater than the first hierarchical depth.

[0136] For example, the method may further include an operation of identifying the second data element using a hierarchical analysis model.

[0137] In one embodiment, a non-transitory computer readable storage medium storing one or more programs may include instructions that, when executed by at least one processor of an electronic device having a display, cause a screen generated based on data elements arranged in a hierarchical structure to be displayed on the display. The one or more programs may include instructions that, when executed by the at least one processor, cause an input to be received for a text input portion within the displayed screen. The one or more programs may include instructions that, when executed by the at least one processor, cause a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure to be identified based on the input. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to identify a second data element related to the first data element among the data elements arranged in the hierarchy. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to determine a type of text to be input through the text input portion based on the second data element. The one or more programs may include instructions that, when executed by the at least one processor, cause the program to display a virtual keyboard on the display together with the text input portion of the screen according to the determined type of text.

[0138] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, electronic devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0139] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0140] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0141] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0142] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0143] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In electronic devices, display; At least one processor comprising a processing circuit; and A memory comprising one or more storage media storing instructions, said instructions, when individually or collectively executed by said at least one processor, causing said electronic device to: Displaying a screen generated based on data elements arranged within a hierarchical structure; Receive input for the text input portion within the displayed screen; Based on the above input, identifying a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure; Identifying a second data element related to the first data element among the data elements arranged within the above hierarchical structure; Determine the type of text to be entered through the text input portion based on the second data element; and Causing a virtual keyboard to be displayed on the display together with the text input portion of the screen according to the type of the determined text. Electronic devices.

2. In paragraph 1, The instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify the second data element based on incrementally searching the data elements with respect to a hierarchical depth from the first data element. Electronic devices.

3. In paragraph 1 or 2, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify the second data element by: Determine whether a data element having a first hierarchical depth from the first data element among the above data elements is related to the first data element; identifying the data element as the second data element based on determining that the data element is related to the first data element; and Based on determining that the above data element is not related to the first data element, determining whether another data element having a second hierarchical depth from the first data element is related to the first data element; The above second layer depth is greater than the above first layer depth. Electronic devices.

4. In any one of paragraphs 1 to 3, The instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify the second data element based on a relevance between the first data element and the second data element determined by the hierarchical analysis model, using a hierarchical analysis model. The above relevance is, Based on the referential relationship between the first data element and the second data element, output from the hierarchical analysis model, Electronic devices.

5. In any one of paragraphs 1 to 4, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Based on the input, by modifying the hierarchy based on the first data element, an additional hierarchy including the data elements is generated, and the first data element in the additional hierarchy is set as a root data element; and Using the above additional hierarchical structure, causing the second data element to be identified, Electronic devices.

6. In any one of paragraphs 1 to 5, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Identifying the virtual keyboard corresponding to the type of the text among the virtual keyboards available within the electronic device; and Causing the virtual keyboard corresponding to the type of the text to be displayed on the display together with the text input portion of the screen, Among the above virtual keyboards, the virtual keyboard is associated with the type of the text compared to one or more of the remaining virtual keyboards. Electronic devices.

7. In any one of paragraphs 1 to 6, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Displaying said screen on said display by executing a first software application that creates said hierarchical structure of said data elements; Based on the above input, executing a second software application to display the virtual keyboard superimposed on the screen; and Based on the execution of said second software application, causing said second data element to be identified among said data elements arranged in said hierarchical structure generated by said first software application. Electronic devices.

8. In electronic devices, display; At least one processor comprising a processing circuit; and A memory comprising one or more storage media storing instructions, said instructions, when individually or collectively executed by said at least one processor, causing said electronic device to: Displaying a screen generated based on data elements arranged within a hierarchical structure; Receive input for the text input portion within the displayed screen; Based on the above input, identifying a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure; Identifying a second data element related to the first data element among the data elements arranged within the above hierarchical structure; determining one or more recommended words associated with one or more contents provided by said second data element; and Causing a virtual keyboard including one or more of said recommended words to be displayed on said display together with said text input portion of said screen; Electronic devices.

9. In paragraph 8, The instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify the second data element based on incrementally searching the data elements with respect to a hierarchy depth from the first data element. Electronic devices.

10. In clause 8 or 9, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to identify the second data element by: Determine whether a data element having a first hierarchical depth from the first data element among the above data elements is related to the first data element; identifying the data element as the second data element based on determining that the data element is related to the first data element; and Based on determining that the above data element is not related to the first data element, determining whether another data element having a second hierarchical depth from the first data element is related to the first data element; The above second layer depth is greater than the above first layer depth. Electronic devices.

11. In any one of paragraphs 8 to 10, The instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: identify the second data element using a hierarchical analysis model; Identify the data type of the second data element; Based on the data type identified as the image, determining at least one subject region of the image corresponding to the second data element; and Causing to display one or more visual objects, each of which represents one or more suggested words associated with the at least one subject area, within the area where the virtual keyboard is displayed; Electronic devices.

12. In any one of paragraphs 8 to 11, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: While the virtual keyboard including the one or more recommended words is displayed, identifying at least one character entered into the text input portion through the virtual keyboard; determining one or more other recommended words using the at least one character and the one or more contents; causing said virtual keyboard including said one or more other suggested words to be displayed on said display together with said text input portion including said at least one character of said screen; and Upon displaying one or more of the other suggested words, the display of some of the suggested words on the virtual keyboard ends. Electronic devices.

13. In any one of paragraphs 8 to 12, The above instructions, when individually or collectively executed by the at least one processor, cause the electronic device to: Displaying on the display a virtual keyboard including one or more executable objects, each of which represents one or more of the recommended words in the text input portion; Receiving another input for an executable object among the one or more executable objects; Causing, based on said other input, to display the recommended word indicated by said executable object within said text input portion on said display; Electronic devices.

14. In the method of an electronic device, An act of displaying a screen generated based on data elements arranged within a hierarchical structure on a display of the electronic device; Receive input for the text input portion within the displayed screen; An operation of identifying a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure based on the input; An action of identifying a second data element related to the first data element among the data elements arranged within the above hierarchical structure; An operation for determining the type of text to be input through the text input portion based on the second data element; and Including an action of displaying a virtual keyboard on the display together with the text input portion of the screen according to the type of the determined text. method.

15. In a non-transitory computer readable storage medium storing one or more programs, the one or more programs, when executed by at least one processor of an electronic device having a display, Displaying a screen generated based on data elements arranged within a hierarchical structure; Receive input for the text input portion within the displayed screen; Based on the above input, identifying a first data element corresponding to the text input portion among the data elements arranged in the hierarchical structure; Identifying a second data element related to the first data element among the data elements arranged within the above hierarchical structure; Determine the type of text to be entered through the text input portion based on the second data element; and Including instructions for causing a virtual keyboard to be displayed on the display together with the text input portion of the screen according to the type of the determined text. Computer readable storage medium.

Images