Device, method, and storage medium for providing translation of text in content

Abstract

This electronic device may comprise: a memory storing instructions; a display; and at least one processor. The instructions, when executed individually or collectively by the at least one processor, may instruct the electronic device to: display a screen of a first software application on the basis of execution of the first software application; display the screen including content having text written in a first language, and including a view area having a first size, on the basis of at least part of an input for the screen; and display the screen including other content having the text written in the first language and other text written in a second language different from the first language, and including the view area having a second size larger than the first size, on the basis of activation of a function of a second software application, while displaying the screen including the view area having the first size.

Description

Device, method, and storage medium for providing translation of text within content

[0001] The descriptions below relate to devices, methods, and storage media for providing translation of text within contents.

[0002] A software application may provide a function for translating text written in a specific language into text written in another language. For example, an electronic device may generate translated text based on text entered by a user using the function provided by the software application. For example, the software application may be installed on the electronic device or provided through a server external to the electronic device.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] An electronic device may include a memory that stores instructions. The electronic device may include a display. The electronic device may include a processor. The instructions, when executed by the processor, may cause the electronic device to display a screen of a first software application through the display based on execution of the first software application. The instructions, when executed by the processor, may cause the electronic device to display, through the display, a screen including content having text written in a first language and including a view area having a first size, based on at least a portion of an input to the screen. The instructions, when executed by the processor, may cause the electronic device to display, through the display, the screen including the view area having the first size, the screen including the text written in the first language and other content having another text written in a second language different from the first language, and the view area having a second size larger than the first size, based on activation of a function of a second software application while displaying the screen including the view area having the first size.

[0005] A method performed by an electronic device may include, based on execution of a first software application, displaying a screen of the first software application. The method may include, based on at least a portion of an input to the screen, displaying the screen, which includes content having text written in a first language and which includes a view area having a first size. The method may include, based on activation of a function of a second software application while displaying the screen including the view area having the first size, displaying the screen, which includes other content having the text written in the first language and another text written in a second language different from the first language, and which includes the view area having a second size larger than the first size.

[0006] A non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when executed by a processor of an electronic device including a display, cause a screen of a first software application to be displayed through the display based on execution of the first software application. The non-transitory computer-readable storage medium may store one or more programs comprising instructions that, when executed by the processor, cause a screen including content having text written in a first language and including a view area having a first size, based on at least a portion of an input to the screen. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor, cause the screen to be displayed through the display, including the view area having a second size larger than the first size, the screen including the text written in the first language and other content having another text written in a second language different from the first language, based on the activation of a function of a second software application while displaying the screen including the view area having the first size.

[0007] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0008] FIG. 2 illustrates an example of a method for an electronic device to translate content included in a screen of a software application and display a screen including translated text, according to one embodiment.

[0009] FIG. 3A illustrates an example block diagram of a functional configuration of an electronic device, according to one embodiment.

[0010] FIG. 3b illustrates an example of the structure of a software application and framework of an electronic device according to one embodiment.

[0011] FIG. 4 illustrates an example of an operational flow for a method for an electronic device to display a screen including content of a first software application depending on whether a function for translation of a second software application is activated, according to one embodiment.

[0012] FIG. 5A illustrates an example of hierarchy information constituting a screen of a first software application according to one embodiment.

[0013] FIG. 5b illustrates an example of first information for a view area of ​​hierarchical information and second information for text, according to one embodiment.

[0014] FIG. 6 illustrates an example of an operational flow for a method of activating a function for translation of a second software application, according to one embodiment.

[0015] FIG. 7 illustrates an example of a signal flow for a method of generating a translation model and providing a translation function using the generated translation model, according to one embodiment.

[0016] FIG. 8 illustrates an example of an operational flow for a method of providing a translation function depending on whether a screen is changed, according to one embodiment.

[0017] FIGS. 9A to 9D illustrate examples of a method for translating text included in a screen and displaying a screen including the translated text, according to one embodiment.

[0018] FIGS. 10A through 10C illustrate examples of a method for activating a function of a second software application based on an input to an icon of a keyboard application, according to one embodiment.

[0019] FIGS. 10d and 10e illustrate examples of a method for changing settings of a second software application by changing settings of a keyboard application, according to one embodiment.

[0020] FIG. 11 illustrates an example of a method for activating a function of a second software application within a system setting, according to one embodiment.

[0021] FIG. 12A illustrates an example of a method for providing summary information of content, according to one embodiment.

[0022] FIG. 12b illustrates an example of a method for providing visual effects for content, according to one embodiment.

[0023] FIG. 12c illustrates an example of a method for translating text within content having an image and displaying a screen containing the translated text, according to one embodiment.

[0024] FIGS. 13A and 13B illustrate examples of a method for changing settings of a second software application through input to a visual object, according to one embodiment.

[0025] FIG. 14 illustrates an example of an operational flow for a method of an electronic device translating content included in a screen of a software application and displaying a screen including translated text, according to one embodiment.

[0026] FIG. 15 illustrates an example of a generative artificial intelligence system according to one embodiment.

[0027] The terms used in this disclosure are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include plural expressions unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by those of ordinary skill in the art described in this disclosure. Terms defined in general dictionaries among the terms used in this disclosure may be interpreted as having the same or similar meaning in the context of the relevant technology, and shall not be interpreted in an idealized or overly formal sense unless explicitly defined in this disclosure. In some cases, even if a term is defined in this disclosure, it cannot be interpreted to exclude embodiments of the present disclosure.

[0028] The various embodiments of the present disclosure described below illustrate a hardware-based approach as an example. However, since the various embodiments of the present disclosure include techniques utilizing both hardware and software, the various embodiments of the present disclosure do not exclude a software-based approach.

[0029] In addition, in the present disclosure, expressions such as "more than" or "less than" may be used to determine whether a specific condition is satisfied or fulfilled. However, this is merely a description for expressing an example and does not exclude descriptions such as "more than" or "less than." Conditions described as "more than" may be replaced with "more than," conditions described as "less than," and conditions described as "more than and less than" may be replaced with "more than and less than." In addition, hereinafter, "A" to "B" mean at least one of the elements from A (including A) to B (including B).

[0030] FIG. 1 is a block diagram of an electronic device within a network environment according to various embodiments.

[0031] Referring to FIG. 1, in a network environment (100), an electronic device (101) may communicate with an electronic device (102) via a first network (198) (e.g., a short-range wireless communication network), or may communicate with at least one of an electronic device (104) or a server (108) via a second network (199) (e.g., a long-range wireless communication network). According to one embodiment, the electronic device (101) may communicate with the electronic device (104) via the server (108). According to one embodiment, the electronic device (101) may include a processor (120), a memory (130), an input module (150), an audio output module (155), a display module (160), an audio module (170), a sensor module (176), an interface (177), a connection terminal (178), a haptic module (179), a camera module (180), a power management module (188), a battery (189), a communication module (190), a subscriber identification module (196), or an antenna module (197). In some embodiments, the electronic device (101) may omit at least one of these components (e.g., the connection terminal (178)), or may have one or more other components added. In some embodiments, some of these components (e.g., the sensor module (176), the camera module (180), or the antenna module (197)) may be integrated into one component (e.g., the display module (160)).

[0032] The processor (120) may, for example, execute software (e.g., a program (140)) to control at least one other component (e.g., a hardware or software component) of the electronic device (101) connected to the processor (120) and perform various data processing or calculations. According to one embodiment, as at least a part of the data processing or calculations, the processor (120) may store commands or data received from other components (e.g., a sensor module (176) or a communication module (190)) in a volatile memory (132), process the commands or data stored in the volatile memory (132), and store result data in a non-volatile memory (134). According to one embodiment, the processor (120) may include a main processor (121) (e.g., a central processing unit or an application processor) or a secondary processor (123) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor)) that can operate independently or together therewith. For example, if the electronic device (101) includes a main processor (121) and a secondary processor (123), the secondary processor (123) may be configured to use less power than the main processor (121) or to be specialized for a specified function. The secondary processor (123) may be implemented separately from the main processor (121) or as a part thereof.

[0033] The auxiliary processor (123) may control at least a portion of functions or states associated with at least one component (e.g., a display module (160), a sensor module (176), or a communication module (190)) of the electronic device (101), for example, on behalf of the main processor (121) while the main processor (121) is in an inactive (e.g., sleep) state, or together with the main processor (121) while the main processor (121) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (123) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (180) or a communication module (190)). In one embodiment, the auxiliary processor (123) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (101) itself where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (108)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0034] The memory (130) can store various data used by at least one component (e.g., processor (120) or sensor module (176)) of the electronic device (101). The data can include, for example, software (e.g., program (140)) and input data or output data for commands related thereto. The memory (130) can include volatile memory (132) or non-volatile memory (134).

[0035] The program (140) may be stored as software in the memory (130) and may include, for example, an operating system (142), middleware (144), or an application (146).

[0036] The input module (150) can receive commands or data to be used in a component of the electronic device (101) (e.g., a processor (120)) from an external source (e.g., a user) of the electronic device (101). The input module (150) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0037] The audio output module (155) can output audio signals to the outside of the electronic device (101). The audio output module (155) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0038] The display module (160) can visually provide information to an external party (e.g., a user) of the electronic device (101). The display module (160) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (160) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0039] The audio module (170) can convert sound into an electrical signal, or vice versa, convert an electrical signal into sound. According to one embodiment, the audio module (170) can acquire sound through the input module (150), output sound through the sound output module (155), or an external electronic device (e.g., electronic device (102)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (101).

[0040] The sensor module (176) can detect the operating status (e.g., power or temperature) of the electronic device (101) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (176) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0041] The interface (177) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (101) with an external electronic device (e.g., the electronic device (102)). In one embodiment, the interface (177) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0042] The connection terminal (178) may include a connector through which the electronic device (101) may be physically connected to an external electronic device (e.g., electronic device (102)). According to one embodiment, the connection terminal (178) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0043] A haptic module (179) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (179) can include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0044] The camera module (180) can capture still images and videos. According to one embodiment, the camera module (180) may include one or more lenses, image sensors, image signal processors, or flashes.

[0045] The power management module (188) can manage power supplied to the electronic device (101). According to one embodiment, the power management module (188) can be implemented, for example, as at least a part of a power management integrated circuit (PMIC).

[0046] A battery (189) may power at least one component of the electronic device (101). In one embodiment, the battery (189) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0047] The communication module (190) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (101) and an external electronic device (e.g., electronic device (102), electronic device (104), or server (108)), and the performance of communication through the established communication channel. The communication module (190) may operate independently from the processor (120) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (190) may include a wireless communication module (192) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (194) (e.g., a local area network (LAN) communication module, or a power line communication module). Among these communication modules, the corresponding communication module can communicate with an external electronic device (104) via a first network (198) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (199) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a LAN or WAN)). These various types of communication modules can be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (192) can verify or authenticate the electronic device (101) within a communication network such as the first network (198) or the second network (199) by using subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (196).

[0048] The wireless communication module (192) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). The NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and connection of multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (192) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (192) can support various technologies for securing performance in a high-frequency band, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (192) can support various requirements specified in the electronic device (101), an external electronic device (e.g., the electronic device (104)), or a network system (e.g., the second network (199)). According to one embodiment, the wireless communication module (192) can support a peak data rate (e.g., 20 Gbps or more) for eMBB realization, a loss coverage (e.g., 164 dB or less) for mMTC realization, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC realization.

[0049] The antenna module (197) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (197) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (197) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (198) or the second network (199), may be selected from the plurality of antennas by, for example, the communication module (190). A signal or power may be transmitted or received between the communication module (190) and an external electronic device through the selected at least one antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (197).

[0050] According to various embodiments, the antenna module (197) may form a mmWave antenna module. According to one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high-frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high-frequency band.

[0051] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0052] According to one embodiment, commands or data may be transmitted or received between the electronic device (101) and an external electronic device (104) via a server (108) connected to a second network (199). Each of the external electronic devices (102 or 104) may be the same or a different type of device as the electronic device (101). According to one embodiment, all or part of the operations executed in the electronic device (101) may be executed in one or more of the external electronic devices (102, 104, or 108). For example, when the electronic device (101) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (101) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (101). The electronic device (101) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (101) may provide an ultra-low latency service by using distributed computing or mobile edge computing, for example. In another embodiment, the external electronic device (104) may include an Internet of Things (IoT) device. The server (108) may be an intelligent server utilizing machine learning and / or a neural network. According to one embodiment, the external electronic device (104) or the server (108) may be included in the second network (199).The electronic device (101) can be applied to intelligent services (e.g., smart home, smart city, smart car, or healthcare) based on 5G communication technology and IoT-related technology.

[0053] An electronic device (101) may provide a keyboard translation function of a keyboard application stored within the electronic device (101). The keyboard translation function may represent a function for translating text being written using the keyboard application. For example, the electronic device (101) may, using the keyboard translation function, generate another text written in a second language different from the first language based on a user's input to an executable object of the keyboard application.

[0054] In addition, the user may request a translation of content provided via the electronic device (101). For example, while using a software application providing a social network service (SNS), the user may wish to translate text included in the content within the software application and check the translated text. In this case, the user may copy the text of the content and then check the translated text through another software application that provides a translation function. In addition, if the software application providing the SNS provides its own translation function, the translation function provided by the software application may display the translated text as a replacement for the text to be translated or may display it in an overlapping state with the text to be translated.

[0055] Hereinafter, a device, a method, and a storage medium according to embodiments of the present disclosure propose a software application (hereinafter, a second software application) for providing real-time translation, regardless of a software application (hereinafter, a first software application) running on an electronic device (101). In other words, the device, the method, and the storage medium according to embodiments of the present disclosure can provide real-time translation using the second software application, regardless of whether the first software application provides a translation function. The device, the method, and the storage medium according to embodiments of the present disclosure can perform translation on the content of the first software application through the second software application, and provide the translated text as reconstructed information on a framework. The device, the method, and the storage medium according to embodiments of the present disclosure can provide the translated content to the user in real time on the framework through the second software application while the first software application is in use. Accordingly, the convenience of allowing the user to check the translated content without performing additional input can be provided, regardless of whether the first software application provides a translation function.

[0056] FIG. 2 illustrates an example of a method for an electronic device to translate content included in a screen of a software application and display a screen including translated text, according to one embodiment.

[0057] The electronic device (101) of FIG. 2 may be an example of the electronic device (101) of FIG. 1. For example, the display (160) of FIG. 2 may include at least a portion of the display module (160) of FIG. 1.

[0058] FIG. 2 illustrates examples (200, 250) of a method in which an electronic device (101) translates text of content included in a screen (211) of a first software application and displays a screen (211) containing the translated text. While FIG. 2 illustrates a case in which the first software application is an SNS software application, the embodiments of the present disclosure are not limited thereto. For example, the first software application may include a software application providing a messenger service or a software application providing real-time streaming.

[0059] For example, the first software application may not provide a translation function on its own. However, the embodiments of the present disclosure are not limited thereto. For example, even when a translation function is provided within the first software application, the electronic device (101) may display translated text through the live translation function of the second software application. For example, the electronic device (101) may display translated text through the live translation function of the second software application before obtaining an input for executing the translation function of the first software application. In addition, when the electronic device (101) utilizes the live translation function of the second software application, the electronic device (101) may display the translated text without changing the configuration (or arrangement) of visual objects (or view objects) displayed within the screen (211) of the second software application.

[0060] Referring to example (200), the electronic device (101) can display a screen (211) of the first software application through the display (160). For example, the electronic device (101) can execute the first software application based on obtaining a user input for an icon (or widget) representing the first software application. At this time, the first software application can be executed in the foreground. While the first software application is being executed, the electronic device (101) can display a screen (211), which is an execution screen of the first software application. For example, the screen (211) can be referred to as a user interface (UI). The screen (211) may be a screen that provides an executable object (205) through interaction with the user (e.g., touch input) while the first software application is running, or provides content including visual objects (e.g., text, images).

[0061] For example, the electronic device (101) may display a screen object (212) based on an input to the screen (211). For example, the electronic device (101) may display the screen object (212) based on obtaining the input to an area of ​​the screen (211) (e.g., an executable object (205)). For example, the screen object (212) may be at least partially superimposed on the screen (211). In the case of the first software application providing SNS, the screen object (212) may include a comment on the content posted on the screen (211). However, the embodiments of the present disclosure are not limited thereto. For example, the screen object (212) may pop up based on the input to the executable object (205) of the screen (211).

[0062] For example, the screen object (212) may include at least one view area. In the example (200) of FIG. 2, the screen object (212) may include three view areas (213, 213a, 213b). For example, each of the view areas (213, 213a, 213b) may be distinguished by identification information of users who use the first software application (or are registered with the first software application). For example, the view area (213) may represent a comment written by a first user (e.g., USER B), the view area (213a) may represent a comment written by a second user (e.g., USER C), and the view area (213b) may represent a comment written by a third user (e.g., USER D).

[0063] For example, each of the view areas (213, 213a, 213b) may include content. For example, the view area (213) may include an image (221) set by the first user, identification information (222) of the first user, and text (214) written by the first user. For example, the text (214) may be written in a first language (e.g., English). For example, the text (214) may include a sentence written in English, “worked so hard last night our captain.” However, the embodiments of the present disclosure are not limited thereto. For example, the first language may include a language other than English (e.g., Japanese, Chinese).

[0064] Referring to example (250), the electronic device (101) can display a screen (211) including translated text using a second software application different from the first software application. For example, the electronic device (101) can execute the second software application in the background. For example, the second software application may be an application installed in the electronic device (101) when the electronic device (101) is manufactured. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) can download the second software application. For example, the second software application may be an application for recognizing data based on the user's interactions with the electronic device (101), such as chatting and pictures, that occur while the user uses the electronic device (101).

[0065] According to one embodiment, the electronic device (101) can determine whether a function of the second software application is activated. The function can be activated through a setting or based on a specific event. Specific details on how the function is activated are described below in FIGS. 6, 10A, and 11. For example, the function of the second software application may represent a function of acquiring information about the screen (211) acquired from the first software application (hereinafter, “hierarchy information”), performing a translation of a recognized text portion based on the acquired hierarchy information, and reconstructing the hierarchy information using the translated text. For example, the function of the second software application may be referred to as a live translation function. Specific details on the hierarchy information are described below in FIGS. 5A and 5B. The above layer information may be layer information for a portion of a screen displayed in a display area of ​​a display (160) of an electronic device (101), among screens that can be provided by the first software application.

[0066] According to one embodiment, the electronic device (101) may reconstruct the layer information for the screen (211) of the first software application as the function of the second software application is activated. According to one embodiment, the electronic device (101) may recognize information about a view area (213) within the layer information (hereinafter, first information) and information about text (214) written in the first language (hereinafter, second information) using the layer information obtained from the first software application. According to one embodiment, the electronic device (101) may translate the text (214) extracted from the second information through the second software application. For example, the electronic device (101) may perform the translation of the text (214) using a translation model stored in the second software application or a translation model installed in the electronic device (101). However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may perform translation of text (214) using a server that provides a translation service. For example, the translation model may include a statistical model (e.g., an artificial intelligence model) learned for translation between multiple languages. For example, the languages ​​to be used for the translation may be set by the user, or may be set based on the language set through the text used by the user or the system settings (or global settings) of the electronic device (101). For example, the languages ​​may include the language of the text to be translated (or the source language) and the language of the translated text (or the target language).

[0067] In one embodiment, the electronic device (101) may refrain from (or skip) translation of some texts on the screen (211). For example, the electronic device (101) may not perform translation on some texts on the screen (211). For example, the texts excluded from translation may include user identification information (222) of the screen (211), a uniform resource locator (URL), or text of some executable object (e.g., "Post a reply") on the screen (211). However, embodiments of the present disclosure are not limited thereto.

[0068] According to one embodiment, the electronic device (101) may generate third information about the translated text (264) through the second software application, and reconstruct (or change) the first information using the generated third information. For example, the translated text (264) may be "You worked hard last night, Captain" written in a second language (e.g., Korean). For example, the third information about the translated text (264) may include at least one of the letters of the translated text (264), the color of the translated text (264), or the size of the translated text (264).

[0069] For example, the electronic device (101) can reconstruct (or change, update) the first information using the third information about the translated text (264). The first information can be reconstructed to include the second information and the third information. Specific details about the hierarchical information are described below in FIGS. 5A and 5B . In addition, the information about the screen (211) can be reconstructed (or changed) according to the changed first information.

[0070] Referring to example (250), the electronic device (101) may display a view area (263) including content having text (214) written in the first language and text (264) translated from the first language into a second language using the reconstructed information about the screen (211). For example, the reconstructed information about the screen (211) may include the first information updated with the third information. For example, the view area (263) may include an image (221) set by the first user, identification information (222) of the first user, text (214) written by the first user in the first language, and text (264) translated into the second language. According to one embodiment, the size of the view area (263) may be changed from the size of the view area (213). For example, when the size of the view area (213) is a first size, the size of the view area (263) may have a second size that is larger than the first size. For example, the second size of the view area (263) may be determined based on the reconstructed information (or the reconstructed first information) about the screen (211). In addition, the first size of the view area (213) may be determined based on the information (or the first information, the first information before being reconstructed) about the screen (211).

[0071] The screen object (212) included in the screen (211) of example (250) may have the same size as the screen object (212) included in the screen (211) of example (200). In other words, information indicating the size of the screen object (212) included in the reconstructed information for the screen (211) may not be reconstructed. However, the number of view areas included in the screen object (212) may be changed. The screen object (212) of example (200) may include three view areas (213, 213a, 213b), and the screen object (212) of example (250) may include two view areas (263, 263a). For example, each of the view areas (263, 263a) included in the screen object (212) of example (250) may further include translated text. Accordingly, the view area (263) may have a larger size than the view area (213), and the view area (263a) may have a larger size than the view area (213a). In other words, since the sizes of the view areas are changed and the size of the screen object (212) is maintained in order to display the translated text, the number of view areas included in the screen object (212) may be changed.

[0072] According to one embodiment, the translated text (264) of the view area (263) may have a first visual effect, and the text (214) of the view area (263) may have a second visual effect. For example, when the first visual effect is a first color, the second visual effect may be a second color with lower saturation than the first color. In this case, unlike the view area (263), the text (214) of the view area (213) may have the first visual effect. In other words, the electronic device (101) may apply the first visual effect to the translated text (264) to increase the user's recognition. In the above example, the case where the visual effect includes the color of the text is described, but the embodiments of the present disclosure are not limited thereto. For example, the visual effect may include a shade, a size, a thickness, or a font (or a typeface) of the text.

[0073] According to one embodiment, the electronic device (101) may set the third information for the text (264) according to the first visual effect, and may set (or change, update) the second information for the text (214) according to the second visual effect. In other words, each of the second information and the third information may be set according to a visual effect.

[0074] In the examples (200, 250) of FIG. 2, an example is shown in which the visual effect of the text (214) of the view area (263) changes from the first visual effect to the second visual effect as the translated text (264) of the view area (263) is displayed, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may display the view area (263) including the translated text (264) and the text (214) having the first visual effect while displaying the translated text (264) of the view area (263). Alternatively, for example, the electronic device (101) may display the view area (263) including the text (214) having the first visual effect and the translated text (264) having the second visual effect while displaying the translated text (264) of the view area (263).

[0075] Additionally, examples (200, 250) of FIG. 2 illustrate a view area (263) including translated text (264) and text (214) positioned below the translated text (264), but embodiments of the present disclosure are not limited thereto. For example, the view area (263) may also include text (214) and translated text (264) positioned below the text (214).

[0076] In the examples (200, 250) of FIG. 2, for convenience of explanation, a view area (or screen) further including text translated from the first language (e.g., English) into the second language (e.g., Korean) is illustrated, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may further display text translated from a plurality of first languages ​​(e.g., English, Japanese) into the second language (e.g., Korean). Various examples of how the electronic device (101) displays translated text are described below in FIGS. 9A to 9D .

[0077] FIG. 3A illustrates an example block diagram of a functional configuration of an electronic device, according to one embodiment.

[0078] The electronic device (101) of FIG. 3A may be an example of the electronic device (101) of FIG. 1. For example, the electronic device (101) may be connected to an external electronic device (not shown) based on a wired network and / or a wireless network. For example, the wired network may include a network such as the Internet, a local area network (LAN), a wide area network (WAN), or a combination thereof. For example, the wireless network may include a network such as long term evolution (LTE), 5g new radio (NR), wireless fidelity (WiFi), Zigbee, near field communication (NFC), Bluetooth, Bluetooth low-energy (BLE), or a combination thereof. The electronic device (101) may be directly connected to the external electronic device, or may be indirectly connected via one or more routers and / or access points (APs). For example, the external electronic device may include a server that provides a translation service.

[0079] Referring to FIG. 3A, according to one embodiment, the electronic device (101) may include at least one of a processor (310), a display (320), and a memory (330). However, the embodiments of the present disclosure are not limited thereto. For example, the processor (310), the display (320), and / or the memory (330) may be electrically and / or operably coupled with each other by a communication bus. Hereinafter, operably coupled hardware components may mean that a direct connection or an indirect connection is established between the hardware components, either wired or wireless, such that a second hardware component is controlled by a first hardware component among the hardware components. Although illustrated based on different blocks, the embodiment is not limited thereto, and some of the hardware components illustrated in FIG. 3A (e.g., at least a portion of the processor (310), the display (320), and the memory (330)) may be included in a single integrated circuit such as a system on a chip (SoC). The type and / or number of hardware components included in the electronic device (101) is not limited to those illustrated in FIG. 3A. For example, the electronic device (101) may include only some of the hardware components illustrated in FIG. 3A.

[0080] According to one embodiment, the processor (310) of the electronic device (101) may include a hardware component for processing data based on one or more instructions. The hardware component for processing data may include, for example, an arithmetic and logic unit (ALU), a floating point unit (FPU), and a field programmable gate array (FPGA). As an example, the hardware component for processing data may include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processing (DSP), a microcontroller (MCU), and / or a neural processing unit (NPU). The number of processors (310) may be one or more. For example, the processor (310) may have a multi-core processor structure such as a dual core, a quad core, or a hexa core. The processor (310) of FIG. 3A may be substantially identically applied to the processor (120) of FIG. 1.

[0081] For example, the processor (310) of the electronic device (101) may include various processing circuits and / or multiple processors. For example, the term "processor" as used herein, including in the claims, may include various processing circuits including at least one processor, one or more of which may be configured to individually and / or collectively perform the various functions described below in a distributed manner. As used herein, when "processor," "at least one processor," and "one or more processors" are described as being configured to perform various functions, these terms encompass, for example, and without limitation, situations where one processor performs some of the recited functions and other processor(s) perform other parts of the recited functions, and also situations where one processor may perform all of the recited functions. Additionally, the at least one processor may include a combination of processors that perform the various functions enumerated / disclosed, for example, in a distributed manner. At least one processor is capable of executing program instructions to accomplish or perform various functions.

[0082] According to one embodiment, the display (320) of the electronic device (101) can output visualized information to the user. The number of displays (320) included in the electronic device (101) may be one or more. For example, the display (320) may be controlled by the processor (310) and / or a graphic processing unit (GPU) (not shown) to output visualized information to the user. The display (320) may include a flat panel display (FPD) and / or electronic paper. The FPD may include a liquid crystal display (LCD), a plasma display panel (PDP), a digital mirror device (DMD), one or more light emitting diodes (LEDs), and / or a micro LED. The LEDs may include organic LEDs (OLEDs). Specific details regarding the display (320) of FIG. 3A may be substantially identical to those regarding the display module (160) of FIG. 1.

[0083] According to one embodiment, the electronic device (101) may include a memory (330). The memory (330) may include a hardware component for storing data and / or instructions input to and / or output from the processor (310). The memory (330) may include, for example, a volatile memory such as a random-access memory (RAM), and / or a non-volatile memory such as a read-only memory (ROM). The volatile memory may include, for example, at least one of a dynamic RAM (DRAM), a static RAM (SRAM), a cache RAM, and a pseudo SRAM (PSRAM). The non-volatile memory may include, for example, at least one of a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a flash memory, a hard disk, a compact disc, and an embedded multimedia card (eMMC). The specific details of the memory (330) of FIG. 3a can be applied substantially identically to the details of the memory (130) of FIG. 1.

[0084] Referring to FIG. 3A, according to one embodiment, one or more instructions (or commands) representing operations and / or actions to be performed on data by a processor (310) of the electronic device (101) may be stored in the memory (330) of the electronic device (101). A set of one or more instructions may be referred to as a program, firmware, an operating system, a process, a routine, a sub-routine, and / or an application. Hereinafter, when an application is installed in the electronic device (e.g., the electronic device (101)), it may mean that one or more instructions provided in the form of an application are stored in the memory (330), and that the one or more applications are stored in a format executable by the processor of the electronic device (e.g., a file having an extension designated by the operating system of the electronic device (101)). According to one embodiment, the electronic device (101) may perform the operations of FIGS. 4, 6, 7, 8, and 14 by executing one or more instructions stored in the memory (330).

[0085] Referring to FIG. 3A, according to one embodiment, programs installed in the electronic device (101) may be classified into one of different layers, including an application layer (331), a framework layer (332), and / or a hardware abstraction layer (HAL) (333), based on a target. For example, programs (e.g., drivers) designed to target the hardware (e.g., display (320)) of the electronic device (101) may be classified within the hardware abstraction layer (333). For example, programs (e.g., view framework, translation framework) designed to target at least one of the hardware abstraction layer (333) and / or the application layer (331) may be classified within the framework layer (332). Programs classified into the framework layer (332) may provide an executable API (application programming interface) based on other programs.

[0086] Referring to FIG. 3A, according to one embodiment, programs designed to target users controlling electronic devices (101) may be classified within the application layer (331). For example, programs classified within the application layer (331) may include a first software application (331a) that provides SNS, messenger, or real-time streaming services, and a second software application (331b) that provides content capture services. However, the embodiments of the present disclosure are not limited thereto. For example, programs classified within the application layer (331) may call APIs to cause the execution of functions supported by programs classified within the framework layer (332).

[0087] Referring to FIG. 3A, according to one embodiment, the memory (330) may include a translation model (340). For example, the translation model (340) may be used to translate a translation target (e.g., content with text or content with an image) requested from a second software application (331b). In FIG. 3A, the translation model (340) is illustrated as being stored in the memory (330), but the embodiments of the present disclosure are not limited thereto. For example, the translation model (340) may be stored in the second software application (331b). Alternatively, for example, the electronic device (101) may perform translation through an external electronic device that provides a translation function instead of the translation model (340).

[0088] According to one embodiment, the memory (330) may store information about multiple languages ​​for performing the translation. For example, the information about the multiple languages ​​may be referred to as a language pack. For example, the language pack may be learned within the translation model (340) or stored in a separate storage space.

[0089] Although not illustrated in FIG. 3A, the electronic device (101) according to one embodiment may include a communication circuit. For example, the communication circuit may include hardware for supporting transmission and / or reception of electrical signals between the electronic device (101) and the external electronic device. The communication circuit may include, for example, at least one of a modem (MODEM), an antenna, and an optical / electronic (O / E) converter. The communication circuit may support transmission and / or reception of electrical signals based on various types of communication means, such as Ethernet, Bluetooth, Bluetooth low energy (BLE), ZigBee, long term evolution (LTE), and 5G new radio (NR). Specific details regarding the communication circuit of FIG. 3A may be substantially identical to the communication module (190) and / or antenna module (197) of FIG. 1.

[0090] Although not illustrated in FIG. 3A, according to one embodiment, the electronic device (101) may include an output means for outputting information in a form other than a visualized form. For example, the electronic device (101) may include a speaker for outputting an acoustic signal or sound. For example, the electronic device (101) may include an actuator (or motor) for providing haptic feedback based on vibration.

[0091] FIG. 3b illustrates an example of the structure of a software application and framework of an electronic device according to one embodiment.

[0092] The electronic device (101) of FIG. 3B may be an example of the electronic device (101) of FIG. 3A. For example, the structure (350) of the software application and the framework of the electronic device (101) illustrated in FIG. 3B may represent an example of the application layer (331) and the framework layer (332) of the memory (330) of the electronic device (101) of FIG. 3A.

[0093] Referring to FIG. 3B, the structure (350) may include a first software application (360), a second software application (370), a view framework (381), and a translation framework (382). In the example of FIG. 3B, the structure of the framework layer in which the view framework (381) and the translation framework (382) are implemented as separate components is illustrated, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may include a framework (380) in which the view framework (381) and the translation framework (382) are implemented as a single component. The framework (380) of FIG. 3B may have a structure or function that may be changed depending on the operating system supported by the electronic device (101). In addition, the embodiments of the present disclosure are not limited to the name of the framework (380), but may also be applied to a configuration that performs substantially the same function as the framework (380).

[0094] According to one embodiment, the first software application (360) may include a software application that provides a service for performing interaction with a user based on a screen (or execution screen) displayed through the display (320) of the electronic device (101) while running on the foreground of the electronic device (101). For example, the first software application (360) may include an SNS, messenger, or streaming service. However, the embodiments of the present disclosure are not limited thereto.

[0095] According to one embodiment, the second software application (370) may be an application (or widget) for providing a recommended application (or service) whose compatibility with the electronic device (101) is guaranteed. For example, the second software application (370) may obtain information from the software application of the electronic device (101) by using a content capture service (or content capture function) (or by analyzing a screen (e.g., screen (211) of FIG. 2) displayed on the display of the electronic device (101). For example, the second software application (370) may provide a live translation function (or function) based on the information. For example, the live translation function of the second software application (370) may represent a function of acquiring hierarchical information obtained from the first software application (360), performing translation of a portion of text recognized through the framework (380) (or the view framework (381) and the translation framework (382)) using the acquired hierarchical information, and reconstructing the hierarchical information using the translated text. The hierarchical information may be hierarchical information for a portion of a screen displayed in a display area of ​​the display (160) of the electronic device (101) among screens that may be provided by the first software application (360).

[0096] In FIG. 3b, the second software application (370) providing the live translation function is described as being implemented at the application layer, similar to the first software application (360). However, the embodiments of the present disclosure are not limited thereto. For example, the live translation function may also be implemented within the framework (380).

[0097] In one embodiment, the second software application (370) may include a translation model (375). For example, the translation model (375) may be implemented as a function of the second software application (370). However, the embodiments of the present disclosure are not limited thereto. For example, the translation model (375) may be stored in a storage space included in an electronic device (101) external to the second software application (370). For example, the translation model (375) may be used to translate a translation target (e.g., content having text or content having images) requested from the second software application (370).

[0098] According to one embodiment, the view framework (381) may be a framework directly connected to the first software application (360). For example, the view framework (381) may recognize a portion of the view area that requires translation using the hierarchical information (or at least a portion of the hierarchical information) obtained from the second software application (370). For example, the portion that requires translation may include a portion of the view area that has been newly added or updated. The view framework (381) may recognize the portion and transmit the recognized portion to the translation framework (382).

[0099] In one embodiment, the translation framework (382) may forward a request for translation of the portion received from the view framework (381) to the translation model (375). For example, the translation framework (382) may recognize that translation of the portion is required and forward the request containing information about the portion to the translation model (375).

[0100] Referring to FIG. 3B, the electronic device (101) can obtain layer information (391) from the first software application (360). According to one embodiment, the layer information (391) can be obtained based on the content capture function of the second software application (370). For example, the execution of the content capture function can be performed at a specified period or when an event, such as a change in the screen of the first software application (360), occurs. For example, the change in the screen can be caused based on a user's scroll input to the electronic device (101), a preset gesture, or an input to an external electronic device (e.g., a stylus pen, a true wireless stereo (TWS)).

[0101] For example, the electronic device (101) can transmit layer information (391) obtained from the first software application (360) to the second software application (370).

[0102] According to one embodiment, the electronic device (101) may extract information about a changed portion from the hierarchy information (391) using the second software application (370). For example, the electronic device (101) may identify identification information of a view area including content to be translated among at least one view area included in the hierarchy information (391) and information indicating the first software application (360). For example, the electronic device (101) may transmit a request (392) including the identified information to the view framework (381). For example, the request (392) may include the hierarchy information (391). For example, the request (392) may include a request for translation and reconfiguration of view objects on the framework (380) using the live translation function of the second software application (370).

[0103] According to one embodiment, the electronic device (101) may use the view framework (381) to recognize first information of the view area corresponding to the identification information from the hierarchy information (391) by using the identification information of the view area included in the request (392). For example, the electronic device (101) may recognize text requiring translation (or a portion requiring translation) within the view area (or the first information of the view area). For example, the electronic device (101) may recognize second information (393) regarding the text among the first information. For example, the electronic device (101) may transmit the second information (393) from the view framework (381) to the translation framework (382). For example, the first information may include user identification information within the view area, an image included in the view area, and second information (393) regarding content (e.g., the text) included in the view area. Specific details related to this are described later in Figures 5a and 5b.

[0104] According to one embodiment, the electronic device (101) may identify, through the second software application (370) or the view framework (381), some texts on the screen that are to be refrained from (or skipped) from translation. For example, the electronic device (101) may not perform translation on some texts on the screen. For example, the some texts excluded from translation may include user identification information of the screen, a uniform resource locator (URL), or the text of some executable object (e.g., "Post a reply") on the screen. However, the embodiments of the present disclosure are not limited thereto.

[0105] According to one embodiment, the electronic device (101) may transmit a request (394) for translation of the text in the second information (393) to the translation model (375) using the translation framework (382). For example, the request (394) may include the second information (393). For example, the request (394) may include a string of text of the second information (393). However, the embodiments of the present disclosure are not limited thereto. For example, the request (394) may further include at least one of the color of the text, the size of the text, the shading of the text, the boldness of the text, or the font information of the text, together with the string of text of the second information (393). For example, information provided in the request (394) together with the string of text of the second information (393) may include information used to apply (or provide) a visual effect to the text. Alternatively, for example, the request (394) may include the language of the text (or source language) and the language to be translated (or target language).

[0106] In the example of FIG. 2, the electronic device (101) may display a screen (211) in which a screen object (212) pops up based on an input for an executable object (205) while displaying a first software application (360). As the screen (211) changes based on the input, the electronic device (101) may obtain layer information about the screen (211) of the first software application (360) through the second software application (370). For example, the layer information may include first information about a view area (213) and second information about text (214) within the view area (213). The electronic device (101) may transmit a request (392) including the layer information to the view framework (381). The electronic device (101) can recognize the second information regarding the text (214), which is content requiring translation, from the first information of the hierarchical information, through the view framework (381). The electronic device (101) can transmit a request (394) including the second information from the view framework (381) to the translation framework (382). The electronic device (101) can recognize a string regarding the text (214) in the second information through the translation framework (382). Thereafter, the electronic device (101) can transmit a request (394) including the string from the translation framework (382) to the translation model (375).

[0107] According to one embodiment, the electronic device (101) may recognize the first language in which the text (214) is written within the electronic device (101) by recognizing the second information about the text (214). The electronic device (101) may determine whether information (or a language pack) about the first language is stored. For example, if the information about the first language is not stored but can be downloaded, the electronic device (101) may display a visual object for downloading the information about the first language. The electronic device (101) may download the information about the first language based on an input to the visual object. If the information about the first language is downloaded, the electronic device (101) may perform a translation of the text (214) using a translation model (375).

[0108] According to one embodiment, the electronic device (101) can translate the text of the second information (393) included in the request (394) through the translation model (375). For example, the electronic device (101) can translate the text written in the first language into another text (395) written in the second language, which is the language to be translated, through the translation model (375). For example, the second language (or target language), which is the language to be translated, can be provided to the translation model (375) from the second software application (370). For example, the second software application (370) can recognize the language of the text being input (or the language of the currently set keyboard (keyboard application)) or the language set in the system settings of the electronic device (101) as the second language. For example, the first language can be recognized from the second information (393) (or a string of the second information (393)) by the translation model (375).

[0109] According to one embodiment, the electronic device (101) can reconstruct (or change, update) the second information (393) using other text (395). For example, the electronic device (101) can reconstruct the hierarchical information (391) (or the first information) using the second information (393) and third information about the other text (395) through the view framework (381). For example, the hierarchical information (391) (or the first information) can include the second information (393) and the third information. For example, the electronic device (101) can generate hierarchical information (396) that further includes the third information through the view framework (381). In other words, the electronic device (101) can generate hierarchical information (396) that includes the second information (393) and the third information from the hierarchical information (391) that includes the second information (393). In one example, the second information (393) included in the layer information (396) may be different from the second information (393) included in the layer information (391). For example, the second information (393) included in the layer information (396) may have a setting value for changing a visual effect changed from the second information (393) included in the layer information (391).

[0110] According to one embodiment, the electronic device (101) may transmit the layer information (396) including the third information from the view framework (381) to the first software application (360). Thereafter, the electronic device (101) may display the screen of the first software application (360) that has been changed based on the layer information (396). For example, the first software application (360) may reconstruct the view area to correspond to the third information and display the screen including the reconstructed view area.

[0111] FIG. 4 illustrates an example of an operational flow for a method for an electronic device to display a screen including content of a first software application depending on whether a function for translation of a second software application is activated, according to one embodiment.

[0112] At least some of the methods of FIG. 4 may be performed by the electronic device (101) of FIG. 3A. For example, at least some of the methods may be controlled by the processor (310) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0113] According to one embodiment, in operation (400), the electronic device (101) may execute a first software application. For example, the electronic device (101) may execute the first software application based on obtaining a user input for an icon representing the first software application. In this case, the first software application may be executed in the foreground.

[0114] According to one embodiment, while the first software application is executed, the electronic device (101) may display a screen, which is an execution screen of the first software application. For example, the screen may be referred to as a user interface (UI). For example, the screen may be a screen that provides an executable object through interaction with the user (e.g., touch input) while the first software application is executed, or provides content including visual objects (e.g., text, images). For example, the screen may include information about a plurality of regions. For example, the screen may include a first region where the screen is displayed, one or more second regions included in the first region, one or more third regions included in each of the one or more second regions, and one or more fourth regions included in each of the one or more third regions. For example, the one or more second regions may be defined as regions where one or more screen objects included in the screen are displayed. For example, each of the one or more screen objects may represent a screen object that is displayed (or pops up) based on an input to the screen. For example, the one or more third regions may be defined as regions in which one or more view regions included in one screen object are displayed. For example, the one or more fourth regions may be defined as regions in which one or more texts included in one view region are displayed. For example, the fourth region may include a region in which content including text is located. However, the embodiments of the present disclosure are not limited thereto. For example, the one or more fourth regions may include content including not only text but also images.

[0115] According to one embodiment, in operation (405), the electronic device (101) may obtain layer information from the first software application. For example, the electronic device (101) may obtain the layer information by using a content capture function of the second software application while the second software application is running in the background. For example, the execution of the content capture function may be performed at a specified period or when an event, such as a change in the screen of the first software application, occurs. For example, the layer information may include information about a first region, a second region, a third region, and a fourth region. The layer information may be layer information about a portion of a screen displayed in a display area of ​​the display (160) of the electronic device (101), among screens that may be provided by the first software application.

[0116] According to one embodiment, in operation (410), the electronic device (101) may determine whether a function of the second software application is activated. For example, the function may be activated through a setting of the electronic device (101) or based on a specific event. For example, the function of the second software application may represent a live translation function that obtains the hierarchical information obtained from the first software application, translates a portion of text (e.g., text of the fourth region) recognized through a framework using the obtained hierarchical information, and reconstructs the hierarchical information using the translated text.

[0117] According to one embodiment, the electronic device (101) may activate the function of the second software application when the first software application is a software application that does not provide a translation function. For example, the electronic device (101) may set the execution of the first software application as the specific event for activating the function of the second software application. Accordingly, the electronic device (101) may activate the function of the second software application when the first software application is executed. Alternatively, for example, the electronic device (101) may display an icon (or menu) for activating the function of the second software application through at least a portion of a display area of ​​the electronic device (101) (e.g., the display module (160) of FIG. 1) when the first software application is executed. Based on at least a portion of an input for the icon, the electronic device (101) may activate the function of the second software application.

[0118] While the above example illustrates a case where the first software application does not provide its own translation function, the embodiments of the present disclosure are not limited thereto. For example, even if the first software application provides its own translation function, the electronic device (101) may activate the function of the second software application.

[0119] In operation (410), the electronic device (101) may perform operation (420) if the function is activated. Alternatively, in operation (410), the electronic device (101) may perform operation (415) if the function is deactivated.

[0120] According to one embodiment, in operation (415), the electronic device (101) may maintain the display of the screen. For example, the electronic device (101) may maintain the display of the screen when the function of the second software application is deactivated. In other words, the electronic device (101) may display the screen without changing the screen of the first software application.

[0121] According to one embodiment, in operation (420), the electronic device (101) can recognize first information and second information. For example, the electronic device (101) can recognize the first information and the second information from the hierarchical information. For example, the electronic device (101) can recognize the first information and the second information through the second software application. For example, when the function of the second software application is activated, the electronic device (101) can recognize the second information for the fourth area including the text to be translated within the hierarchical information and the first information for the third area including the fourth area through the second software application.

[0122] According to one embodiment, in operation (425), the electronic device (101) may transmit the first information and the second information to the framework. For example, the electronic device (101) may transmit the first information and the second information recognized through the second software application to the framework. For example, the framework may include a view framework (e.g., view framework (381) of FIG. 3B) that manages view objects included in the hierarchical information. For example, the view objects may include content including a screen, a screen object, a view area, and text (and / or images).

[0123] According to one embodiment, in operation (430), the electronic device (101) may determine whether the text includes text written in a language different from the user language. For example, the electronic device (101) may determine, based on the second information, whether the text of the content included in the fourth area includes text written in a language different from the user language. For example, the user language may be a language set by the user and may indicate a target language. For example, the electronic device (101) may determine, when the function is activated, whether text written in a language different from the user language exists among the texts included in the screen.

[0124] According to one embodiment, the electronic device (101) may identify, through the second software application or the framework (e.g., view framework), some texts on the screen that are to be refrained from (or skipped) from translation. For example, the electronic device (101) may not perform translation on some texts on the screen. For example, the some texts excluded from translation may include user identification information of the screen, a uniform resource locator (URL), or the text of some executable object (e.g., "Post a reply") on the screen. However, the embodiments of the present disclosure are not limited thereto.

[0125] In operation (430), the electronic device (101) may perform operation (435) if the text on the screen includes text written in a language different from the user language. Conversely, in operation (430), the electronic device (101) may perform operation (415) if the text on the screen does not include text written in a language different from the user language.

[0126] According to one embodiment, in operation (435), the electronic device (101) may execute a model for translation (hereinafter, referred to as a translation model). For example, if the text on the screen includes text written in a language different from the user language, the electronic device (101) may transmit a request for translation from the translation framework of the framework to the second software application. For example, the request may include the second information. Or, for example, the request may include a string of text of the second information.

[0127] In response to the request, the electronic device (101) may execute the translation model to translate text written in a language different from the user language. For example, the translation model may be implemented as a function of the second software application. However, the embodiments of the present disclosure are not limited thereto. For example, the translation model may be installed in a storage space within the memory (330) of the electronic device (101). If the translation model is installed in the storage space, the translation model may be executed upon a call from the second software application that received the request.

[0128] According to one embodiment, in operation (440), the electronic device (101) may generate a translated text. For example, the electronic device (101) may transmit information (e.g., the second information) regarding the text written in a language different from the user language recognized by the framework to the translation model. For example, the electronic device (101) may use the second information through the translation model to translate the text within the second information. For example, the electronic device (101) may generate another text translated into the user language from the text. For example, the electronic device (101) may generate third information regarding the other text.

[0129] According to one embodiment, the electronic device (101) may translate the text within the request through the translation model. For example, the electronic device (101) may translate the text written in a language different from the user language into another text written in the user language (or target language) through the translation model. For example, the user language (or target language) may be provided to the translation model from the second software application. For example, the second software application may recognize the language of the text being input or a language set in the system settings of the electronic device (101) as the user language (or target language).

[0130] While the above example illustrates a method for performing translation using the translation model, the embodiments of the present disclosure are not limited thereto. For example, without using the translation model, the electronic device (101) may also translate text written in a language different from the user's language through an external electronic device (e.g., a server) that provides a translation service.

[0131] According to one embodiment, in operation (445), the electronic device (101) may transmit the third information to the framework. The framework may include the translation framework or the view framework. If the framework is the translation framework, the third information may be transmitted to the view framework.

[0132] According to one embodiment, in operation (450), the electronic device (101) may update the hierarchical information. For example, the electronic device (101) may update (or reconfigure, change) the hierarchical information including the second information based on the third information into the hierarchical information including the second information and the third information. In other words, the hierarchical information may further include the third information along with the second information.

[0133] According to one embodiment, in operation (455), the electronic device (101) may display a changed screen. For example, the electronic device (101) may change (or reconfigure, update) the screen based on the hierarchical information including the third information and the second information while the first software application is running, and display the changed screen through the display (320). For example, the changed screen may include the text and the translated other text within the view area of ​​the screen.

[0134] The screen based on the hierarchical information including the second information of operation (405) may include a view area including the text. Alternatively, the changed screen based on the hierarchical information including the second information and the third information of operation (455) may include a view area including the text and the other text. For example, the view area including the text (or the view area based on the second information) may have a first size. For example, the size of the view area including the text and the other text (or the view area based on the second information and the third information) may have a second size larger than the first size.

[0135] According to one embodiment, the other text of the view area including the text and the other text may have a first visual effect, and the text of the view area including the text and the other text may have a second visual effect. For example, when the first visual effect is a first color, the second visual effect may be a second color with lower saturation than the first color. In this case, unlike the view area including the text and the other text, the text of the view area including the text may have the first visual effect. In other words, the electronic device (101) may apply the first visual effect to the other text in order to increase the user's recognition. In order to change the visual effect of the text from the first visual effect to the second visual effect, the second information of the layer information updated in operation (450) may be set differently from the second information of the layer information acquired in operation (405).

[0136] FIG. 5A illustrates an example of hierarchy information constituting a screen of a first software application according to one embodiment. FIG. 5B illustrates an example of first information regarding a view area of ​​the hierarchy information and second information regarding text according to one embodiment.

[0137] Referring to FIG. 5A, an example of layer information (500) obtained from the first software application is illustrated. For example, the layer information (500) may include information on view objects included in the screen of the first software application. The layer information (500) may be layer information on a portion of a screen displayed in a display area of ​​the display (160) of the electronic device (101), among screens that may be provided by the first software application. For example, the screen of the first software application may be referred to as a user interface (UI). For example, the screen of the first software application may be a screen that provides an executable object through interaction with a user (e.g., touch input) while the first software application is running, or provides content including a visual object (e.g., text, image).

[0138] For example, the layer information (500) may include information (510) about a first region, information (521, 522, 523) about one or more second regions, information (531, 532, 533) about one or more third regions, and information (540) about one or more fourth regions. The information (531) about the third region (or information (532), information (533)) may be referred to as first information (531) (or first information (532), first information (533)) hereinafter. The information (540) about the fourth region may be referred to as second information (540) hereinafter. In FIG. 5A, an example is described in which one first region includes three second regions, one of the three second regions includes three third regions, and one third region includes one fourth region; however, the embodiments of the present disclosure are not limited thereto. The number of second regions included in one first region, the number of third regions included in one second region, and the number of fourth regions included in one third region may be varied.

[0139] For example, the first region may represent an region where the screen (e.g., the screen (211) of FIG. 2) is displayed. For example, the second region may represent an region where a screen object included in the screen (e.g., the screen object (212) of FIG. 2) is displayed. For example, the third region may represent an region where a view region included in the screen object (e.g., the view region (213) of FIG. 2) is displayed. For example, the fourth region may represent an region where content included in the view region (e.g., text (214) of FIG. 2) is displayed. For example, the content may include at least one of text or an image.

[0140] According to one embodiment, the first information (531) for the third area may include user identification information (542) within the view area, an image (541) included in the view area, and second information (540) for the fourth area included in the third area. For example, the identification information (542) may be identification information (222) within the view area (213) of FIG. 2. For example, the image (541) may be image (221) within the view area (213) of FIG. 2.

[0141] Referring to FIG. 5a, hierarchical information (500) may include the hierarchical status of view objects (e.g., the screen, the screen object, the view area, and the content) included in the screen, and the number of view objects. Specific details related thereto are described below in FIG. 5b.

[0142] FIG. 5B illustrates an example (550) of first information (531) for a third area and second information (540) for a fourth area included in the third area. The parameters included in each of the first information (531) and the second information (540) illustrated in example (550) are merely exemplary, and the present disclosure is not limited thereto. For example, the first information (531) may be referred to as a view. For example, the second information (540) may be referred to as a textview. However, the present disclosure is not limited thereto. For example, if the content of the second information (540) includes an image, the second information (540) may be referred to as an imageview.

[0143] According to one embodiment, the first information (531) may include an identifier (551) of a view area, coordinates (552, 553) indicating a location of the view area, and length values ​​(554, 555) indicating a size of the view area. For example, the identifier (551) may represent information for distinguishing between a plurality of view areas included in the second area (or one screen object) including the third area. For example, the coordinates (552, 553) may indicate a location of the view area when a designated location on the screen is assumed to be the origin (0, 0). For example, the designated location may be a specific point (e.g., the upper left corner) of the screen. For example, the location of the view area indicated by the coordinates (552, 553) may be a designated location of the view area. For example, coordinate (552) may indicate an x-coordinate, and coordinate (553) may indicate a y-coordinate. For example, the specified location of the view area may be the center of the view area. For example, coordinates (552, 553) may change according to a specified rule when the second information (540) is changed or new information (e.g., a text view for translated text) is added. For example, length values ​​(554, 555) may indicate the size of the view area. For example, length value (554) may indicate the width (or horizontal length) of the view area. For example, length value (555) may indicate the height (or vertical length) of the view area. However, the embodiments of the present disclosure are not limited thereto. For example, first information (531) may include information indicating the design of the view area.

[0144] According to one embodiment, the second information (540) may include a string (561) of text included in the view area, a color (562) of the text, and a size (563) of the text. However, the embodiment of the present disclosure is not limited thereto. For example, the second information (540) may include a shade, a size, a boldness, or a font (or a font) of the text included in the view area. In addition, although FIG. 5B illustrates an example (550) in which the content included in the view area is text, the embodiment of the present disclosure is not limited thereto. For example, if the content included in the view area is an image, the second information (540) may include information about the image included in the view area (e.g., the image, the size of the image).

[0145] FIG. 6 illustrates an example of an operational flow for a method of activating a function for translation of a second software application, according to one embodiment.

[0146] At least some of the methods of FIG. 6 may be performed by the electronic device (101) of FIG. 3A. For example, at least some of the methods may be controlled by the processor (310) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0147] The method of FIG. 6 illustrates examples of operations performed within operation (410) of FIG. 4. For example, operations (600), (605), and (610) may be performed within operation (410).

[0148] According to one embodiment, in operation (600), the electronic device (101) may determine whether the function (or live translation function) of the second software application is activated based on system settings. For example, the system settings may include an application capable of performing overall settings of the electronic device (101). For example, the electronic device (101) may determine whether the function of the second software application is activated within a menu for a keyboard application of the system settings.

[0149] In one embodiment, the system settings may include the menu for the keyboard application. For example, the menu may include a submenu for activating the function of the second software application. For example, the system settings may include a plurality of menus. Among the plurality of menus, the menu for the keyboard application may include a plurality of submenus including the submenu related to the second software application. For example, the submenu related to the second software application may include an icon for activating the function of the second software application and icons for setting languages ​​(e.g., source language, target language) to be used within the function. Examples of the system settings, the menu, and the submenu are illustrated in FIG. 11 below.

[0150] In operation (600), if the function of the second software application is activated, the electronic device (101) can perform operation (610). Conversely, in operation (600), if the function of the second software application is deactivated, the electronic device (101) can perform operation (605).

[0151] According to one embodiment, in operation (605), the electronic device (101) may determine whether an event for activating the function of the second software application has occurred. For example, when the electronic device (101) obtains an input for an icon for a function for translating text being written using a software application (e.g., the keyboard application) (hereinafter, referred to as a keyboard translation function), the electronic device (101) may activate the function of the second software application. For example, the event may include obtaining an input for an icon for translating text displayed in a bar-shaped visual object within an executable object having a keyboard shape provided through the keyboard application. For example, the bar-shaped visual object may represent a portion where text written by a user is displayed based on at least a portion of the input to the executable object. However, the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may activate the function of the second software application based on the event of obtaining a designated gesture or designated input. For example, the designated gesture may include a movement of the user (e.g., eye movement, hand) obtained through a camera of the electronic device (101) (e.g., camera module (180) of FIG. 1) or a movement of an external electronic device (e.g., stylus pen) connected to the electronic device (101). For example, the designated input may include a touch input of the user (e.g., long press, double touch, input to a pop-up button) and a touch input to an external electronic device (e.g., stylus pen) connected to the electronic device (101).

[0152] In operation (605), if the event for activating the function of the second software application occurs, the electronic device (101) may perform operation (610). Conversely, if the event for activating the function of the second software application does not occur in operation (605), the electronic device (101) may perform operation (600) again.

[0153] According to one embodiment, in operation (610), the electronic device (101) may detect a translation target. For example, when the function of the second software application is activated, the electronic device (101) may detect a translation target (e.g., content having text within the fourth region) within the hierarchical information acquired from the first software application. Accordingly, the electronic device (101) may recognize second information about the content within the hierarchical information and first information about the view region in which the content is displayed.

[0154] Referring to FIG. 6, the activation of the function of the second software application based on at least a portion of the input for the system settings may be understood as the generation of an automatic event for activating the function of the second software application. Furthermore, the activation of the function of the second software application based on the designated gesture or the designated input may be understood as the generation of a passive event for activating the function of the second software application. In the example of FIG. 6, it is illustrated that whether the automatic event is detected is first determined, and then whether the passive event is detected is determined, but the embodiments of the present disclosure are not limited thereto.

[0155] FIG. 7 illustrates an example of a signal flow for a method of generating a translation model and providing a translation function using the generated translation model, according to one embodiment.

[0156] The first software application (701) of FIG. 7 may be an example of the first software application (331a) of FIG. 3a and the first software application (360) of FIG. 3b. The framework (702) of FIG. 7 may be included in the framework layer (332) of FIG. 3a and may be an example of the framework (380) of FIG. 3b. The second software application (703) of FIG. 7 may be an example of the second software application (331b) of FIG. 3a and the second software application (370) of FIG. 3b. The first software application (701), the framework (702), the second software application (703), and the translation model (704) may be included in the electronic device (101).

[0157] FIG. 7 illustrates an example (710) of a method for generating a translation model (704), an example (720) of a method for performing a screen translation, and an example (730) of a method for performing a changed screen translation. For convenience of explanation, FIG. 7 illustrates examples (710, 720, 730) of generating a translation model, performing a screen translation when a function is activated, and performing a changed screen translation when the screen is changed while the function is activated; however, the embodiments of the present disclosure are not limited thereto. For example, only some of the examples (710, 720, 730) of FIG. 7 may be performed. Furthermore, FIG. 7 illustrates a method for utilizing a translation model (704); however, the embodiments of the present disclosure are not limited thereto. For example, when using a server that provides translation services instead of a translation model (704), communication with the server may be performed instead of the translation model (704), and examples (710) of operations for generating a translation model (704) may be omitted.

[0158] Referring to example (710), in operation (711), the electronic device (101) may transmit a translation model creation event from the first software application (701) to the framework (702). For example, the translation model creation event may include an event for activating the function of the second software application. However, operation (711) may be omitted. For example, the electronic device (101) may create a translation model (704) upon initial execution (or download) of the second software application, regardless of the translation model creation event.

[0159] In one embodiment, in operation (712), the electronic device (101) may request the second software application (703) from the framework (702) to create a translation model. For example, the translation framework of the framework (702) may request the creation of a translation model in response to the translation model creation event.

[0160] According to one embodiment, in operation (713), the electronic device (101) may generate a translation model (704) through the second software application (703). For example, the electronic device (101) may generate the translation model (704) in response to a request for generation of the translation model.

[0161] Although not shown in example (710), the translation model (704) may send a response to the second software application (703) to inform that the translation model (704) has been generated in the background. In addition, the second software application (703) may forward the response to the framework (702).

[0162] Referring to example (720), in operation (721), the electronic device (101) may transmit layer information from the first software application (701) to the framework (702). At this time, the layer information may be layer information for a portion of a screen displayed in the display area of ​​the display (160) of the electronic device (101), among the screens that may be provided by the first software application (701). Although not illustrated in FIG. 7, the layer information may be transmitted from the first software application (701) to the framework (702) through the second software application (703).

[0163] According to one embodiment, in operation (722), the electronic device (101) may transmit text information from the framework (702) to the second software application (703). For example, the text information may include second information of the hierarchical information. Or, for example, the text information may include a string representing text within the second information. According to one embodiment, in operation (723), the electronic device (101) may transmit the text information from the second software application (703) to the translation model (704).

[0164] According to one embodiment, the translation model (704) can perform a translation of the text of the text information. Accordingly, the translation model (704) can generate a translated text.

[0165] According to one embodiment, in operation (724), the electronic device (101) may transmit translated text information from the translation model (704) to the second software application (703). For example, the translated text information may be referred to as third information. According to one embodiment, in operation (725), the electronic device (101) may transmit the translated text information from the second software application (703) to the framework (702).

[0166] According to one embodiment, the framework (702) can reconstruct (or change, update) the hierarchical information using the translated text information. For example, the electronic device (101) can generate the hierarchical information further including the translated text information on the framework (702). The hierarchical information further including the translated text information can be referred to as changed hierarchical information.

[0167] According to one embodiment, in operation (726), the electronic device (101) may transmit the changed layer information from the framework (702) to the first software application (701). For example, the electronic device (101) may use the first software application (701) to change (or reconfigure, update) the screen based on the changed layer information. Thereafter, the electronic device (101) may display the changed screen while the first software application (701) is running in the foreground.

[0168] Referring to example (730), in operation (731), the electronic device (101) may transmit new layer information from the first software application (701) to the framework (702). At this time, the new layer information may be layer information for a portion displayed in the display area of ​​the display (160) of the electronic device (101), among the updated screens of the first software application (701) based on the input of the screen of the layer information transmitted in operation (721). For example, the content included in the updated screen of operation (731) may overlap at least partially with the content included in the screen of operation (721). Alternatively, for example, the content included in the updated screen of operation (731) may be different from the content included in the screen of operation (721). Hereinafter, the new layer information may be referred to as the layer information. Although not shown in FIG. 7, the new layer information can be transmitted from the first software application (701) to the framework (702) via the second software application (703).

[0169] According to one embodiment, in operation (732), the electronic device (101) may transmit text information from the framework (702) to the second software application (703). For example, the text information may include second information of the hierarchical information. Or, for example, the text information may include a string representing text within the second information. According to one embodiment, in operation (733), the electronic device (101) may transmit the text information from the second software application (703) to the translation model (704).

[0170] According to one embodiment, the translation model (704) can perform a translation of the text of the text information. Accordingly, the translation model (704) can generate a translated text.

[0171] According to one embodiment, in operation (734), the electronic device (101) may transmit translated text information from the translation model (704) to the second software application (703). For example, the translated text information may be referred to as third information. According to one embodiment, in operation (735), the electronic device (101) may transmit the translated text information from the second software application (703) to the framework (702).

[0172] According to one embodiment, the framework (702) can reconstruct (or change, update) the layer information using the translated text information. The layer information to be reconstructed may be the new layer information transmitted in operation (731). For example, the electronic device (101) can generate the layer information further including the translated text information on the framework (702). The layer information further including the translated text information may be referred to as changed layer information.

[0173] According to one embodiment, in operation (736), the electronic device (101) may transmit the changed layer information from the framework (702) to the first software application (701). For example, the electronic device (101) may use the first software application (701) to change (or reconfigure, update) the screen based on the changed layer information. Thereafter, the electronic device (101) may display the changed screen while the first software application (701) is running in the foreground.

[0174] FIG. 8 illustrates an example of an operational flow for a method of providing a translation function depending on whether a screen is changed, according to one embodiment.

[0175] At least some of the methods of FIG. 8 may be performed by the electronic device (101) of FIG. 3A. For example, at least some of the methods may be controlled by the processor (310) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0176] The method of FIG. 8 may include examples of operations performed when a change occurs on the screen based on a user input after or while the operation (455) of FIG. 4 is performed. In this case, it is assumed that the function of the second software application is activated.

[0177] In one embodiment, in operation (800), the electronic device (101) may recognize a change in the screen. For example, the electronic device (101) may change the screen based on a user input while displaying the screen, which further includes translated text. For example, the user may input an executable object within the screen or a scroll input. However, the embodiments of the present disclosure are not limited thereto.

[0178] According to one embodiment, in operation (805), the electronic device (101) may obtain layer information of the changed screen. For example, the electronic device (101) may obtain layer information of the changed screen based on the input from the first software application. For example, the layer information may be layer information for the changed screen displayed in the display area of ​​the display (160) of the electronic device (101) among the screens that may be provided by the first software application.

[0179] According to one embodiment, the electronic device (101) can recognize, from the hierarchical information, second information for a fourth area including text to be translated within the hierarchical information and first information for a third area including the fourth area.

[0180] According to one embodiment, in operation (810), the electronic device (101) may determine whether the content included in the fourth area includes text written in a language different from the user language. For example, the electronic device (101) may determine, based on the second information, whether the text of the content included in the fourth area includes text written in a language different from the user language. For example, the user language may be a language set by the user and may represent a target language. For example, the electronic device (101) may determine whether, among the texts included in the changed screen when the function is activated, there is text written in a language different from the user language.

[0181] In operation (810), the electronic device (101) may perform operation (815) if the text in the changed screen includes text written in a language different from the user language. Conversely, in operation (810), the electronic device (101) may perform operation (820) if the text in the screen does not include text written in a language different from the user language.

[0182] According to one embodiment, in operation (815), the electronic device (101) may perform a translation and generate modified hierarchical information. For example, the electronic device (101) may perform a translation on the text written in a language different from the user language. For example, the electronic device (101) may perform the translation using information about the text (e.g., second information). For example, the electronic device (101) may generate the modified hierarchical information based on information about the translated text (e.g., third information). For example, the modified hierarchical information may include information about the text and a view area including the translated text (e.g., first information).

[0183] According to one embodiment, in operation (820), the electronic device (101) may display a view area.

[0184] For example, the electronic device (101) may display the changed screen including the view area if the changed screen includes text written in a language different from the user language. In this case, the view area may include content having the text and the translated text.

[0185] In contrast, if the electronic device (101) does not include text written in a language different from the user language among the texts within the screen, the electronic device (101) may display the changed screen including the view area. In this case, the view area may include content having the text. In other words, since translation is not required for the texts within the changed screen, the changed screen including the untranslated view area may be displayed.

[0186] FIGS. 9A to 9D illustrate examples of a method for translating text included in a screen and displaying a screen including the translated text, according to one embodiment.

[0187] FIG. 9a illustrates an example of a method for displaying a screen including view areas (910b, 920b) containing text written in a first language and other text written in a second language within a screen (905) of a first software application providing a messenger service.

[0188] Referring to FIG. 9A, in response to at least a portion of an input for an icon (not shown) for the first software application, the electronic device (101) may display a screen (905a). In the example of FIG. 9A, the screen (905a) is a conversation screen between a user of the electronic device (101) (e.g., Kim of FIG. 9B) and a specific user (e.g., Linsey Smith), and the screen (905a) is illustrated as having a structure of view objects substantially identical to the screen object, but the embodiments of the present disclosure are not limited thereto. This may vary depending on a method of configuring view objects according to the first software application.

[0189] According to one embodiment, the electronic device (101) may obtain layer information of the screen (905a) from the first software application when the live translation function (or function) of the second software application is activated. For example, the layer information of the screen (905a) may include first information (e.g., first information (531) of FIG. 5B ) for each of the plurality of view areas (910a, 915a, 920a). For example, the view area (910a) may include content having text (e.g., "Good morning") written in a first language (e.g., English). Additionally, for example, the view area (915a) may include content having text (e.g., "Hello") written in a second language (e.g., Korean). Additionally, for example, view area (920a) may include text written in a first language (e.g., English), such as "I checked the attached apk and it doesn't seem to be working as expected." For example, view area (910a) and view area (920a) may display a message transmitted from the specific user (e.g., Linsey Smith) to the user (e.g., Kim) of the electronic device (101). For example, view area (915a) may display a message transmitted from the user (e.g., Kim) of the electronic device (101) to the specific user (e.g., Linsey Smith).

[0190] For example, first information for a view area (910a) (e.g., first information (531) in FIG. 5B) may include second information (e.g., second information (540) in FIG. 5B) regarding text written in a first language (e.g., English) (e.g., "Good morning"). For example, first information for a view area (915a) may include second information regarding text written in a second language (e.g., Korean) (e.g., "Hello"). For example, first information for a view area (920a) may include second information regarding text written in a first language (e.g., English) (e.g., "I checked the attached apk and it doesn't seem to be working as expected.").

[0191] According to one embodiment, the electronic device (101) may translate text written in the first language and generate another translated text when the function of the second software application is activated. For example, the electronic device (101) may generate text translated into a second language (e.g., Korean) (e.g., "Hello") from text written in the first language (e.g., English) (e.g., "Good morning"). Additionally, the electronic device (101) may generate text translated into a second language (e.g., Korean) (e.g., "I checked the attached apk and it doesn't seem to be working as expected") from text written in the first language (e.g., English) (e.g., "I checked the attached apk and it doesn't seem to be working as expected"). For example, the electronic device (101) may omit (or refrain from) translation of text (e.g., “Hello”) in the view area (915a) even if the function of the second software application is activated. For example, the electronic device (101) may exclude the text in the view area (915a) from translation based on recognizing that it is written in a second language (e.g., Korean).

[0192] According to one embodiment, the electronic device (101) can reconstruct (or change, update) the hierarchical information of the screen (905a) using the translated text on the framework (e.g., the framework (380) of FIG. 3b). For example, the electronic device (101) can reconstruct the hierarchical information of the screen (905a). The reconstructed hierarchical information of the screen (905a) can further include third information about the second information and the translated text.

[0193] According to one embodiment, the electronic device (101) may display a changed screen (905b) through the first software application using the reconstructed layer information. For example, the changed screen (905b) may include view areas (910b, 920b) that have been changed using the reconstructed layer information. For example, the view area (910b) may include text written in a first language (e.g., English) (e.g., "Good morning") and text written in a second language (e.g., Korean) (e.g., "Hello"). Additionally, the view area (920b) may include text written in the first language (e.g., English) (e.g., "I checked the attached apk and it doesn't seem to be working as expected.") and text written in a second language (e.g., Korean) (e.g., "I checked the attached apk and it doesn't seem to be working as expected."). Additionally, for example, the changed screen (905b) may include a view area (915b). For example, the view area (915b) of the changed screen (905b) may be substantially the same as (or correspond to) the view area (915a) of the screen (905a).

[0194] According to one embodiment, the size of the view areas (910b, 920b) may be different from the size of the view areas (910a, 920a). For example, the size of the view area (910b) may be larger than the size of the view area (910a). For example, the size of the view area (920b) may be larger than the size of the view area (920a). In this case, the width of the view areas (910b, 920b) may be substantially the same as the width of the view areas (910a, 920a). In other words, the electronic device (101) may change the size of the view areas to include more content while maintaining the arrangement (or layout) of the view areas.

[0195] According to one embodiment, texts included in view areas (910b, 920b) may have different visual effects. For example, text (e.g., "Good morning") written in a first language (e.g., English) in view area (910b) may have a first visual effect. For example, text (e.g., "Hello") written in a second language (e.g., Korean) in view area (910b) may have a second visual effect. For example, when the first visual effect is a first color, the second visual effect may be a second color with lower saturation than the first color.

[0196] In the above example, an example is shown in which the visual effect of the text (e.g., "Good morning") is changed from the second visual effect to the first visual effect as the translated text (e.g., "Hello") of the view area (910b) is displayed, but the embodiments of the present disclosure are not limited thereto. For example, the visual effect of the text (e.g., "Good morning") of the view area (910a) and the view area (910b) may have the first visual effect, and a third visual effect, which is a third color with a higher saturation than the first visual effect, may be applied to the translated text (e.g., "Hello") of the view area (910b).

[0197] FIG. 9b illustrates an example of a method for displaying a screen including a view area (927) containing text written in a first language and other text written in a second language within a screen (925) of a first software application providing the messenger service.

[0198] Referring to FIG. 9B, in response to at least a portion of an input for an icon (not shown) for the first software application, the electronic device (921) may display a screen (925). In the example of FIG. 9B, the screen (925) is a conversation screen between a specific user of the electronic device (921) (e.g., Linsey Smith of FIG. 9A) and a user of the electronic device (101) (e.g., Kim), and the screen (925) is illustrated as having a structure of view objects substantially identical to the screen object, but the embodiments of the present disclosure are not limited thereto. This may vary depending on a method of configuring view objects according to the first software application.

[0199] According to one embodiment, the electronic device (921) may display a screen (925) including a view area (927) containing translated text when the live translation function (or feature) of the second software application is activated. For example, the screen (925) may include a view area (926) containing text written in a first language (e.g., English) (e.g., "Good morning"), and a view area (928) containing text written in the first language (e.g., English) (e.g., "I checked the attached apk and it doesn't seem to be working as expected."). Additionally, for example, the screen (925) may include a view area (927) containing text written in the first language (e.g., English) (e.g., "Good morning") and text translated into a second language (e.g., Korean) (e.g., "Hello"). For example, view area (926) and view area (928) may display a message transmitted from a specific user of electronic device (921) (e.g., Linsey Smith) to said user of electronic device (101) (e.g., Kim). For example, view area (927) may display a message transmitted from said user of electronic device (101) (e.g., Kim) to said specific user of electronic device (921) (e.g., Linsey Smith).

[0200] Referring to the above, when the function of the second software application is activated, the electronic device (921) can recognize text (e.g., “Hello”) written in the source language, Korean, in the view area (927), generate text translated into the target language, English, (e.g., “Good morning”), and then display the view area (927). For example, the screen (925) can include a reconstructed view area (927). In other words, the screen (925) can display a reconstructed screen.

[0201] FIG. 9c illustrates an example of a method for displaying a screen that includes view areas containing translated text from texts written in multiple languages.

[0202] Referring to FIG. 9C, the electronic device (101) can display a screen (930) of a first software application, which is an SNS software application. FIG. 9C illustrates a case where the first software application is an SNS software application, but the embodiments of the present disclosure are not limited thereto.

[0203] In one embodiment, the screen (930) may include a plurality of view areas (940, 950). For example, the view areas (940, 950) may include text before translation and text after translation, depending on the activation of the function of the second software application. For example, the view area (940) may include text written in a first language (e.g., English) (“It's a really great travel destination!”) ​​(941) and text written in a second language (e.g., Korean) (“It's a really great travel destination!”) ​​(942). For example, the text (941) may be text before translation. For example, the text (942) may be translated text. Additionally, for example, the view area (950) may include text written in a first language (e.g., Japanese) (“It’s a beautiful place”) (951) and text written in a second language (e.g., Korean) (“It’s a beautiful place”) (952). For example, the text (951) may be text before translation. For example, the text (952) may be translated text.

[0204] According to one embodiment, texts included in view areas (940, 950) may have different visual effects. For example, text (941) in view area (940) may have a first visual effect. For example, text (942) in view area (940) may have a second visual effect. For example, when the first visual effect is a first color, the second visual effect may be a second color with lower saturation than the first color.

[0205] As described above, the electronic device (101) can translate texts written in different source languages ​​into texts written in the same target language by utilizing the function of the second software application. The electronic device (101) can provide translated texts by utilizing the function of the second software application without executing a separate translation function provided by the first software application.

[0206] Figure 9d illustrates examples of screens that include view areas that contain text translated into a target language, determined based on the relevance between the view areas.

[0207] Referring to FIG. 9d, the electronic device (101) can display a screen (960) of a first software application, which is an SNS software application. FIG. 9d illustrates a case where the first software application is an SNS software application, but the embodiments of the present disclosure are not limited thereto.

[0208] In one embodiment, the screen (960) may include a plurality of view areas (940, 970). For example, the view areas (940, 970) may include text before translation and text after translation, as the live translation function (or feature) of the second software application is activated. For example, the view area (940) may include text written in a first language (e.g., English) (“It's a really great travel destination!”) ​​(941) and text written in a second language (e.g., Korean) (“It's a really great travel destination!”) ​​(942). For example, the text (941) may be text before translation. For example, the text (942) may be translated text. Additionally, for example, the view area (970) may include text written in a first language (e.g., Korean) (“Let’s go together next time”) (971) and text written in a second language (e.g., English) (“Let’s go together next time”) (972). For example, the text (971) may be text before translation. For example, the text (972) may be translated text.

[0209] According to one embodiment, the view area (970) may be a dependent view area created for the view area (940). For example, the identification information of the view area (940) (e.g., the identification information (551) of FIG. 5B) may be linked to the identification information of the view area (970). Linking between identification information may include having the same identification information or having the identification information set according to a predefined rule. For example, if the view area (940) is an area where comments are displayed on a social network service (SNS), the view area (970) may be an area where replies to the comments are displayed on the social network service (SNS).

[0210] According to one embodiment, the electronic device (101) may recognize a relationship (e.g., a link) between view areas (940, 970) while performing translation using the function of the second software application. For example, the electronic device (101) may determine a target language of text (972) to be translated from text (971) of view area (970) based on the relationship. For example, the target language of text (972) may be determined as a first language (e.g., English) of view area (940) linked to view area (970).

[0211] Additionally, according to one embodiment, the screen (980) may include a plurality of view areas (950, 990). For example, the view areas (950, 990) may include text before translation and text after translation, depending on the activation of the function of the second software application. For example, the view area (950) may include text written in a first language (e.g., Japanese) (“It’s a beautiful place”) (951) and text written in a second language (e.g., Korean) (“It’s a beautiful place”) (952). For example, the text (951) may be text before translation. For example, the text (952) may be translated text. Additionally, for example, the view area (990) may include text written in a first language (e.g., Korean) (“It’s a world-famous place!”) (991) and text written in a second language (e.g., Japanese) (“It’s a world-famous place!”) (992). For example, the text (991) may be text before translation. For example, the text (992) may be translated text.

[0212] According to one embodiment, the view area (990) may be a dependent view area created for the view area (950). For example, the identification information of the view area (950) (e.g., the identification information (551) of FIG. 5B) may be linked to the identification information of the view area (990). Linking between identification information may include having the same identification information or having the identification information set according to a predefined rule. For example, if the view area (950) is an area where comments are displayed on a social network service (SNS), the view area (990) may be an area where replies to the comments are displayed on the social network service (SNS).

[0213] According to one embodiment, the electronic device (101) may recognize a relationship (e.g., a link) between view areas (950, 990) while performing translation using the function of the second software application. For example, the electronic device (101) may determine a target language of text (992) to be translated from text (991) of view area (990) based on the relationship. For example, the target language of text (992) may be determined as a first language (e.g., Japanese) of view area (950) linked to view area (990).

[0214] Figures 10A through 10C illustrate examples of a method for activating a function of a second software application based on an input to an icon of a keyboard application, according to one embodiment. Figures 10D and 10E illustrate examples of a method for changing settings of a second software application by changing settings of a keyboard application, according to one embodiment.

[0215] FIGS. 10A to 10C illustrate examples (1001, 1002, 1003, 1004) of a method for activating the function of a second software application based on an electronic device (101) obtaining input for an icon within an executable object of the keyboard application while a first software application providing a messenger service is running in the foreground. For example, the electronic device (101) of FIGS. 10A to 10C may be an example of the electronic device (101) of FIG. 3A.

[0216] Referring to FIGS. 10A to 10C , in an example (1001), the electronic device (101) may display a screen (1005) while the first software application is running on the foreground. For example, the screen (1005) may include a screen representing a chat between a user of the electronic device (101) (e.g., Kim of FIG. 10C ) and another user (e.g., Linsey Smith) via the first software application. For example, the screen (1005) may include a plurality of view areas (1006, 1007). For example, the view area (1006) may include text written in a first language (e.g., English) (e.g., "It seems that the new function added today is not working properly. Can you quickly check this part?"). For example, the view area (1007) may include text written in a first language (e.g., English) (“The same problem occurs with the apk you just sent. Please check again.”).

[0217] In example (1001), the electronic device (101) may display an executable object (1010) of the keyboard application. For example, the executable object (1010) may include an object in the shape of a keyboard provided through the keyboard application. For example, the executable object (1010) may at least partially overlap the screen (1005).

[0218] In FIGS. 10A to 10C, the electronic device (101) is illustrated as a bar-type smartphone, but the embodiments of the present disclosure are not limited thereto. For example, the electronic device (101) may include an electronic device including a flexible display (e.g., a foldable electronic device or a rollable electronic device). Accordingly, the executable object (1010) may be displayed across the screen of the flexible display. It is assumed that the flexible display includes a first display area, a second display area, and a third display area (or bending area) configured to be bendable between the first display area and the second display area. For example, the executable object (1010) in the screen (1005) may be displayed in the second display area. For example, the remaining portion of the screen (1005) that is different from the executable object (1010) (e.g., the portion containing the conversation content) may be displayed from the third display area to the first display area.

[0219] In one embodiment, the executable object (1010) may include an icon (1015) for a keyboard translation function and an input area (1017) for displaying translated text being input. For example, the icon (1015) may represent an icon for activating a function for translating text being written in the keyboard application. In example (1001), the icon (1015) may be in a deactivated state. For example, the input area (1017) may represent an area (or visual object) for displaying translated text based on at least a portion of an input to the executable object (1010). For example, the input area (1017) may include a visual object in the shape of a bar. However, the present disclosure is not limited thereto.

[0220] In example (1002), according to one embodiment, the electronic device (101) may activate an icon (1015). For example, the electronic device (101) may display an activated icon (1015a) based on an input for the icon (1015). The activated icon (1015a) may have a different color, shade, font, or background image than the icon (1015). For example, the activated icon (1015a) may be displayed in a different color than the icon (1015).

[0221] For example, the electronic device (101) may display an executable object (1010) further including an input area (1018) in which text to be translated (or text being input) is displayed, upon displaying an icon (1015a). For example, the input area (1018) may represent an area (or visual object) for displaying text being written based on at least a portion of an input to the executable object (1010). For example, the electronic device (101) may display an icon (1019) for changing the language to be translated (or target language) together with the input area (1018), upon displaying the icon (1015a). For example, an image (or shape) within the icon (1019) may be changed depending on the target language. For example, the image of the icon (1019) may include alphabets (e.g., “EN”), since the target language is English. However, the embodiments of the present disclosure are not limited thereto. For example, if the target language is Chinese, the image of the icon (1019) may include Chinese characters.

[0222] According to one embodiment, the electronic device (101) may activate the function of the second software application based on the input to the icon (1015). For example, the electronic device (101) may display changed view areas (1026, 1027) as the function of the second software application is activated. For example, the view area (1026) may include text written in a first language (e.g., English) (e.g., "It seems that the new function added today is not working properly. Can you quickly check this part?") and text written in a second language (e.g., Korean) (e.g., "It seems that the new function added today is not working properly. Can you quickly check this part?"). Additionally, for example, the view area (1027) may include text written in a first language (e.g., English) (“The same problem occurs with the apk you just sent. Please check again.”) and text written in a second language (e.g., Korean) (“The same problem occurs with the apk you just sent. Please check again.”). In this case, the view area (1026) may be larger in size than the view area (1006). Additionally, the view area (1027) may be larger in size than the view area (1007).

[0223] In example (1003), according to one embodiment, the electronic device (101) may display text being written by a user based on at least a portion of the input to the executable object (1010). For example, the electronic device (101) may display an input area (1018) corresponding to at least a portion of the input and including text written in a second language (e.g., Korean) (e.g., "Hello. Good morning.") (1031). For example, the electronic device (101) may display an input area (1017) corresponding to at least a portion of the input and including text written in a first language (e.g., English) (e.g., "Hello, good morning.") (1032) that has been translated from text written in a second language (e.g., Korean) (e.g., "Hello. Good morning.") (1031) based on the keyboard translation function. According to one embodiment, the executable object (1010) may include an icon (1035) for transmitting text. For example, the icon (1035) may be displayed when an activated icon (1015a) is displayed.

[0224] Referring to example (1003), the electronic device (101) can display text (1031) written in the second language together with text (1032) written in the first language based on the keyboard translation function.

[0225] In example (1004), according to one embodiment, the electronic device (101) may transmit the text being written to the other user of the conversation based on the input to the icon (1035). For example, the electronic device (101) may transmit the text (1032) written in the first language (e.g., English) based on the input to the icon (1035). In this case, the text (1031) written in the second language (e.g., Korean) may not be transmitted to the other user.

[0226] According to one embodiment, the electronic device (101) may display a view area (1040) including text (1032) written in the first language and text (1031) written in the second language (e.g., Korean) while the function of the second software application is activated. Referring to example (1004), the electronic device (101) may display a view area (1027) and a view area (1040). For example, the view area (1040) may include text written in the second language (e.g., Korean) (e.g., "Hello. Good morning.") and text written in the first language (e.g., English) (e.g., "Hello, good morning."). At this time, in order to indicate that the text (e.g., "Hello, good morning") written in the second language (e.g., Korean) is not transmitted to the other user, the electronic device (101) may display the color of the text (e.g., "Hello, good morning") written in the second language (e.g., Korean) differently from the color of the text (e.g., "Hello, good morning") written in the first language (e.g., English). For example, the first visual effect (e.g., color) of the text (e.g., "Hello, good morning") written in the first language (e.g., English) may have the first color. For example, the second visual effect (e.g., color) of the text (e.g., "Hello, good morning") written in the second language (e.g., Korean) may have the second color. For example, the saturation of the first color may be higher than the saturation of the second color.

[0227] FIG. 10C illustrates an example of a screen displayed on an electronic device (1041) when a user (e.g., Kim) of the electronic device (101) of FIGS. 10A and 10B transmits text (e.g., “Hello. Good morning.”) written in the second language (e.g., Korean) to a user (e.g., Linsey Smith) of the electronic device (1041).

[0228] Referring to FIG. 10C, the electronic device (1041) may display a screen (1045) in response to at least a portion of an input for an icon (not shown) for the first software application. For example, the screen (1045) may include a plurality of view areas (1046, 1047, 1048). For example, each of the plurality of view areas (1046, 1047, 1048) may include text written in a first language (e.g., English). For example, the view area (1048) may include text written in the first language (e.g., English) (e.g., "hello good morning"). Referring to FIGS. 10A and 10B , a user (e.g., Kim) of an electronic device (101) may input text (e.g., “Hello. Good morning”) written in a second language (e.g., Korean) to the electronic device (101), and then transmit the translated text (e.g., “hello good morning”) translated into a first language (e.g., English) to the user (e.g., Linsey Smith) of the electronic device (1041) through a translation function of a keyboard application. Accordingly, the electronic device (1041) may display a view area (1048) including the text (e.g., “hello good morning”) written in the first language (e.g., English).

[0229] FIGS. 10d and 10e illustrate examples (1051, 1052, 1053, 1054) of changing languages ​​(e.g., source language and target language) of the functions of the second software application based on the electronic device (101) obtaining input for an icon within an executable object of the keyboard application while the first software application providing the messenger service is running in the foreground. For example, the electronic device (101) of FIGS. 10d and 10e may be an example of the electronic device (101) of FIG. 3a.

[0230] Referring to example (1051), according to one embodiment, the electronic device (101) may display an executable object (1010). For example, the electronic device (101) may display an executable object (1010) of the keyboard application. For example, the executable object (1010) may include an object in the shape of a keyboard provided through the keyboard application. For example, the executable object (1010) may at least partially overlap the screen of the first software application.

[0231] According to one embodiment, the executable object (1010) may include an icon (1015a) for a keyboard translation function, an input area (1017) for displaying translated text, and an icon (1019) for changing languages ​​(e.g., source language and target language) of the keyboard translation function. For example, the icon (1015a) may represent an icon for activating a function for translating text being written in the keyboard application. In example (1051), the icon (1015a) may be in an activated state. For example, the input area (1017) may represent an area (or visual object) for displaying translated text based on at least a portion of an input to the executable object (1010). For example, the icon (1019) may represent an icon for changing the languages ​​of the keyboard translation function of the keyboard application.

[0232] In example (1052), according to one embodiment, the electronic device (101) may display icons (1061, 1062) for changing the languages ​​based on an input to the icon (1019). For example, the electronic device (101) may display an activated icon (1019a) based on obtaining the input to the icon (1019). For example, the electronic device (101) may display icons (1061, 1062) in response to displaying the icon (1019a). For example, the icon (1061) may be an icon for setting a source language. For example, the icon (1062) may be an icon for setting a target language.

[0233] Referring to examples (1052) and (1053), according to one embodiment, the electronic device (101) may display a visual object (1070) for changing the source language in response to an input to the icon (1061). For example, the visual object (1070) may include icons (1071, 1072, 1073) corresponding to multiple languages. For example, the icon (1071) may correspond to Auto. For example, the Auto may indicate the language of the text to be translated recognized by the electronic device (101). For example, the icon (1072) may correspond to English. For example, the icon (1073) may correspond to Korean. Within the visual object (1070), the icon (1071) corresponding to Auto may be fixed to the topmost portion within the visual object (1070). For example, when a visual object (1070) is displayed, an icon (1071) may be displayed at the top. For example, icons (1072) and (1073) may be arranged within the visual object (1070) in the order of a language set in a keyboard application within the electronic device (101) or a recently used language. For example, the electronic device (101) may change the source language to English based on obtaining input for the icon (1072).

[0234] Referring to examples (1052) and (1054), according to one embodiment, the electronic device (101) may display a visual object (1080) for changing the target language in response to an input for the icon (1062). For example, the visual object (1080) may include icons corresponding to multiple languages. For example, the icon (1081) may correspond to Korean. Unlike the visual object (1070), the visual object (1080) may not include an icon corresponding to the automatic language. For example, the electronic device (101) may change the target language to Korean based on obtaining an input for the icon (1081).

[0235] Although not illustrated in FIG. 10e, according to one embodiment, the visual object (1070) (or visual object (1080)) may further include an icon for adding a source language (or a target language). For example, the icon for adding may include text (e.g., “+ add languages”). For example, the electronic device (101) may, in response to receiving an input for the icon for adding, display languages ​​that can be downloaded. For example, the electronic device (101) may download one of the displayed languages. For example, downloading a language may be understood as substantially the same as storing information about the language (e.g., a language pack). For example, the electronic device (101) may change the downloaded language to the source language.

[0236] According to one embodiment, the electronic device (101) can change the languages ​​of the function of the second software application by changing the languages ​​of the keyboard translation function. For example, the electronic device (101) can change the languages ​​of the keyboard translation function based on an input to the icon (1061) (or icon (1062)). In this case, the languages ​​(e.g., source language and target language) of the function of the second software application can correspond to the languages ​​of the keyboard translation function. However, the embodiments of the present disclosure are not limited thereto. For example, the languages ​​of the function of the second software application can be set in a menu within the system settings, as illustrated in FIG. 11 below.

[0237] FIG. 11 illustrates an example of a method for activating a function of a second software application within a system setting, according to one embodiment.

[0238] FIG. 11 illustrates an example (1100) of a method for activating the function of the second software application within the system settings for the overall settings of the electronic device (101). For example, the electronic device (101) of FIG. 11 may be an example of the electronic device (101) of FIG. 3A.

[0239] Referring to example (1100), the electronic device (101) may display a screen (1110) for setting the system. For example, the electronic device (101) may display the screen (1110) based on at least a portion of a user's input. For example, the screen (1110) may include a plurality of menus. For example, the plurality of menus may include a menu for connection with an external electronic device, a menu for sound output from the electronic device (101), a menu for display of the electronic device (101), and a menu (1115) for a keyboard application. The screen (1110) of FIG. 11 is merely an example for convenience of explanation, and embodiments of the present disclosure are not limited thereto.

[0240] According to one embodiment, the electronic device (101) may display a screen (1120) including a plurality of submenus for configuring the keyboard application based on an input to the menu (1115). For example, the plurality of submenus of the screen (1120) may include a submenu indicating the types of available languages, submenus for smart typing, and a submenu (1125) for live translation. For example, the live translation may indicate the function of the second software application. The screen (1120) of FIG. 11 is merely an example for convenience of explanation, and embodiments of the present disclosure are not limited thereto.

[0241] According to one embodiment, the electronic device (101) may display a screen (1130) including icons for setting the function of the second software application based on an input to the submenu (1125). For example, the screen (1130) may include an icon (1131) for activating the function of the second software application, and icons (1132, 1133) for setting languages ​​(e.g., source language and target language) of the function of the second software application. For example, the icon (1131) may include a toggle. For example, the icon (1132) may be used for setting the target language of the function of the second software application. For example, the icon (1133) may be used for setting the source language of the function of the second software application. For example, based on the input for the icon (1132) or the icon (1133), the visual object (1070) and the visual object (1080) of FIG. 10e can be displayed.

[0242] FIG. 12A illustrates an example of a method for providing summary information of content, according to one embodiment.

[0243] FIG. 12A illustrates examples (1201, 1202) of how an electronic device (101) provides the summary information through the function of the second software application. For example, the electronic device (101) of FIG. 12A may be an example of the electronic device (101) of FIG. 3A.

[0244] Referring to example (1201), the electronic device (101) may display a screen (1205) of a first software application providing SNS. For example, the screen (1205) may display a view area (1210). For example, the view area (1210) may include text (e.g., "Staying positive doesn't mean you have to be happy all the time. It means that even on the hard days you know better ones are coming.") (1215) written in a first language (e.g., English).

[0245] Although not illustrated in FIG. 12A, according to one embodiment, the electronic device (101) may activate the function of the second software application. For example, the electronic device (101) may activate the function based on an event (e.g., an automatic event or a manual event) for activating the function of the second software application.

[0246] Referring to example (1202), the electronic device (101) can display a reconfigured screen (1205a). For example, the electronic device (101) can display the reconfigured screen (1205a) through the activated function of the second software application. For example, the screen (1205a) can include a reconfigured view area (1220). For example, the view area (1220) can include text written in a first language (e.g., English) (e.g., "Staying positive doesn't mean you have to be happy all the time. It means that even on the hard days you know better ones are coming.") (1223) and text written in a second language (e.g., Korean) (e.g., "Being positive doesn't mean you have to be happy all the time. Because being positive means thinking that better things are coming even in difficult situations.") (1222). For example, text (1222) may be text translated from text (1215) (or text (1223)) through the function of the second software application. For example, text (1222) may be generated from text (1215) (or text (1223)) based on a translation model.

[0247] In one embodiment, the view area (1220) may further include summary information (1221). For example, the view area (1220) may further include summary information (1221) generated through the activated function of the second software application. For example, the summary information (1221) may be generated based on another model while generating the text (1222) from the text (1215) (or the text (1223)) based on the translation model. In one embodiment, the electronic device (101) may determine whether the length of the text (1215) (or the text (1223)) exceeds a reference length. For example, if the length exceeds the reference length, the electronic device (101) may generate a summary of the text (1215) (or the text (1223)) using the other model. For example, the summary may be referred to as summary information.

[0248] For example, the other model may include a large language model (LLM). For example, the other model may be stored within the second software application or installed within the electronic device (101). For example, the other model may be referred to as an analytical model. While the above example describes a case where the translation model and the other model are different, the embodiments of the present disclosure are not limited thereto. For example, the other model may be identical to the translation model.

[0249] For example, the summary information (1221) may include text (e.g., "positive meaning") written in the second language (e.g., Korean). For example, the other model may use the text (1215) (or text (1223)) to generate a text that summarizes the text (1215) (or text (1223)). For example, the summary information (1221) may include the summarized text.

[0250] FIG. 12b illustrates an example of a method for providing visual effects for content, according to one embodiment.

[0251] FIG. 12B illustrates examples (1231, 1232) of how an electronic device (101) provides the summary information through the function of the second software application. For example, the electronic device (101) of FIG. 12B may be an example of the electronic device (101) of FIG. 3A.

[0252] Referring to example (1231), the electronic device (101) may display a screen (1235) of a first software application providing SNS. For example, the screen (1235) may display view areas (1241, 1242). For example, the view area (1241) may include text written in a first language (e.g., English) (e.g., "This decision was a really stupid decision.") (1241a). For example, the view area (1242) may include text written in a first language (e.g., English) (e.g., "We can overcome this with good solutions in the future.") (1242a).

[0253] Although not illustrated in FIG. 12b, according to one embodiment, the electronic device (101) may activate the function of the second software application. For example, the electronic device (101) may activate the function based on an event (e.g., an automatic event or a manual event) for activating the function of the second software application.

[0254] Referring to example (1232), the electronic device (101) can display a reconfigured screen (1235a). For example, the electronic device (101) can display the reconfigured screen (1235a) through the activated function of the second software application. For example, the screen (1235a) can include reconfigured view areas (1251, 1252). For example, the view area (1251) can include text written in a first language (e.g., English) (e.g., "This decision was a really stupid decision.") (1251a) and text written in a second language (e.g., Korean) (e.g., "This decision was a really stupid decision.") (1251b). For example, the view area (1252) may include text written in a first language (e.g., English) (e.g., "We can overcome this problem with good solutions in the future.") (1252a) and text written in a second language (e.g., Korean) (e.g., "We can overcome this problem with good solutions in the future.") (1252b).

[0255] According to one embodiment, the electronic device (101) may provide notification information for the reconstructed view areas (1251, 1252). For example, the electronic device (101) may apply first notification information (1251c) to the view area (1251). For example, the electronic device (101) may apply second notification information (1252c) to the view area (1252). For example, the first notification information (1251c) may include a display of a first visual object (e.g., an image indicating good) within the view area (1251). For example, the second notification information (1252c) may include a display of a second visual object (e.g., an image indicating bad) within the view area (1252). However, the embodiments of the present disclosure are not limited thereto. For example, the first notification information (1251c) may include displaying the background color of the view area (1251) as a first color, and the second notification information (1252c) may include displaying the background color of the view area (1252) as a second color different from the first color.

[0256] In one embodiment, the electronic device (101) may analyze the text (1241a) (or text (1251a)) written in the first language (e.g., English) using the other model while generating the text (1251b) written in the second language (e.g., Korean) from the text (1241a) (or text (1251a)) written in the first language (e.g., English) using the other model. For example, the electronic device (101) may analyze the content of the text (1241a) (or text (1251a)) written from the text (1241a) (or text (1251a)) written in the first language (e.g., English) using the other model. For example, the electronic device (101) may determine that the content of the text (1241a) (or text (1251a)) is negative using the other model. After this, if the content of the text (1241a) (or text (1251a)) is negative, the electronic device (101) can display the view area (1251) to which the first visual effect is applied.

[0257] Additionally, according to one embodiment, the electronic device (101) may analyze the text (1242a) (or text (1252a)) written in the first language (e.g., English) using the other model while generating the text (1252b) written in the second language (e.g., Korean) from the text (1242a) (or text (1252a)) written in the first language (e.g., English) using the other model. For example, the electronic device (101) may analyze the content of the text (1242a) (or text (1252a)) written from the text (1242a) (or text (1252a)) written in the first language (e.g., English) using the other model. For example, the electronic device (101) may confirm that the content of the text (1242a) (or text (1252a)) is positive using the other model. After this, if the content of the text (1242a) (or text (1252a)) is negative, the electronic device (101) can display the view area (1252) to which the second visual effect is applied.

[0258] FIG. 12c illustrates an example of a method for translating text within content having an image and displaying a screen containing the translated text, according to one embodiment.

[0259] FIG. 12C illustrates examples (1261, 1262) of a method for displaying a screen including content having another image that includes translated text and a translation of the text within the image by the electronic device (101) through the function of the second software application. For example, the electronic device (101) of FIG. 12C may be an example of the electronic device (101) of FIG. 3A.

[0260] Referring to example (1261), the electronic device (101) may display a screen (1265) of a first software application providing a messenger service. For example, the screen (1265) may display view areas. For example, the view area (1270) may include content having an image. For example, the image may include text (e.g., "I love you so much") written in a first language (e.g., Korean) and an emoji. For example, the emoji may include a sticker or an emoticon. For example, the image may include a picture or a graphics interchange format (GIF) file.

[0261] Although not illustrated in FIG. 12c, according to one embodiment, the electronic device (101) may activate the function of the second software application. For example, the electronic device (101) may activate the function based on an event (e.g., an automatic event or a manual event) for activating the function of the second software application.

[0262] According to one embodiment, the electronic device (101) may obtain layer information from the first software application. For example, the layer information may include first information about the view area (1270). For example, the first information about the view area (1270) may include the image. For example, the electronic device (101) may extract the image through the content capture function of the second software application and extract text (e.g., "I love you so much") from the image. In other words, the electronic device (101) may extract text within the image by analyzing the image (or photo, GIF file) through the content capture function.

[0263] For example, the electronic device (101) can recognize whether the text needs to be translated on the framework. For example, if the text needs to be translated, the electronic device (101) can translate the text into text (e.g., "I love you") written in a second language (e.g., English) through a translation model. Thereafter, the electronic device (101) can generate (or reconstruct) text (e.g., "I love you") written in the second language (e.g., English), text (e.g., "I love you so much") written in the first language (e.g., Korean), and another image including the emoji on the framework. For example, the electronic device (101) can reconstruct the first information for the view area (1270) based on information about the other images.

[0264] Referring to example (1262), the electronic device (101) can display a reconfigured screen (1265a). For example, the electronic device (101) can display the reconfigured screen (1265a) through the activated function of the second software application. For example, the screen (1265a) can include reconfigured view areas. For example, the view area (1280) can be generated through the reconfigured first information. For example, the view area (1280) can be a view area that has been changed (or reconfigured) from the view area (1270). For example, the view area (1280) can include content having the other image. For example, the other image can include text written in the second language (e.g., English) (e.g., "I love you"), text written in the first language (e.g., Korean) (e.g., "I love you so much"), and the emoji.

[0265] Referring to FIG. 12C, the reconstructed view area (1280) may be substantially the same size as the view area (1270). For example, the size of the view area (1280) may be the same as the view area (1270). In this case, even if the content of the view area (1280) is changed (e.g., the addition of translated text within the image), the size of the view area (1280) may be the same as the view area (1270). However, the embodiments of the present disclosure are not limited thereto. For example, the size of the view area (1280) may change as the content of the view area (1280) is changed.

[0266] In FIG. 12C, the other image is illustrated as including text written in the second language (e.g., English) (e.g., "I love you"), text written in the first language (e.g., Korean) (e.g., "I love you so much"), and the emoji, although embodiments of the present disclosure are not limited thereto. For example, the other image may also include text written in the second language (e.g., English) (e.g., "I love you") and the emoji.

[0267] FIGS. 13A and 13B illustrate examples of a method for changing settings of a second software application through input to a visual object, according to one embodiment.

[0268] FIG. 13A illustrates examples (1301, 1302, 1303) of executing the function of the second software application based on the electronic device (101) obtaining input for a floating (or displayed) visual object while the first software application providing the messenger service is running in the foreground. For example, the electronic device (101) of FIG. 13A may be an example of the electronic device (101) of FIG. 3A.

[0269] Referring to example (1301) of FIG. 13A, according to one embodiment, the electronic device (101) may display a screen (1305) in response to at least a portion of an input for an icon (not shown) for the first software application. For example, the screen (1305) may include a screen representing a chat between a user of the electronic device (101) (e.g., Kim of FIG. 10C) and another user (e.g., Linsey Smith) via the first software application. For example, the screen (1305) may include a view area (1310). For example, the view area (1310) may include text (e.g., "Hi, do you have time to talk to me?") written in a first language (e.g., English).

[0270] According to one embodiment, when the electronic device (101) starts displaying the screen (1305) representing the conversation with the other user, the electronic device (101) may display a visual object (1315). For example, when the screen (1305) is displayed, the electronic device (101) may display a visual object (1315) floating on the screen (1305). For example, the visual object (1315) may be used to inquire whether the live translation function of the second software application is to be executed. For example, the visual object (1315) may include an icon (1317) for removing the visual object (1315). For example, the visual object (1315) may be displayed in a portion of the screen (1305) (e.g., the upper right).

[0271] According to one embodiment, the electronic device (101) may execute the live translation function of the second software application in response to receiving an input (1319) for a region different from the region of the icon (1317) among the visual objects (1315). Referring to example (1302), the electronic device (101) may display a changed view region (1320). For example, the view region (1320) may include text written in a first language (e.g., English) (e.g., "Hi, do you have time to talk to me?") and text written in a second language (e.g., Korean) (e.g., "Hello, do you have time to talk to me?"). Also, referring to example (1302), the electronic device (101) may, in response to receiving an input (1319), display a visual object (1330) for changing the languages ​​(e.g., source language and target language) of the function of the second software application. At this time, the electronic device (101), in response to receiving the input (1319), may stop displaying the visual object (1315) and display the visual object (1330). For example, the visual object (1330) may be displayed in a floating state on the screen (1305). According to one embodiment, the visual object (1330) may include an icon (1331) for changing the source language (e.g., English) and an icon (1332) for changing the target language (e.g., Korean).

[0272] According to one embodiment, the electronic device (101) may stop displaying the visual object (1330) when a specified time interval has elapsed from the timing at which the visual object (1330) was displayed. Referring to example (1303), the electronic device (101) may stop displaying the visual object (1330) and display the view area (1320) of the screen (1305) after the specified time interval (e.g., N seconds) from the timing at which the visual object (1330) was displayed.

[0273] Although not illustrated in FIG. 13A, in one embodiment, the electronic device (101) may, in response to receiving an input for the icon (1317), stop displaying the visual object (1315) without executing the live translation function of the second software application. If the electronic device (101) displays another screen representing a chat with another user (e.g., Linsey Smith), a visual object that is used to inquire whether to execute the live translation function of the second software application may be displayed on the other screen.

[0274] According to one embodiment, the electronic device (101) may display the visual object (1315) again when a specified time interval (e.g., M seconds) has elapsed from the time at which the display of the visual object (1315) was stopped in response to obtaining an input for the icon (1317). For example, the electronic device (101) may display the visual object (1315) again in the partial region upon recognizing that a specified time interval (e.g., M seconds) has elapsed from the time at which the display of the visual object (1315) was stopped. Alternatively, the electronic device (101) may display the visual object (1315) again upon obtaining an input for composing a message (or an input for executing a keyboard application) after stopping the display of the visual object (1315). Alternatively, the electronic device (101) may display the visual object (1315) again based on obtaining input for the portion of the area where the visual object (1315) is displayed after the display of the visual object (1315) has been stopped. For example, the input for the portion of the area may include a swipe input.

[0275] FIG. 13B illustrates examples (1351, 1352) of changing the languages ​​(e.g., source language and target language) of the function of the second software application based on obtaining input for the icon (1331) or the icon (1332) in the example (1302) of FIG. 13A. For example, the electronic device (101) of FIG. 13B may be an example of the electronic device (101) of FIG. 3A.

[0276] Referring to example (1351), according to one embodiment, the electronic device (101) may display a visual object (1360) that displays a list for changing the source language upon receiving an input for the icon (1331). For example, the visual object (1360) may include an icon (1361) representing a language currently used as the source language and icons representing languages ​​that can be used as the source language. Additionally, for example, the visual object (1360) may include an icon (1362) for adding a source language. For example, the icon (1362) for adding may include text (e.g., "+ add language").

[0277] Referring to example (1351), the icon (1361) representing the language currently used as the source language may have a different visual effect than the languages ​​representing the languages ​​that can be used as the source language. For example, the visual effect may include the addition of a color or check mark to the text representing the language.

[0278] Referring to examples (1351) and (1352), according to one embodiment, the electronic device (101) may display a visual object (1370) for adding source language packs upon obtaining an input (1365) for an icon (1362) for adding. For example, the electronic device (101) may display a visual object (1370) including a first portion (1371) that includes information (or language packs) about languages ​​that have already been installed (or downloaded) and a second portion (1372) that includes information about languages ​​that can be installed (or downloaded). For example, the first portion (1371) may include information about English, Korean, and Spanish. For example, the second portion (1372) may include information about language #1 (e.g., Japanese), language #2, language #3, language #4, language #5, and language #6. The electronic device (101) may download information about language #1 based on obtaining an input (1375) for an icon (1373) that executes a download of information about language #1 (e.g., Japanese). Although not illustrated in FIG. 13B, when information about language #1 is downloaded, language #1 may be removed from the second portion (1372) of the visual object (1370) and displayed in the first portion (1371).

[0279] FIG. 14 illustrates an example of an operational flow for a method of an electronic device translating content included in a screen of a software application and displaying a screen including translated text, according to one embodiment.

[0280] At least some of the methods of FIG. 14 may be performed by the electronic device (101) of FIG. 3A. For example, at least some of the methods may be controlled by the processor (310) of the electronic device (101). In the following embodiments, the operations may be performed sequentially, but are not necessarily performed sequentially. For example, the order of the operations may be changed, and at least two operations may be performed in parallel.

[0281] According to one embodiment, in operation (1410), the electronic device (101) may display a screen of a first software application. According to one embodiment, while the first software application is running, the electronic device (101) may display the screen, which is an execution screen of the first software application. For example, the screen may be referred to as a user interface (UI). The first software application may be running in the foreground.

[0282] For example, the screen may be a screen that provides executable objects through user interaction (e.g., touch input) while the first software application is running, or provides content including visual objects (e.g., text, images). For example, the screen may include information about multiple areas.

[0283] According to one embodiment, in operation (1420), the electronic device (101) may display a screen including content having text written in a first language and including a view area having a first size. For example, the electronic device (101) may display the screen including content having text written in the first language and including the view area having the first size, based on at least a portion of an input to the screen.

[0284] According to one embodiment, in operation (1430), the electronic device (101) may display the screen including the view area having a second size larger than the first size, the screen including the text written in the first language and other content having another text written in a second language different from the first language, based on the activation of the function of the second software application. For example, the electronic device (101) may display the screen including the view area having the first size, based on the activation of the function of the second software application while displaying the screen including the view area having the first size, the screen including the other content and including the view area having the second size.

[0285] According to one embodiment, the electronic device (101) may obtain layer information from the first software application while displaying the screen including the view area having the first size. For example, the electronic device (101) may obtain the layer information by using a content capture function of the second software application while the second software application is running in the background. For example, the execution of the content capture function may be performed at a specified period or when an event such as a change in the screen of the first software application occurs. For example, the layer information may include information about a first area, a second area, a third area, and a fourth area. The layer information may be layer information about a portion of a screen displayed in a display area of ​​the display (160) of the electronic device (101) among screens that may be provided by the first software application.

[0286] According to one embodiment, the electronic device (101) can determine whether the function of the second software application is activated. For example, the function can be activated through a setting of the electronic device (101) or based on a specific event. For example, the function of the second software application can represent a live translation function that obtains the hierarchical information obtained from the first software application, translates a portion of text (e.g., text of the fourth region) recognized through the framework using the obtained hierarchical information, and reconstructs the hierarchical information using the translated text.

[0287] According to one embodiment, the electronic device (101) can recognize the first information and the second information when the function of the second software application is activated. For example, the electronic device (101) can recognize the first information and the second information from the hierarchical information. For example, the electronic device (101) can recognize the first information and the second information through the second software application. For example, when the function of the second software application is activated, the electronic device (101) can recognize the second information about the text written in the first language and the first information about the view area having the first size within the hierarchical information through the second software application.

[0288] According to one embodiment, the electronic device (101) may transmit the first information and the second information to the framework. For example, the electronic device (101) may transmit the first information and the second information recognized through the second software application to the framework. For example, the framework may include a view framework that manages view objects included in the hierarchical information. For example, the view objects may include content including a screen, a screen object, a view area, and text (and / or images).

[0289] According to one embodiment, the electronic device (101) may determine whether the screen includes text written in a language different from the user language (or target language). For example, the electronic device (101) may determine, based on the second information, whether the screen includes text written in a language different from the second language (e.g., the first language). It is assumed that the user language (or the target language) is the second language. For example, the electronic device (101) may determine, when the function is activated, whether text written in a language different from the second language exists among the texts included in the screen.

[0290] According to one embodiment, the electronic device (101) may execute a model for translation (hereinafter, referred to as a translation model) upon recognizing the text written in the first language within the screen. For example, if the text written in the first language within the screen is included, the electronic device (101) may transmit a request for translation from the translation framework of the framework to the second software application. For example, the request may include the second information. Or, for example, the request may include a string of text of the second information.

[0291] According to one embodiment, in response to the request, the electronic device (101) may execute the translation model for translating the text written in the first language. For example, the translation model may be implemented as a function of the second software application. However, the embodiments of the present disclosure are not limited thereto. For example, the translation model may be installed in a storage space within the memory (330) of the electronic device (101). If the translation model is installed in the storage space, the translation model may be executed upon a call from the second software application that received the request.

[0292] According to one embodiment, the electronic device (101) can generate a translated text. For example, the electronic device (101) can transmit information (e.g., the second information) about the text written in the first language recognized by the framework to the translation model. For example, the electronic device (101) can use the second information through the translation model to translate the text written in the first language within the second information. For example, the electronic device (101) can generate another text translated into the second language from the text. For example, the electronic device (101) can generate third information about the another text.

[0293] According to one embodiment, the electronic device (101) may translate the text within the request through the translation model. For example, the electronic device (101) may translate the text written in the first language into another text written in the second language through the translation model. For example, the second language may be provided to the translation model from the second software application. For example, the second software application may recognize the language of the text being input or a language set in the system settings of the electronic device (101) as the user language (or target language).

[0294] While the above example illustrates a method for performing translation using the translation model, the embodiments of the present disclosure are not limited thereto. For example, without using the translation model, the electronic device (101) may also translate text written in a language different from the user's language through an external electronic device (e.g., a server) that provides a translation service.

[0295] According to one embodiment, the electronic device (101) may transmit the third information to the framework. The framework may include the translation framework or the view framework. If the framework is the translation framework, the third information may be transmitted to the view framework.

[0296] According to one embodiment, the electronic device (101) can update the hierarchical information. For example, the electronic device (101) can update (or reconfigure, change) the hierarchical information including the second information based on the third information into the hierarchical information including the second information and the third information. In other words, the hierarchical information can further include the third information together with the second information.

[0297] According to one embodiment, the electronic device (101) may display a changed screen. For example, while the first software application is running, the electronic device (101) may change (or reconfigure, update) the screen based on the hierarchical information including the third information and the second information, and display the changed screen through the display (320). For example, the changed screen may include the text and the translated other text within the view area.

[0298] For example, the view area including the text (or the view area based on the second information) may have the first size. For example, the size of the view area including the text and the other text (or the view area based on the second information and the third information) may have the second size, which is larger than the first size.

[0299] FIG. 15 illustrates an example of a generative artificial intelligence system according to one embodiment. The generative artificial intelligence system (1500) of FIG. 15 illustrates an example of a system including a generative AI model (1530). The generative artificial intelligence system (1500) may be included in an electronic device (101). For example, the generative AI model (1530) may include the translation model (340) of FIG. 3A or the translation model (375) of FIG. 3B.

[0300] A user query / response interface (1510) can receive user input. The user input may be in the form of natural language, images, and / or videos. Furthermore, context information may also be transmitted when the user input is transmitted. The context information may include various additional information at the time of user input. For example, the various additional information may include information about the application currently being used by the user or information about the user's location. Furthermore, the user input may be in a mixed form of natural language, images, sounds, and context information. Furthermore, the user input may also be in a non-natural language form, such as selecting a menu. The user query / response interface (1510) can output the results of the generative artificial intelligence system (1500) to the user. The output may be in the form of natural language or specific content, and may also be provided in the form of an action requested by the user. The user query / response interface (1510) can output the results of the generative artificial intelligence system (1500) to the user. The output can be in natural language form, in the form of specific content, or in the form of actions requested by the user.

[0301] The AI ​​framework (1520) can receive user input and coordinate and control each component necessary to perform the user's intention based on the user's query.

[0302] User input received from the user question response interface (1510) can be transmitted to a prompt design component (1521). The prompt design component (1521) can be used to generate a prompt suitable for inputting the user input into an LLM or LMM. The prompt design component (1521) can be an AI component that uses a machine learning algorithm or a neural network to develop better prompts over time. The prompt design component (1521) can access knowledge repositories (1540) containing user preference data, a prompt library, and prompt examples based on the user input to generate a prompt, and transmit the generated prompt to the LLM or LMM.

[0303] The API / plug-in management component (1522) can communicate with external information when there is a request for additional information when passing user input as input to the generative model. The API / plug-in management component (1522) can establish a channel for communicating with the outside of the AI ​​Interface through the API, and can enable access to various data sources through the established channel. In addition, the API / plug-in management component (1522) can request an action through the API when the application / service component (1550) needs to perform an action that performs the user input as a final result rather than an intermediate result. Information obtained from the outside can be used to generate a prompt in the prompt design component (1521) together with the user input, or can be passed as an input to the generative AI model (1530).

[0304] The refiner component (1523) can fine-tune the output from the generative model. For example, the refiner component (1523) can verify that the content generated through the LLM and / or LMM is not irrelevant, biased, or harmful. Furthermore, the refiner component (1523) can determine the degree to which the content matches the user's desired result and, if necessary, perform additional processing. The refiner component (1523) can additionally configure and provide hints to the user to avoid undesirable output.

[0305] A generative AI model (1530) may generally refer to an artificial intelligence neural network that creates new types of data based on user input information. The generative AI model (1530) may include an image-generating model and / or a language-generating model. Representative models for generating images include a generative adversarial network (GAN) and a variational auto encoder (VAE), and examples include a Diffusion-based generative model that uses a VAE and a Transformer structure. A language-generating model is a model trained to statistically output the most appropriate output value based on an input value, and representative examples include models such as CHAT-GPT 3 and CHAT-GPT 4. In addition, language-generating models include large multimodal models (LMMs) that can recognize various types of data input, such as text, images, and voice, and generate new data corresponding to them.

[0306] As described above, the device, method, and storage medium according to embodiments of the present disclosure can provide the user with translated content in real time on a framework via the second software application while using the first software application. Accordingly, regardless of whether the first software application provides a translation function, the user can conveniently view the translated content without performing additional input.

[0307] In addition, the device, method, and storage medium according to embodiments of the present disclosure can perform translation of the contents of the first software application through the second software application and provide the translated text as reconstructed information on the framework. When displaying a screen using the reconstructed information on the framework, the configuration of the screen (e.g., the arrangement, size, visual effects, or contents of view objects) can be optimized and changed according to rules specified in the first software application. Accordingly, the device, method, and storage medium according to embodiments of the present disclosure can naturally display the screen further including the translated text according to the live translation function.

[0308] As described above, the electronic device (101) may include a memory (330) that stores instructions. The electronic device (101) may include a display (320). The electronic device (101) may include a processor (310). The instructions, when executed by the processor (310), may cause the electronic device (101) to display a screen of a first software application through the display (320) based on the execution of the first software application. The instructions, when executed by the processor (310), may cause the electronic device (101) to display, through the display (320), a screen including content having text written in a first language and including a view area having a first size, based on at least a portion of an input to the screen. The instructions, when executed by the processor (310), may cause the electronic device (101) to display, through the display (320), a screen including content having text written in the first language and another text written in a second language different from the first language, and including the view area having a second size larger than the first size, based on a function of a second software application being activated while displaying the screen including the view area having the first size.

[0309] In one embodiment, the first software application may not provide a function for translation from the first language to the second language within the first software application.

[0310] According to one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to, in response to at least a portion of the input to the screen, display a screen object of the first software application through the display (320) so as to at least partially overlap the screen, and to obtain hierarchy information constituting the screen from the first software application while the second software application including the function is executed in the background. The screen object may include the view area including the content.

[0311] According to one embodiment, the layer information may be provided from the first software application to the second software application based on execution of another function for capturing content of the second software application.

[0312] According to one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to recognize, from the hierarchy information, first information about the view area having the first size and second information about the text written in the first language. The instructions, when executed by the processor (310), may cause the electronic device (101) to transfer the first information and the second information to a framework of the electronic device (101). The first information may include an identifier of the view area, coordinates indicating a location of the view area, and length values ​​indicating the first size of the view area. The second information may include a string of the text written in the first language, a color of the text, and a size of the text.

[0313] In one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to determine, through the framework, whether the view area includes text written in a language different from the second language. The instructions, when executed by the processor (310), may cause the electronic device (101) to execute an artificial intelligence model for translation in response to a request obtained from the framework based on determining that the view area includes text written in the first language different from the second language.

[0314] According to one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to generate the other text written in the second language from the text written in the first language using the executed artificial intelligence model. The instructions, when executed by the processor (310), may cause the electronic device (101) to transmit third information about the other text generated to the framework. The instructions, when executed by the processor (310), may cause the electronic device (101) to update the layer information based on the third information through the framework. The view area having the second size may be displayed based on the third information.

[0315] According to one embodiment, the artificial intelligence model may be stored in the memory (330) of the electronic device (101) or included in the second software application.

[0316] In one embodiment, the feature may include a translation feature for displaying the text written in the first language together with other text written in the second language.

[0317] According to one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to display, through the display (320), another screen changed from the screen based on at least a portion of another input to the screen while the function is activated and the screen including the view area having the second size is displayed. The other screen may include another view area having text written in the first language and other text written in the second language.

[0318] In one embodiment, the other input may include a scroll input for the screen. The other screen may include at least a portion of the view area.

[0319] According to one embodiment, the function may be included in an executable object having a keyboard shape provided through a keyboard application of the electronic device (101), and may be activated based on an input to an icon for translation of text displayed in a bar of the executable object.

[0320] According to one embodiment, the function may be activated based on an input to a submenu related to the function within a menu for a keyboard application of the electronic device (101). The menu for the keyboard application may be included in the system settings of the electronic device (101). The submenu may include an icon for activating the function and icons for setting the first language and the second language.

[0321] In one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to generate a summary of the text written in the first language exceeding a reference length using an analytic model. The instructions, when executed by the processor (310), may cause the electronic device (101) to display, through the display (320), the screen including the view area further including the summary together with the text and the other text.

[0322] According to one embodiment, the instructions, when executed by the processor (310), may cause the electronic device (101) to determine a visual effect according to the content having the text, using an analytical model, from the text written in the first language. The instructions, when executed by the processor (310), may cause the electronic device (101) to display, through the display (320), the screen including the view area having the visual effect.

[0323] In one embodiment, the analysis model may include a large language model (LLM) trained based on the text written in the first language.

[0324] In one embodiment, the content of the view area having the first size may include an image including the text written in the first language. The other content of the view area having the second size may include another image including the text written in the first language and the other text written in the second language.

[0325] In one embodiment, the text written in the first language in the view area having the first size may have a first color. The text written in the first language in the view area having the second size may have a second color having a lower saturation than the saturation of the first color. The other text written in the second language in the view area having the second size may have the first color.

[0326] A method performed by an electronic device (101) as described above may include an action of displaying a screen of a first software application based on execution of the first software application. The method may include an action of displaying the screen, based on at least a portion of an input to the screen, including content having text written in a first language and including a view area having a first size. The method may include an action of displaying the screen, including content having text written in the first language and another text written in a second language different from the first language, including the view area having a second size larger than the first size, based on activation of a function of a second software application while displaying the screen including the view area having the first size.

[0327] As described above, a non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by a processor (310) of an electronic device (101) including a display (320), cause a screen of a first software application to be displayed through the display (320) based on execution of the first software application. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor (310), cause a screen including contents having text written in a first language and including a view area having a first size, based on at least a portion of an input to the screen. The non-transitory computer-readable storage medium may store one or more programs including instructions that, when executed by the processor (310), cause the screen including the view area having the first size to be displayed, through the display (320), the screen including the view area having the second size larger than the first size, and including other content having the text written in the first language and another text written in a second language different from the first language, based on the activation of a function of a second software application while displaying the screen including the view area having the first size.

[0328] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0329] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another component (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0330] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0331] Various embodiments of the present document may be implemented as software (e.g., a program (140)) including one or more instructions stored in a storage medium (e.g., an internal memory (136) or an external memory (138)) readable by a machine (e.g., an electronic device (101)). For example, a processor (e.g., a processor (120)) of the machine (e.g., an electronic device (101)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0332] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as a computer program product. The computer program product may be traded between sellers and buyers as a product. The computer program product may be distributed in the form of a device-readable storage medium (e.g., compact disc read-only memory (CD-ROM)) or may be provided through an application store (e.g., Play Store). TM ) or directly between two user devices (e.g., smart phones), online distribution (e.g., downloading or uploading). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0333] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

1. In an electronic device (101), A memory (330) storing instructions and including one or more storage media; display (320); and At least one processor (310) comprising a processing circuit, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: Based on the execution of the first software application, the screen of the first software application is displayed through the display (320); Displaying, through the display (320), the screen, which includes content having text written in a first language and includes a view area having a first size, based on at least a portion of the input to the screen; and Based on the activation of a function of a second software application while displaying the screen including the view area having the first size, causing the screen to be displayed through the display (320), including the text written in the first language and other content having another text written in a second language different from the first language, and including the view area having a second size larger than the first size. Electronic device (101).

2. In claim 1, The first software application does not provide a function for translation from the first language to the second language within the first software application. Electronic device (101).

3. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: In response to at least some of the input to the above screen: Displaying the screen object of the first software application, at least partially overlapping the screen, through the display (320); and Causing the second software application including the above function to obtain hierarchy information constituting the screen from the first software application while the second software application is running in the background, The above screen object includes the view area including the content, Electronic device (101).

4. In claim 3, The above layer information is provided from the first software application to the second software application based on the execution of another function for capturing the contents of the second software application. Electronic device (101).

5. In claim 3, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: Recognize first information about the view area having the first size and second information about the text written in the first language from the hierarchical information; and Causing the first information and the second information to be transmitted to the framework of the electronic device (101), The first information includes an identifier of the view area, coordinates indicating a location of the view area, and length values ​​indicating the first size of the view area, and The second information includes a string of the text written in the first language, a color of the text, and a size of the text. Electronic device (101).

6. In claim 5, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: Through the above framework, determining whether the view area contains text written in a language different from the second language; and Causing an artificial intelligence model for translation to be executed in response to a request obtained from the framework, based on determining that the view area includes the text written in the first language different from the second language. Electronic device (101).

7. In claim 6, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: Using the artificial intelligence model executed above, generating another text written in the second language from the text written in the first language; Passing third information about the other generated text to the framework; and Through the above framework, the hierarchical information is updated based on the third information, The view area having the second size is displayed based on the third information. Electronic device (101).

8. In claim 1, The above function includes a translation function for displaying the text written in the first language together with another text written in the second language. Electronic device (101).

9. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: While the above function is activated and the screen including the view area having the second size is displayed, based on at least some of other inputs to the screen, another screen changed from the screen is caused to be displayed through the display (320). The other screen comprises another view area having text written in the first language and another text written in the second language. Electronic device (101).

10. In claim 1, The above function is activated based on an input to a submenu related to the above function within a menu for a keyboard application of the electronic device (101). The above menu for the above keyboard application is included in the system settings of the electronic device (101), The above submenus are: Icon for activating the above function; and Contains icons for setting the first language and the second language. Electronic device (101).

11. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: generating a summary of said text using an analytic model from said text written in said first language exceeding a reference length; and Causing the screen to be displayed, through the display (320), including the view area further including the summary together with the text and the other text; Electronic device (101).

12. In claim 1, The above instructions, when individually or collectively executed by the at least one processor (310), cause the electronic device (101) to: From the text written in the first language, using an analytic model, determining a visual effect according to the content having the text; and Causing the screen, including the view area having the above visual effect, to be displayed through the display (320). Electronic device (101).

13. In claim 1, The text written in the first language in the view area having the first size has a first color, The text written in the first language in the view area having the second size has a second color having a lower saturation than the saturation of the first color, and The other text written in the second language in the view area having the second size has the first color, Electronic device (101).

14. In a method performed by an electronic device (101), An action of displaying a screen of a first software application based on the execution of the first software application; An operation of displaying a screen including content having text written in a first language and including a view area having a first size, based on at least a portion of an input to the screen; and An operation of displaying a screen including the view area having the first size, based on activation of a function of a second software application while displaying the screen including the view area having the first size, the screen including other content having the text written in the first language and another text written in a second language different from the first language, and including the view area having a second size larger than the first size. method.

15. A non-transitory computer-readable storage medium, when executed individually or collectively by at least one processor (310) of an electronic device (101) including a display (320), causes the electronic device (101) to: Based on the execution of the first software application, the screen of the first software application is displayed through the display (320); Displaying, through the display (320), the screen, which includes content having text written in a first language and includes a view area having a first size, based on at least a portion of the input to the screen; and storing one or more programs including instructions that cause the screen, including the view area having the first size, to be displayed through the display (320), the screen including the view area having the second size larger than the first size, the screen including the text written in the first language and other content having another text written in a second language different from the first language, based on the activation of a function of the second software application while displaying the screen including the view area having the first size; A non-transitory computer-readable storage medium.

Images