Electronic device, method, and non-transitory computer-readable storage medium for translating utterance in call

Abstract

The present invention provides a method of an electronic device, the method comprising the operations of: in order to display, using a display, a first user interface (UI) of a first application for a voice over Internet protocol (VoIP) call with another electronic device performed using a communication circuit, executing the first application; on the basis of an indication for establishing the VoIP call, displaying, together with the first UI and using the display, a UI object which receives an input for performing a translation service during the VoIP call with the other electronic device; receiving an input for the UI object; and displaying, together with the first UI and using the display, a second UI of a second application for the translation service on the basis of the input.

Description

Electronic device, method, and non-transitory computer-readable storage medium for translating speech within a call

[0001] The following descriptions relate to electronic devices, methods, and non-transitory computer-readable storage media for translating speech within a call.

[0002] Digital assistants (also known as virtual assistants or intelligent automated assistants) can provide a beneficial human-machine interface. For example, digital assistants can be used to translate from one language to another. For example, digital assistants can be used to translate a user's speech during a phone call.

[0003] The above information may be provided as background art to aid in understanding the present disclosure. No claim or determination is made as to whether any of the above-described matters constitute prior art related to the present disclosure.

[0004] An electronic device is provided. The electronic device may include a display. The electronic device may include communication circuitry. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute a first application for a voice over internet protocol (VoIP) call with another electronic device using the communication circuitry, to display a first user interface (UI) of the first application using the display. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, using the display, a UI object that receives an input for performing a translation service during the VoIP call with the other electronic device, based on an indication for establishing the VoIP call, together with the first UI. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to receive an input for the UI object. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, using the display, a second UI of a second application for the translation service, together with the first UI, based on the input.

[0005] A method is provided. The method can be executed in an electronic device having a communication circuit and a display. The method can include an operation of executing a first application to display a first user interface (UI) of a first application for a voice over internet protocol (VoIP) call with another electronic device performed using the communication circuit, using the display. The method can include an operation of displaying, using the display, together with the first UI, a UI object for receiving an input for performing a translation service during the VoIP call with the other electronic device based on an indication for establishing the VoIP call. The method can include an operation of receiving an input for the UI object. The method can include an operation of displaying, using the display, together with the first UI, a second UI of a second application for the translation service based on the input.

[0006] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display and communication circuitry, cause the electronic device to execute a first application to display, using the display, a first user interface (UI) of a first application for a voice over internet protocol (VoIP) call with another electronic device that is performed using the communication circuitry. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display, using the display, a UI object that receives an input for performing a translation service during the VoIP call with the other electronic device, based on an indication for establishing the VoIP call, together with the first UI. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to receive an input for the UI object. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to display, using the display, a second UI of a second application for the translation service, together with the first UI, based on the input.

[0007] An electronic device is provided. The electronic device may include a display. The electronic device may include communication circuitry. The electronic device may include at least one processor including a processing circuit. The electronic device may include a memory storing instructions and including one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to initiate a translation service during a first call performed through an application providing the second UI based on receiving a signal from the other electronic device, using the communication circuitry, indicating acceptance of a call connection request in relation to the first call, while displaying a second UI for a translation service together with a first UI for a first call with another electronic device performed through circuit switching using the communication circuitry. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to initiate a translation service during the second call performed through the application based on performing automatic speech recognition (ASR) of a user's utterance of the other electronic device initially received from the other electronic device using the communication circuitry in connection with the second call with the other electronic device while displaying the second UI using the display together with the third UI for the second call with the other electronic device performed according to packet switching using the communication circuitry.

[0008] A method is provided. The method can be executed in an electronic device including a communication circuit and a display. The method can include an operation of displaying a second UI for a translation service together with a first UI for a first call with another electronic device performed by circuit switching using the communication circuit, using the display, and starting a translation service during the first call performed through an application providing the second UI based on receiving a signal from the other electronic device using the communication circuit, indicating acceptance of a call connection request in relation to the first call, wherein the second UI is provided together with a third UI for a second call with the other electronic device performed by packet switching using the communication circuit, using the display, and starting a translation service during the second call performed through the application based on performing automatic speech recognition (ASR) of a user's utterance of the other electronic device, initially received from the other electronic device using the communication circuit, in relation to the second call with the other electronic device.

[0009] A non-transitory computer-readable storage medium is provided. The non-transitory computer-readable storage medium may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display and a communication circuit, cause the electronic device to display a second UI for a translation service together with a first UI for a first call with another electronic device performed by circuit switching using the communication circuit, using the display, and to initiate a translation service during the first call performed through an application providing the second UI based on receiving a signal indicating acceptance of a call connection request related to the first call from the other electronic device using the communication circuit. The one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to start a translation service during the second call performed through the application based on performing automatic speech recognition (ASR) of a user's speech of the other electronic device initially received from the other electronic device using the communication circuitry in connection with the second call with the other electronic device while displaying the second UI using the display together with the third UI for the second call with the other electronic device performed according to packet switching using the communication circuitry.

[0010] Figure 1 illustrates an example of a service provided through a first application.

[0011] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0012] Figure 3 is a flowchart illustrating a method of performing a translation service during a VoIP (voice over internet protocol) call.

[0013] Figure 4 is a flowchart illustrating a method for providing an access point for translation services during a VoIP call.

[0014] Figure 5 illustrates an example of a first UI of a first application for making a VoIP call.

[0015] FIG. 6 is a flowchart illustrating a method performed in relation to user identification information of another electronic device.

[0016] Figure 7 illustrates an example of setting source and target languages ​​within contact information related to identification information.

[0017] Figure 8a illustrates an example of UI objects within a quick panel for translation services during a VoIP call.

[0018] Figure 8b illustrates a UI for user settings of the translation service.

[0019] Figure 9 illustrates an example of a UI object for a translation service during a VoIP call displayed with the first UI.

[0020] Figure 10 illustrates a second UI overlaid on a first UI.

[0021] Figure 11 illustrates a second UI overlaid on a first UI displayed according to a split view.

[0022] Figure 12 is a flowchart illustrating a method for initiating a translation service during a VoIP call.

[0023] Figure 13 illustrates an example of information within a second UI displayed in connection with initiating a translation service during a VoIP call.

[0024] Figure 14 is a flowchart illustrating a method for correcting information related to speech during a VoIP call while performing a translation service during the VoIP call.

[0025] Figure 15 illustrates an example of a trained model used to correct information related to speech during a VoIP call.

[0026] Figure 16 is a flowchart illustrating a method for terminating a translation service during a VoIP call.

[0027] Figure 17 illustrates an example of a second UI that is interrupted to terminate a translation service during a VoIP call.

[0028] FIG. 18 is a block diagram of an electronic device within a network environment according to various embodiments.

[0029] VoIP calls can be a voice communication method for transmitting voice communication sessions over an IP network, such as the Internet. These VoIP calls can be described as Internet telephony, broadband telephony, and broadband telephone services. Unlike telephone calls, which are performed using circuit switching, these VoIP calls can be performed using packet switching.

[0030] The application used for the above VoIP call may provide other services distinct from the VoIP call service. For example, the application may provide services for SMS (short message service), voice messaging, and / or video conferencing over the Internet. The services provided through the application are described with reference to FIG. 1.

[0031] Figure 1 illustrates an example of a service provided through a first application.

[0032] Referring to FIG. 1, the first application may be available for VoIP calls. The first application may provide messaging services in addition to the VoIP call service.

[0033] For example, the first application may provide a UI (110) for the message service. For example, the electronic device may display a UI (110-1) including a list (111) of chats related to a user of the electronic device. The electronic device may receive an input (113) for an object (112) among objects in the list (111) for searching conversations within each of the chats. Based on the input (113), the electronic device may change or switch the state of the UI (110). For example, the electronic device may display a UI (110-2) including a chat window (114) that provides conversations within a chat represented by an object (112), an object (115) for a video call with the other party of the chat represented by the object (112), an object (116) for a VoIP call with the other party, an object (117) for displaying information related to the other party, and a message input field (118) for a conversation within the chat. For example, the display of the UI (110-1) may be interrupted according to the display of the UI (110-2). For example, the UI (110-1) may be switched or changed to the UI (110-2) based on an input (113). For example, the electronic device may receive an input (119) for the object (116). The electronic device may display a UI (160) that is changed from the UI (110) based on the input (119).

[0034] The UI (160) may be used for a VoIP call performed through the first application. For example, while the electronic device attempts to connect to another electronic device (e.g., the electronic device of the other party) for the VoIP call, the electronic device may display a UI (160-1) including identification information (161) of the other party (e.g., the other party's name (161-1), the other party's phone number (161-2), and / or the other party's profile picture (161-3)), an object (162) for a mute function, an object (163) for a speaker phone function, and an object (164) for stopping (or terminating) the attempted VoIP call. The UI (160-1) may further include a blank area (165).

[0035] For example, the electronic device may change or switch the state of the UI (160) based on the success of the connection attempt with the other electronic device. For example, the electronic device may display a UI (160-2) including an object (171) for a mute function, an object (166) for a video call function, an object (167) for changing a device that outputs the other party's speech, an object (168) for pausing the VoIP call, an object (169) for displaying objects for performing additional functions, and identification information (173) of the other party while the electronic device is connected to the other electronic device for the VoIP call. For example, the display of the UI (160-1) may be stopped according to the display of the UI (160-2). For example, the UI (160-1) may be switched or changed to the UI (160-2) based on the success of the connection attempt with the other electronic device. For example, the arrangement of identification information (173) within UI (160-2) may be the same as or different from the arrangement of identification information (161) within UI (160-1).

[0036] For example, the electronic device may provide a translation service during the VoIP call. The translation service may be performed in conjunction with UI (160) among UI (110) and UI (160). The translation service may be performed through a second application different from the first application. Although the translation service is performed in conjunction with the VoIP call, since the second application is different from the first application, an operation to identify the second application when the VoIP call is being performed through the first application (and / or an attempt to perform the VoIP call through the first application) may be performed within the electronic device.

[0037] The electronic device may perform operations to identify, for the second application, an attempt to make a VoIP call through the first application. As a non-limiting example, the attempt to make a VoIP call identified for the second application may be performed while displaying a UI (160-1) through the first application. The electronic device may perform operations to identify, for the second application, a success of the attempt to make a VoIP call (or a success of a connection to a VoIP call with another electronic device). As a non-limiting example, the success of the attempt to make a VoIP call identified for the second application may be performed while displaying a UI (160-2) through the first application.

[0038] Figure 2 is a simplified block diagram of an exemplary electronic device.

[0039] Referring to FIG. 2, the electronic device (200) may include at least one processor (210), memory (220), communication circuit (230), display (240), microphone (250), and speaker (260). The electronic device (200) may include at least a portion of the electronic device (1801) of FIG. 18, or may correspond to at least a portion of the electronic device (1801) of FIG. 18.

[0040] At least one processor (210) may include any processing circuitry operative to control the performance and operations of one or more components (e.g., communication circuitry (230), display (240), microphone (250), and / or speaker (260)) of the electronic device (200). The at least one processor (210) may execute instructions stored in the memory (220) to perform operations described with reference to FIGS. 3 to 18. The at least one processor (210) may be implemented as a single chip (or a single chip set), such as a system on chip (SoC). For example, the at least one processor (210) may also be implemented as multiple chips (or multiple chip sets).

[0041] For example, at least one processor (210) may include a central processing unit (CPU) (e.g., including a processing circuit). For example, at least one processor (210) may include a neural processing unit (NPU). For example, at least one processor (210) may include at least a portion of the processor (1820) of FIG. 18 or may correspond to at least a portion of the processor (1820) of FIG. 18.

[0042] For example, at least one processor (210) may be used to execute or run one or more software applications, such as an operating system software application, a firmware software application, a media playback software application, a media editing software application, a software application for communicating with other users, a translation software application, a digital assistant software application, and / or any other suitable software applications.

[0043] The memory (220) may include one or more storage media. For example, the one or more storage media may include a hard drive, flash memory, permanent memory such as read-only memory (ROM), semi-permanent memory such as random access memory (RAM), any other suitable type of storage assembly, or any combination thereof. The memory (220) may include a cache memory, which is one or more different types of memory used to temporarily store data for the function or feature of the electronic device (200). The memory (220) may be fixedly embedded within the electronic device (200) or incorporated into one or more suitable types of components (e.g., a subscriber identity module (SIM) card and / or a secure digital (SD) memory card) that can be repeatedly inserted into and removed from the electronic device (200). For example, the memory (220) may include at least a portion of the memory (1830) of FIG. 18 or correspond to at least a portion of the memory (1830) of FIG. 18.

[0044] The memory (220) may store one or more software applications, such as an operating system software application, a firmware software application, a media playback software application, a media editing software application, a software application for communicating with other users, a translation software application, a digital assistant software application, and / or any other suitable software applications. For example, the one or more software applications may include instructions executable by at least a portion of at least one processor (210).

[0045] The communication circuit (230) may be used to communicate with other electronic devices (or external electronic devices). For example, the communication circuit (230) may be used to support calls (e.g., telephone calls) that are performed according to circuit switching. For example, the communication circuit (230) may be used to support calls (e.g., VoIP calls) that are performed according to packet switching. For example, the communication circuit (230) may be used to transmit packets for VoIP calls to the other electronic device, and / or to receive packets for VoIP calls from the other electronic device. For example, the communication circuit (230) may be used to transmit the results of a translation obtained from a translation software application to the other electronic device. For example, the communication circuit (230) may include at least a portion of the communication module (1890) of FIG. 18 or may correspond to at least a portion of the communication module (1890) of FIG. 18.

[0046] The display (240) may be used to display visual information, visual data, screens, and / or user interfaces. For example, the display (240) may be used to display a UI of the first application. For example, the display (240) may be used to display a UI of the second application. For example, the display (240) may include at least a portion of the display module (1860) of FIG. 18 or correspond to at least a portion of the display module (1860) of FIG. 18.

[0047] The microphone (250) may be used to capture audio generated in connection with the electronic device (200). For example, the microphone (250) may receive speech from a user of the electronic device (200) while making a VoIP call.

[0048] The speaker (260) can be used to output audio. For example, the speaker (260) can output speech from a user of another electronic device acquired through a VoIP call conducted via the first application. For example, the speaker (260) can output the results of a translation conducted via the second application as audio.

[0049] Figure 3 is a flowchart illustrating a method of performing a translation service during a VoIP (voice over internet protocol) call.

[0050] Referring to FIG. 3, in operation 301, at least one processor (210) may identify whether a VoIP call with another electronic device is initiated through the first application. For example, at least one processor (210) may identify whether a VoIP call with the other electronic device is initiated through the first application for the purpose of the second application (e.g., used for a translation service). For example, operation 301 may be performed for operation 302. For example, at least one processor (210) may maintain operation 301 until a VoIP call with the other electronic device is initiated through the first application. Operation 301 will be described in more detail with reference to FIG. 4.

[0051] The method of identifying the start of the VoIP call with the other electronic device can be implemented in various ways within the electronic device (200).

[0052] At least one processor (210) (or the second application) can identify whether a touch input is received on the display (240) while the UI of the first application is displayed, and obtain first data for the UI of the first application in response to the touch input received while the UI of the first application is displayed. The first data can be obtained before the touch input is released. At least one processor (210) (or the second application) can obtain second data for the UI of the first application in response to the release of the touch input. At least one processor (210) (or the second application) can identify (or determine) whether to start the VoIP call with the other electronic device based on a difference between the first data and the second data.

[0053] At least one processor (210) (or the second application) can identify whether, while the UI of the first application is displayed, it obtains (or identifies) a command from the first application to set the audio mode to a communication audio mode for a VoIP call, and can identify (or determine) to start the VoIP call with the other electronic device based on the command.

[0054] At least one processor (210) (or the second application) can obtain data regarding the UI of the first application. The data can be obtained based on a touch input received while the UI of the first application is displayed. The data can be obtained periodically while the UI of the first application is displayed. At least one processor (210) (or the second application) can use the data to identify whether a call connection screen is provided through the UI of the first application, and based on the identification that the call connection screen is provided, identify (or determine) to start the VoIP call with the other electronic device.

[0055] At least one processor (210) (or the second application) can identify audio data output through the speaker (260) used for the VoIP call with the other electronic device while the UI of the first application is displayed. At least one processor (210) (or the second application) can identify (or determine) to start the VoIP call with the other electronic device based on identifying that the audio data while the UI of the first application is displayed is audio data indicating a connection (or start) of a call. At least one processor (210) (or the second application) can identify (or determine) to end the VoIP call with the other electronic device based on identifying that the audio data while the UI of the first application is displayed is audio data indicating a disconnection (or end) of a call.

[0056] In operation 302, at least one processor (210) may provide an access point for a translation service during the VoIP call with another electronic device based on identifying that a VoIP call is initiated (or performed) with the other electronic device. For example, the at least one processor (210) may display, as the access point, a UI object (or an executable object) (e.g., a handler) superimposed on the UI of the first application for the VoIP call with the other electronic device. The UI object may enable (or perform) (or execute) providing a translation service through the second application in connection with the VoIP call with the other electronic device. For example, the at least one processor (210) may display, as the access point, another UI object within a quick panel. The another UI object may enable (or perform) (or execute) providing the translation service through the second application in connection with the VoIP call with the other electronic device.

[0057] The access point may be provided based on satisfying additional conditions other than the conditions for initiating a VoIP call with another electronic device. The additional conditions may include the presence of data in a source language and a target language set in connection with the user identification information of the other electronic device. The identification information may be acquired based on performing optical character recognition (OCR) on data (or a screen) regarding the UI of the first application provided for the VoIP call with the other electronic device. Obtaining the identification information will be described in more detail with reference to FIG. 6.

[0058] Action 302 will be described in more detail with reference to FIG. 4.

[0059] In operation 303, at least one processor (210) may receive an input for the access point. For example, the input may indicate performing the translation service performed via the second application in connection with the VoIP call with the other electronic device.

[0060] Operations 302 and 303 of FIG. 3 are merely exemplary. Depending on the embodiment, operation 304 may be performed without performing operations 302 and 303. For example, at least one processor (210) may perform operation 304 without performing operations 302 and 303.

[0061] In operation 304, at least one processor (210) may display a second UI for the translation service together with a first UI for the VoIP call with the other electronic device based on the input received in operation 303. The second UI may be displayed together with the first UI to provide the translation service in relation to the VoIP call with the other electronic device. For example, the second UI may be at least partially overlapped on the first UI. The second UI at least partially overlapped on the first UI may be a pop-up window on the first UI. The second UI at least partially overlapped on the first UI may overlay or overlie at least a portion of the first UI. The first UI and the second UI may be displayed as a PIP (picture in picture). For example, the second UI may be displayed simultaneously with the first UI according to a split view. The above first UI and the above second UI can be displayed as PBP (picture by picture).

[0062] In operation 305, at least one processor (210) may identify whether a condition for starting to perform the translation service in relation to the VoIP call with the other electronic device is satisfied. For example, at least one processor (210) may postpone starting the translation service in relation to the VoIP call with the other electronic device until the condition is satisfied. For example, displaying the second UI together with the first UI may be performed based on the input in operation 303, but starting (or performing) (or executing) the translation service in relation to the VoIP call with the other electronic device may be postponed until the condition is satisfied. As a non-limiting example, satisfying the condition may indicate performing the fastest automatic speech recognition (ASR) of the user's utterance of the other electronic device. As a non-limiting example, satisfying the condition may indicate that data regarding the utterance of the user of the other electronic device is first received. As a non-limiting example, satisfying the above condition may indicate identifying a successful attempt at making the VoIP call with the other electronic device.

[0063] Action 305 will be described in more detail with reference to FIG. 12.

[0064] In operation 306, at least one processor (210) may perform the translation service during the VoIP call with the other electronic device based on identifying that the condition is satisfied.

[0065] In operation 307, at least one processor (210) may identify whether the VoIP call with the other electronic device is to be terminated while performing the translation service during the VoIP call with the other electronic device. For example, since the translation service is a service that requires termination upon termination of the VoIP call with the other electronic device, at least one processor (210) may perform operation 307.

[0066] As a non-limiting example, at step 308, at least one processor (210) may, while identifying that the VoIP call with the other electronic device is not terminated (or while identifying that the VoIP call with the other electronic device is ongoing), correct information related to speech during the VoIP call with the other electronic device based on the quality of the translation service during the VoIP call with the other electronic device. Step 308 may be optionally performed. Step 308 will be described in more detail with reference to FIG. 14.

[0067] At step 309, at least one processor (210) may terminate the translation service performed via the second application based on identifying that the VoIP call with the other electronic device has ended. Step 309 will be described in more detail with reference to FIG. 16.

[0068] Figure 4 is a flowchart illustrating a method for providing an access point for translation services during a VoIP call.

[0069] Referring to FIG. 4, in operation 401, at least one processor (210) may display a first UI of the first application for a VoIP call with another electronic device, which is performed using a communication circuit (230), using a display (240). For example, the display of the first UI may be performed according to the execution of the first application. The first UI is described with reference to FIG. 5.

[0070] Figure 5 illustrates an example of a first UI of a first application for making a VoIP call.

[0071] Referring to FIG. 5, at least one processor (210) may display a first UI (510), such as a state (500), while attempting to connect the VoIP call with the other electronic device. For example, the first UI (510) may include identification information (511) of a user of the other electronic device, an object (512) for a mute function, an object (513) for a speaker phone function, and an object (514) for stopping (or suspending) the attempt of the VoIP call. The identification information (511) may include a name (511-1) of the user of the other electronic device, a phone number (511-2) of the user of the other electronic device, and / or a profile picture (511-3) of the user of the other electronic device. As a non-limiting example, attempting to connect the VoIP call with the other electronic device through the first application and / or performing the VoIP call with the other electronic device through the first application may be unknown to the second application, unnoticeable to the second application, unrecognized by the second application, and / or unrecognized by the second application. For example, since performing the VoIP call with the other electronic device through the first application is unrecognized by the second application, at least one processor (210) may perform operation 402 described below. As a non-limiting example, displaying the first UI (510) of the first application may be unknown to the second application, unnoticeable to the second application, unrecognized by the second application, and / or unrecognized by the second application.For example, since displaying the first UI (510) of the first application is not recognized by the second application, at least one processor (210) may perform operation 402 described below.

[0072] For example, the first UI (510) of the first application may be displayed in full screen, as in state (500). For example, the first UI (510) of the first application may also be displayed according to a pop-up view, as in state (550). For example, the first UI (510) of the first application may be partially overlapped on the home screen (560) (or wallpaper (560)) (or desktop screen (560)) (or UI (560) of another application). For example, the first UI (510) of the first application may be displayed according to a split view. For example, the first UI (510) of the first application may be displayed simultaneously with another screen that does not overlap with the first UI (510) of the first application. For example, the first UI (510) of the first application displayed according to the split view may be arranged side by side with the other screen. The first UI (510) of the first application displayed according to the split view will be described with reference to FIG. 11.

[0073] Referring again to FIG. 4 , at step 402, at least one processor (210) may determine whether a condition for providing an access point for translation services during the VoIP call with the other electronic device is satisfied. For example, since the second application is unaware of step 401 and / or the VoIP call with the other electronic device, at least one processor (210) may perform step 402.

[0074] The above conditions can be set in various ways within the electronic device (200).

[0075] For example, the condition may include that an indication for establishing a VoIP call is triggered. For example, the indication may correspond to a command from the first application to set the audio mode to a communication audio mode for a VoIP call. For example, at least one processor (210) may identify that the condition is satisfied based on a command from the first application to set the audio mode to the communication audio mode for a VoIP call. The communication audio mode may be described as an audio mode that assigns volume control of a VoIP call to a volume button of the electronic device (200). The communication audio mode may be set when a call based on packet switching (e.g., a VoIP call) is established, unlike a call audio mode that is set when a call based on circuit switching (e.g., a telephone call) is established. As a non-limiting example, the audio mode (or the audio mode of the electronic device (200)) may be changed (switched) from a normal audio mode used (or enabled) within the electronic device (200) when ringing and a call is not established, to the communication audio mode, according to the command. As a non-limiting example, the audio mode may be changed from a ringing audio mode used when an incoming call is signaled, to the communication audio mode, according to the command.

[0076] For example, at least one processor (210) may identify that the above condition is satisfied based on further identifying that language data (e.g., data of a source language and a target language) (e.g., a language pack) for translating the user's speech of the other electronic device within the VoIP call and translating the user's speech of the electronic device (200) within the VoIP call are stored within the memory (220). The number of the source languages ​​may be 2 or more. The number of the target languages ​​may be 2 or more. As a non-limiting example, when the VoIP call is a group call conducted with three or more users, the number of the source languages ​​may be 2 or more, and the number of the target languages ​​may be 2 or more. Identifying whether the language data is stored within the memory (220) may be optionally performed. For example, at least one processor (210) may display a menu guiding acquisition of the language data from an external electronic device (e.g., a server) based on identifying that the language data is not stored in the memory (220). When an input received through the menu indicates acquisition of the language data, at least one processor (210) may identify that the condition is satisfied.

[0077] Identifying what language data is available for the translation service within the VoIP call with the other electronic device can be performed using identification information of the user of the other electronic device. The identification information of the user of the other electronic device can be obtained from the first UI of the first application for the VoIP call with the other electronic device. Obtaining the identification information of the user of the other electronic device from the first UI is described with reference to FIG. 5.

[0078] Referring to FIG. 5, at least one processor (210) may capture a first UI (510) based on a command from the first application (e.g., a command to set the audio mode to the communication audio mode). For example, at least one processor (210) may obtain the identification information of the user of the other electronic device by performing OCR (optical character recognition) on the captured first UI (510). For example, at least one processor (210) may identify the name (511-1) of the user of the other electronic device and / or the phone number (511-2) of the user of the other electronic device based on the OCR. As a non-limiting example, at least one processor (210) may obtain a profile picture (511-3) of the user of the other electronic device as the identification information of the user of the other electronic device by performing image analysis on the captured first UI (510).

[0079] Referring to FIG. 5, at least one processor (210) may capture a first UI (510) based on a touch input. For example, at least one processor (210) may capture the first UI (510) based on a touch input to a designated area (or a predetermined area) (or a reference area) (e.g., an object (116) for a VoIP call in FIG. 1). The first UI (510) may be displayed based on a touch input to an icon (or UI object) for a VoIP call.

[0080] As a non-limiting example, at least one processor (210) may obtain the identification information of the user of the other electronic device by cropping a portion of the captured first UI (510) and performing OCR on the cropped portion of the first UI (510). For example, at least one processor (210) may obtain information about the arrangement of the first UI of the first application from the memory (220) based on the command and / or the capture, and may use the information to identify an area of ​​the first UI (510) corresponding to the identification information (511), and crop the identified area of ​​the first UI (510), thereby obtaining the cropped portion of the first UI (510). For example, at least one processor (210) may obtain the identification information of the user of the other electronic device by performing OCR on the cropped portion of the first UI (510). The OCR may be performed with respect to a portion of the area of ​​the first UI (510) corresponding to the identification information (511). For example, when the identification information (511) includes one or more characters (e.g., emoticons and / or emoji graphical objects) that cannot be recognized through OCR, the OCR may be performed with respect to a portion of the area of ​​the first UI (510) that does not include the one or more characters.

[0081] For example, the identification information of the user of the other electronic device may or may not be obtained based on the OCR for the first UI (510) or the OCR for the cropped portion of the first UI (510). At least one processor (210) may perform an additional operation to identify the user of the other electronic device when the identification information of the user of the other electronic device is not obtained based on the OCR for the first UI (510) or the OCR for the cropped portion of the first UI (510). The additional operation is described with reference to FIG. 6.

[0082] FIG. 6 is a flowchart illustrating a method performed in relation to user identification information of another electronic device.

[0083] Referring to FIG. 6, in operation 601, at least one processor (210) may perform OCR with respect to a screen including a first UI (510), as described in FIG. 5.

[0084] In operation 602, at least one processor (210) may identify whether the identification information of the user of the other electronic device is obtained according to the performed OCR. For example, the at least one processor (210) may identify that the identification information of the user of the other electronic device is not obtained according to the OCR based on a failure in recognition of one or more characters through the OCR. For example, the at least one processor (210) may identify that the identification information of the user of the other electronic device is not obtained according to the OCR based on an error that occurs in relation to the OCR due to a country code in a phone number of the user of the other electronic device. For example, the at least one processor (210) may identify that the identification information of the user of the other electronic device is not obtained according to the OCR based on a failure in the OCR based on the identification information (511) of the user of the other electronic device within the first UI (510) that is obscured by another object. For example, at least one processor (210) may identify that the identification information of the user of the other electronic device is not obtained by the OCR based on a failure of the OCR according to the display length of the identification information (511) of the user of the other electronic device within the first UI (510).

[0085] For example, at least one processor (210) may perform operation 605 on the condition that the identification information of the user of the other electronic device is obtained according to the OCR, and otherwise perform operation 603.

[0086] In operation 603, at least one processor (210) may obtain the identification information of the user of the other electronic device by obtaining additional information using a speech of the user of the other electronic device (e.g., speech within a VoIP call with the other electronic device) based on the identification information of the user of the other electronic device that was not obtained by the OCR. For example, the at least one processor (210) may obtain, as the additional information, voice information of the user of the other electronic device within a previous call (e.g., including a VoIP call and / or a phone call) with the other electronic device, and / or language classification information of the speech, using the speech. For example, the at least one processor (210) may obtain, as the additional information, information on the percentage of phone numbers of the user of the other electronic device that have errors by the OCR (hereinafter, error percentage information).

[0087] For example, at least one processor (210) may obtain identification information estimated as the identification information of the user of the other electronic device from the contact information by applying the additional information to the contact information stored in the memory (220). For example, at least one processor (210) may obtain identification information estimated as the identification information of the user of the other electronic device from the contact information by performing scoring of the identification information in the contact information using the voice information, the language classification information, and the error rate information. For example, at least one processor (210) may obtain the estimated identification information as the identification information of the user of the other electronic device when the score of the estimated identification information is higher than a reference score. For example, at least one processor (210) may identify candidate identification information in the contact information estimated as the identification information of the user of the other electronic device when the score of the estimated identification information is lower than the reference score. For example, at least one processor (210) may display a pop-up window on the display (240) requesting selection of the identification information of the user of the other electronic device from among the candidate identification information, namely, first identification information, second identification information, and third identification information. For example, at least one processor (210) may obtain the identification information selected according to a user input from among the first identification information, the second identification information, and the third identification information in the pop-up window as the identification information of the user of the other electronic device.

[0088] In operation 605, at least one processor (210) may identify whether data (e.g., the language data) of the source language and the target language set in the contact information in relation to the identification information of the user of the other electronic device exist in the electronic device (200). The setting of the source language and the target language is described with reference to FIG. 7.

[0089] Figure 7 illustrates an example of setting source and target languages ​​within contact information related to identification information.

[0090] Referring to FIG. 7, the electronic device (200) may provide a UI (700) for setting a source language and a target language in association with identification information in contact information through the display (240). For example, the UI (700) may include a list (710) of source and target languages ​​set (or registered) in association with the identification information in the contact information. For example, the list (710) may include a source language (702) and a target language (703) associated with identification information (701), which is the identification information of the user of the other electronic device. The source language (702) may be a target language in association with the user of the electronic device (200). The target language (703) may be a source language in association with the user of the electronic device (200). For example, the list (710) may be updated according to a call (e.g., a phone call and / or a VoIP call) made by another user. For example, the UI (700) may further include an object (704) for setting the source language and the target language in conjunction with identification information.

[0091] Referring again to FIG. 6, at least one processor (210) may perform operation 607 on the condition that the data of the source language and the target language exist within the electronic device (200), and otherwise perform operation 608.

[0092] In operation 607, at least one processor (210) may identify that the condition is satisfied based on identifying that the data exists, and perform operation 403 of FIG. 4.

[0093] At operation 608, at least one processor (210) may identify that the condition is not satisfied based on identifying that the data does not exist, and perform operation 406 of FIG. 4.

[0094] Referring back to FIG. 4 , at step 402, at least one processor (210) may determine that the condition is satisfied based on further identifying that there is a history of calls (e.g., including VoIP calls and phone calls) with a user of the other electronic device. For example, if there is no history of calls with the user of the other electronic device, the at least one processor (210) may determine that the condition is not satisfied. The at least one processor (210) may determine that the condition is satisfied based on further identifying a state of the electronic device (200) being logged in with an authenticated user account of the user of the electronic device (200). For example, the at least one processor (210) may determine that the condition is satisfied based on further identifying data indicating that the user of the electronic device (200) agrees to the translation service during the VoIP call. For example, at least one processor (210) may identify that the above condition is not satisfied based on identifying the electronic device (200) in guest mode. For example, at least one processor (210) may identify that the above condition is not satisfied based on identifying the status of the electronic device (200) logged in with a kid user account (or an authenticated kid user account). However, the present invention is not limited thereto.

[0095] For example, at least one processor (210) may perform operation 403 based on identifying (or determining) that the condition is satisfied. For example, at least one processor (210) may perform operation 406 based on identifying (or determining) that the condition is not satisfied.

[0096] In operation 403, at least one processor (210) may, based on a determination that the above condition is satisfied, provide an access point for the translation service during the VoIP call with the other electronic device. The access point may be provided visually. The access point is described with reference to FIGS. 8A and 9.

[0097] Figure 8a illustrates an example of UI objects within a quick panel for translation services during a VoIP call.

[0098] Referring to FIG. 8A, at least one processor (210) may receive a drag input (801) moving away from an edge area (800) (or edge) (or top area (800)) of the display (240) while displaying a UI (510) using the display (240), such as a state (500). Based on the drag input (801), the at least one processor (210) may change or transition the state (500) of the display (240) to the state (810) of the display (240).

[0099] Within the state (810), at least one processor (210) may display a quick panel (820) appearing from the edge region (800) using the display (240). For example, at least one processor (210) may add a UI object (830) to the quick panel (820), such as within the state (810), based on identifying that the condition is satisfied (e.g., the command from the first application) in operation 402. The quick panel (820) within the state (810) may be different from the quick panel (820) within the state (850) that is displayed based on identifying that the condition is not satisfied. The quick panel (820) in the state (810) may include a UI object (830) and a UI object (845) (e.g., a UI object for setting a mode of a microphone (250) for a VoIP call) based on identifying that the above condition is satisfied, and the quick panel (820) in the state (850) may include the UI object (845) among the UI objects (830) and (845) based on identifying that the VoIP call with the other electronic device does not satisfy the above condition. For example, since the quick panel (820) in the state (850) does not include the UI object (830), the size of the UI object (845) in the quick panel (820) in the state (850) may be larger than the size of the UI object (845) in the quick panel (820) in the state (810). As a non-limiting example, the quick panel (820) within the state (850) may be displayed based on a user setting (e.g., a global setting of the electronic device (200)) that disables the service of live translation (or real-time translation), and the quick panel (820) within the state (810) may be displayed based on a user setting that activates the service of real-time translation. The UI for the user setting is described with reference to FIG. 8B.

[0100] Figure 8b illustrates a UI for user settings of the translation service.

[0101] Referring to FIG. 8B, at least one processor (210) may display a UI (890) for user settings for a call on the display (240). The UI (890) may include an object (898) (or a UI object (898)) for a translation service (or a real-time translation service) related to the call. The at least one processor (210) may display a UI (891) on the display (240) based on an input (899) for the object (898). For example, the at least one processor (210) may change the UI displayed on the display (240) from the UI (890) to the UI (891) based on the input (899). The UI (891) may include text (892) including guidance related to the translation service (or the real-time translation service). The UI (891) may include a list (893) of applications that support a translation service (or a real-time translation service). The list (893) may include one or more objects, each representing one or more applications that support a real-time translation service, are installed in the electronic device (200), and provide a VoIP call function. For example, the list (893) may include an object (893-1) representing an application A, which is one of the one or more applications, an object (893-2) representing an application B, which is another of the one or more applications, and an object (893-3) representing an application C, which is still another of the one or more applications. The object (893-1) may include a UI element (894-1) that activates or deactivates a real-time translation service in relation to a VoIP call performed via application A. The object (893-2) may include a UI element (894-2) that enables or disables the service of real-time translation in connection with a VoIP call conducted via application B.The object (893-3) may include a UI element (894-3) that enables or disables the service of real-time translation in connection with a VoIP call performed via application C.

[0102] For example, if application A is the first application in the above description and the UI element (894-1) indicates that the service of real-time translation is activated, at least one processor (210) may display the quick panel (820) in the state (810). For example, if application A is the first application in the above description and the UI element (894-1) indicates that the service of real-time translation is deactivated, at least one processor (210) may display the quick panel (820) in the state (850).

[0103] Referring back to FIG. 8A, the UI object (830) may be configured to receive input (or user input). For example, the UI object (830) may be available for the translation service during the VoIP call with the other electronic device. Within the state (810), the UI object (830) may be within a first state indicating that the translation service during the VoIP call with the other electronic device is disabled. For example, at least one processor (210) may change or transition the state (810) of the display (240) to the state (840) of the display (240) based on a touch input (835) to the UI object (830) within the first state.

[0104] Within the state (840), at least one processor (210) may display a quick panel (820) using the display (240), which includes a UI object (830) within a second state changed from the first state. The UI object (830) within the second state may indicate that the translation service is activated during the VoIP call with the other electronic device. For example, at least one processor (210) may perform operation 405 based on a touch input (835).

[0105] For example, at least one processor (210) may display a UI object for the translation service during a VoIP call with the other electronic device together with the first UI. The UI object displayed together with the first UI is described with reference to FIG. 9.

[0106] Figure 9 illustrates an example of a UI object for a translation service during a VoIP call displayed with the first UI.

[0107] Referring to FIG. 9, at least one processor (210) may, based on identifying that the condition is satisfied in operation 402 (e.g., the command from the first application), display a UI object (900) together with the first UI (510) using the display (240). The UI object (900) may be configured to receive an input (or user input). The UI object (900) may, for example, be overlapped or floated on the first UI (510). As a non-limiting example, the UI object (900) may be displayed at a predetermined location. As a non-limiting example, the UI object (900) may differ from the UI object (830) in that it is displayed under the condition that a call history of the user of the other electronic device exists. As a non-limiting example, the UI object (900) may be displayed according to a user setting via the UI (891) in FIG. 8b.

[0108] As a non-limiting example, the UI object (900) may be moved on the first UI (510) in response to a touch input (e.g., a drag input) to the UI object (900). For example, the UI object (900) may change a state of the UI object (900) in response to a touch input (e.g., a single tap input) to the UI object (900), similar to the change in state of the UI object (830). For example, at least one processor (210) may perform operation 405 based on a touch input (e.g., a single tap input) to the UI object (900).

[0109] Referring again to FIG. 4, at operation 405, at least one processor (210) may, based on receiving input for the access point, display the second UI of the second application for the translation service, together with the first UI, using the display (240). The second UI may be superimposed on the first UI. The second UI superimposed on the first UI is described with reference to FIG. 10.

[0110] Figure 10 illustrates a second UI overlaid on a first UI.

[0111] Referring to FIG. 10, as in state (1000), at least one processor (210) may display a first UI (510) and a second UI (1020) overlaid on at least a portion of the first UI (510) using the display (240). For example, at least a portion of the second UI (1020) overlaid on at least a portion of the first UI (510) may be translucent. For example, at least a portion of the first UI (510) may be visible through the second UI (1020). For example, at least one processor (210) may recognize the layout of the first UI (510) of the first application based on the input in operation 404. For example, at least one processor (210) may identify an area (1021) in which a second UI (1020) is to be displayed based on the recognition of the layout, and display the second UI (1020) superimposed on the first UI (510) within the area (1021). As a non-limiting example, the second UI (1020) may include text (1024) guiding the start condition of the translation service during the VoIP call with the other electronic device within a translation window (1025) of the second UI (1020). The text (1024) will be described within the description of FIG. 12.

[0112] For example, the first UI (510) may include a second area distinct from the first area including a blank area and / or objects (e.g., object (512), object (513), and object (514)). For example, the second UI partially overlapping the first UI (510) may be displayed on the blank area and / or the second area.

[0113] For example, as in state (1050), at least one processor (210) can display a second UI (1022) partially overlapping the first UI (510) within a blank area (1023) (or a second area (1023) different from the first area (1026) including objects (object (512), object (513), and object (514)). For example, at least one processor (210) can identify the blank area (1023) of the first UI (510) where the second UI (1022) is to be displayed, based on the recognition of the layout of the first UI (510), and display the second UI (1022) overlapping on the blank area (1023) of the first UI (510). As a non-limiting example, the second UI (1022) may include text (1024) within a translation window (1027) of the second UI (1022) that guides the initiation conditions of the translation service during the VoIP call with the other electronic device. The text (1024) will be described within the description of FIG. 12.

[0114] For example, the first UI (510) may be displayed according to a split view. For example, at least one processor (210) may simultaneously display the first UI (510) and the third UI side by side according to the split view. The second UI may be overlapped on the first UI (510) displayed according to the split view. The second UI overlapped on the first UI (510) displayed according to the split view is described with reference to FIG. 11.

[0115] Figure 11 illustrates a second UI overlaid on a first UI displayed according to a split view.

[0116] Referring to FIG. 11, at least one processor (210) can simultaneously display a first UI (510) and a third UI (1130) of a third application according to the split view, as in state (1100). For example, the first UI (510) and the third UI (1130) can be arranged side by side.

[0117] Within the state (1100), the second UI (1120) (e.g., the second UI (1020)) for the translation service within the VoIP call with the other electronic device may be superimposed on at least a portion of the first UI (510) displayed together with the third UI (1130) according to the split view. For example, the second UI (1120) may be displayed according to the input received in operation 404. For example, the second UI (1120) may be superimposed on at least a portion of the first UI (510) within a foreground execution state of the first application in which the first UI (510) among the first UI (510) and the third UI (1130) is selected according to a user input. For example, the second UI (1120) may be overlapped on at least a portion of the first UI (510) within a background execution state of the first application, in which the third UI (1130) among the first UI (510) and the third UI (1130) is selected according to a user input. For example, at least one processor (210) may display the second UI (1120) overlapped on at least a portion of the first UI (510) while the third application is in a foreground execution state and the first application is in a background execution state, based on execution information (e.g., running task app info) of applications of the electronic device (200) (e.g., indicating that the first UI (510) is visible).

[0118] The arrangement of the second UI (1120) in FIG. 11 is merely exemplary. The second UI (1120) may be displayed together with the first UI (510) and the third UI (1130) at a predetermined location, independently of the locations of the first UI (510) and the third UI (1130). The second UI (1120) displayed at the predetermined location may at least partially overlap at least a portion of the first UI (510) and / or at least a portion of the third UI (1130). The second UI (1120) displayed at the predetermined location may at least partially overlap at least a portion of the third UI (1130).

[0119] Referring again to FIG. 4 , at step 406, at least one processor (210) may determine not to perform the translation service in connection with the VoIP call with the other electronic device based on the identification in step 402 that the condition is not satisfied. For example, at least one processor (210) may refrain from providing the access point using the display (240).

[0120] As described above, the electronic device (200) can execute the second application in conjunction with the first application by identifying whether the state of the first application for the VoIP call with the other electronic device satisfies the condition (e.g., the command from the first application).

[0121] Since the translation service during the VoIP call with the other electronic device includes at least temporarily storing the utterance of the user of the other electronic device and at least temporarily storing text corresponding to the utterance of the user of the other electronic device, performing the translation service during the VoIP call with the other electronic device may (legally) require notifying the user of the other electronic device. For example, the translation service during the VoIP call with the other electronic device may require legal notice.

[0122] For example, since the timing at which the first application issues the command (e.g., the command to set the audio mode to the communication audio mode) may not be recognized by the second application, at least one processor (210) may not initiate the translation service during the VoIP call with the other electronic device based on displaying the second UI together with the first UI according to operation 405. For example, at least one processor (210) may postpone performing the translation service during the VoIP call with the other electronic device until it identifies a successful attempt to connect the VoIP call with the other electronic device. For example, at least one processor (210) may identify whether an utterance of the user of the other electronic device is caused within the VoIP call (e.g., operation 305) to identify a successful attempt to connect the VoIP call with the other electronic device. This operation is described with reference to FIG. 12.

[0123] Figure 12 is a flowchart illustrating a method for initiating a translation service during a VoIP call.

[0124] Referring to FIG. 12, in operation 1201, at least one processor (210) may display the second UI together with the first UI (e.g., the first UI (510)). For example, operation 1201 may correspond to operation 405 (or operation 304).

[0125] In operation 1202, at least one processor (210) may identify whether an earliest ASR (automatic speech recognition or automotive speech recognition) (or ASR operation) of an utterance (e.g., a first utterance) of the user of the other electronic device that was first received using the communication circuit (230) from the other electronic device is performed. The ASR may be performed for the translation service. The ASR may be performed during a VoIP call with the other electronic device. For example, the ASR may be described as an ASR performed using a packet from the other electronic device. For example, the ASR may be distinguished from another ASR of audio (e.g., an utterance of the user of the electronic device (200)) received via the microphone (250) of the electronic device (200).

[0126] At least one processor (210) can perform operation 1203 based on performing the fastest ASR of the first utterance, and can perform operation 1204 based on not performing the fastest ASR of the first utterance.

[0127] In operation 1203, at least one processor (210) may display, using a display (240), information indicating that the translation service of the VoIP call with the other electronic device has started, based on performing the fastest ASR of the first utterance, within the second UI displayed together with the first UI. The information is described with reference to FIG. 13.

[0128] Figure 13 illustrates an example of information within a second UI displayed in connection with initiating a translation service during a VoIP call.

[0129] Referring to FIG. 13, at least one processor (210) may display text (1330) within a second UI (1310) (e.g., second UI (1020)) overlaid on the first UI (510), as in state (1300), based on performing the fastest ASR of the first utterance. The text (1330) may include information indicating that the translation service during the VoIP call with the other electronic device is started. For example, at least one processor (210) may start the translation service during the VoIP call with the other electronic device based on performing the fastest ASR of the first utterance and displaying the text (1330). For example, the text (1330) may be displayed within a translation window (1320) (e.g., translation window (1025)) of the second UI (1310).

[0130] For example, at least one processor (210) may transmit a signal to the other electronic device using the communication circuit (230) that causes the other electronic device to output a voice signal (e.g., a digital voice signal) (e.g., a voice signal of a digital assistant) corresponding to the text (1330) based on performing the fastest ASR of the first utterance and displaying the text (1330). For example, the user's utterance of the other electronic device in connection with the VoIP call may be received from the other electronic device while the voice signal is being output in connection with the other electronic device. For example, the at least one processor (210) may refrain from providing a translation service for the utterance based on the utterance that ends before the output of the voice signal is completed. For example, the at least one processor (210) may provide a translation service for the utterance based on the utterance that is maintained after the output of the voice signal is completed.

[0131] Referring back to FIG. 12 , at operation 1204, at least one processor (210) may postpone performing the translation service during the VoIP call with the other electronic device until the earliest ASR of the first utterance is performed. For example, at least one processor (210) may postpone performing the translation service during the VoIP call with the other electronic device independently of displaying the second UI together with the first UI (e.g., the first UI (510)). For example, at least one processor (210) may refrain from performing the translation service for an utterance of the user of the electronic device (e.g., received via the microphone (250)) within the VoIP call with the other electronic device before performing the earliest ASR of the first utterance. For example, at least one processor (210) may refrain from displaying, within the second UI displayed together with the first UI, the result of a translation of an utterance of the user of the electronic device within the VoIP call with the other electronic device before performing the fastest ASR of the first utterance. For example, at least one processor (210) may display, within the second UI displayed together with the first UI, information (or notification information) indicating that the translation service during the VoIP call with the other electronic device starts upon the start of an utterance of the user of the other electronic device until the fastest ASR of the first utterance is performed. For example, the information may indicate that a translation service for the VoIP call is being performed. The information is described with reference to FIG. 13.

[0132] Referring to FIG. 13, at least one processor (210) may display text (1340) within a second UI (1310) superimposed on the first UI (510), such as a state (1350), before performing the fastest ASR of the first utterance. For example, the text (1340) may include information indicating that the translation service during the VoIP call with the other electronic device starts upon the start of the utterance of the user of the other electronic device. For example, the text (1340) may be displayed within a translation window (1320) of the second UI (1310). The text (1340) may provide information about a service provider of the translation service. The text (1340) may indicate that the translation service is provided and the utterance of the user of the other electronic device is recorded.

[0133] For example, a condition for starting the translation service during a VoIP call via the second application (e.g., performing the fastest ASR of the first utterance) may be different from a condition for starting the translation service during a phone call via the second application. For example, at least one processor (210) may start the translation service during the first call performed via the second application based on performing the fastest ASR of the utterance of the user of the other electronic device that is first received from the other electronic device in relation to the first call (e.g., VoIP call) with the other electronic device, while displaying the second UI for the translation service together with the first UI for the first call (e.g., VoIP call) with the other electronic device using the display (240) according to packet exchange. For example, at least one processor (210) may start a translation service during a second call performed through the second application based on receiving a signal (e.g., “200 OK”) from the other electronic device, using the communication circuit (230), indicating acceptance of a call connection request in relation to the second call while displaying the second UI together with the third UI for a second call (e.g., a phone call) with the other electronic device using the display (240) according to a circuit exchange. The second UI may be included within the third UI. The second UI and the third UI may also be displayed as one UI (or a single UI).

[0134] As described above, the electronic device (200) can provide a translation service performed according to the recognition of the user of the other party's electronic device by performing the translation service during the VoIP call under the condition of performing the fastest ASR of the first utterance of the user of the other party's electronic device.

[0135] Figure 14 is a flowchart illustrating a method for correcting information related to speech during a VoIP call while performing a translation service during the VoIP call.

[0136] In operation 1401, at least one processor (210) may perform the translation service within the VoIP call with the other electronic device based on satisfying a condition for starting the translation service within the VoIP call with the other electronic device (e.g., operation 305 or operation 1202). For example, operation 1401 may correspond to operation 306 or operation 1203.

[0137] In operation 1402, at least one processor (210) may identify a quality of the translation service during the VoIP call with the other electronic device while performing the translation service during the VoIP call with the other electronic device. For example, at least one processor (210) may perform operation 1402 by identifying a quality of communication with the other electronic device via the communication circuit (230). For example, at least one processor (210) may perform operation 1402 by identifying a quality of ASR, STT (speech to text), and / or translation performed in relation to the translation service during the VoIP call with the other electronic device.

[0138] In operation 1403, at least one processor (210) may identify whether the quality identified in operation 1402 is higher than a reference quality. For example, at least one processor (210) may repeatedly perform operations 1402 and 1403 under a condition that the quality is higher than the reference quality. For example, at least one processor (210) may perform operation 1404 under a condition that the quality is lower than the reference quality (or under a condition that the quality is lower than or equal to the reference quality).

[0139] In operation 1404, at least one processor (210) may correct information related to speech during the VoIP call with the other electronic device based on the quality being lower than the reference quality. The correction is described with reference to FIG. 15.

[0140] Figure 15 illustrates an example of a trained model used to correct information related to speech during a VoIP call.

[0141] Referring to FIG. 15, the electronic device (200) can perform operation 1404 using a trained model (1500). The model (1500) can be described as an artificial intelligence (AI) model associated with the electronic device (200). For example, the model (1500) can be included within the electronic device (200). For example, the model (1500) can be included within an external electronic device connectable to the electronic device (200).

[0142] For example, the model (1500) may include a language model (1501) and / or a reaction rate-based inference model (1502).

[0143] For example, at least one processor (210) may obtain input data (1510) including a result of ASR of an utterance (e.g., "Hello?") during the VoIP call with the other electronic device based on the quality being lower than the reference quality according to operation 1403. For example, the input data (1510) may be obtained based on the utterance received from the other electronic device under the condition that the response speed of the VoIP call is lower than the reference speed. For example, the result in the input data (1510) may include an utterance having an error (e.g., "Hello?"). For example, at least one processor (210) may provide the input data (1510) to the model (1500). For example, at least one processor (210) may use the model (1500) to infer the intention of a user (e.g., the user of the other electronic device) represented by the input data (1510) and obtain output data (1520) from the model (1500) based on the inference. For example, the output data (1520) may include information (e.g., “Hello?”) corresponding to an utterance inferred from the result (e.g., “Hello?”) in the input data (1510). For example, at least one processor (210) may use the model (1500) to correct information related to an utterance during a VoIP call with the other electronic device.

[0144] As a non-limiting example, the reliability of the output data (1520) may be lower than a reference reliability. For example, at least one processor (210) may display user guidance using the display (240) based on the reliability of the output data (1520) being lower than the reference reliability. For example, the user guidance may indicate re-performance of the utterance. For example, the user guidance may also indicate that the communication connection with the other electronic device is not in a good condition.

[0145] Figure 16 is a flowchart illustrating a method for terminating a translation service during a VoIP call.

[0146] Referring to FIG. 16, in operation 1601, at least one processor (210) may identify the termination of the VoIP call with the other electronic device. For example, at least one processor (210) may identify the termination of the VoIP call with the other electronic device based on another command to set the audio mode to a different audio mode distinct from the communication audio mode. For example, the other command may be triggered based on an input to an object within the first UI (e.g., the first UI (510)). The input is described with reference to FIG. 17.

[0147] Figure 17 illustrates an example of a second UI that is interrupted to terminate a translation service during a VoIP call.

[0148] Referring to FIG. 17, at least one processor (210) may display a second UI (1710) (e.g., second UI (1020)) at least partially overlapping at least a portion of the first UI (510) while performing the translation service during the VoIP call with the other electronic device. For example, while displaying the second UI (1710) together with the first UI (510), the at least one processor (210) may receive an input (1730) for an object (1720) within the first UI (510) for terminating the VoIP call with the other electronic device. For example, since receiving the input (1730) is performed through the first application, receiving the input (1730) may not be recognized by the second application. The first application may cause another command according to the input (1730). For example, receiving an input (1730) may not be recognized by the second application, but at least one processor (210) may identify the termination of the VoIP call with the other electronic device based on the other command according to the input (1730).

[0149] Referring back to FIG. 16, in operation 1602, at least one processor (210) may store information related to the translation service during the VoIP call with the other electronic device based on operation 1601. For example, at least one processor (210) may store the source language and the target language used within the translation service in association with the identification information of the user of the other electronic device within the contact information. For example, at least one processor (210) may store a portion of the user's speech within the translation service in association with the identification information of the user of the other electronic device within the contact information. For example, at least one processor (210) may also store information of an application providing the VoIP call (e.g., information of the first application) in association with the identification information of the user of the other electronic device within the contact information. The information of the application may be captured according to the capturing described with reference to FIG. 5. For example, the information stored in operation 1602 may be used for operation 301, operation 402, and / or operation 603.

[0150] In operation 1603, at least one processor (210) may terminate the translation service during the VoIP call with the other electronic device after performing operation 1602. For example, at least one processor (210) may terminate the display of the second UI of the second application.

[0151] The operations of the electronic device (200) described above may also be performed by the electronic device (1801) of FIG. 18.

[0152] FIG. 18 is a block diagram of an electronic device (1801) within a network environment (1800) according to various embodiments. Referring to FIG. 18, in the network environment (1800), the electronic device (1801) may communicate with the electronic device (1802) via a first network (1898) (e.g., a short-range wireless communication network), or may communicate with at least one of the electronic device (1804) or the server (1808) via a second network (1899) (e.g., a long-range wireless communication network). In one embodiment, the electronic device (1801) may communicate with the electronic device (1804) via the server (1808). According to one embodiment, the electronic device (1801) may include a processor (1820), a memory (1830), an input module (1850), an audio output module (1855), a display module (1860), an audio module (1870), a sensor module (1876), an interface (1877), a connection terminal (1878), a haptic module (1879), a camera module (1880), a power management module (1888), a battery (1889), a communication module (1890), a subscriber identification module (1896), or an antenna module (1897). In some embodiments, the electronic device (1801) may omit at least one of these components (e.g., the connection terminal (1878)), or may have one or more other components added. In some embodiments, some of these components (e.g., sensor module (1876), camera module (1880), or antenna module (1897)) may be integrated into a single component (e.g., display module (1860)).

[0153] The processor (1820) may control at least one other component (e.g., a hardware or software component) of the electronic device (1801) connected to the processor (1820) by executing, for example, software (e.g., a program (1840)), and may perform various data processing or operations. According to one embodiment, as at least a part of the data processing or operations, the processor (1820) may store commands or data received from other components (e.g., a sensor module (1876) or a communication module (1890)) in a volatile memory (1832), process the commands or data stored in the volatile memory (1832), and store result data in a non-volatile memory (1834). According to one embodiment, the processor (1820) may include a main processor (1821) (e.g., a central processing unit or an application processor) or an auxiliary processor (1823) (e.g., a graphics processing unit, a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor) that can operate independently or together with the main processor (1821). For example, when the electronic device (1801) includes the main processor (1821) and the auxiliary processor (1823), the auxiliary processor (1823) may be configured to use less power than the main processor (1821) or to be specialized for a given function. The auxiliary processor (1823) may be implemented separately from the main processor (1821) or as a part thereof.

[0154] The auxiliary processor (1823) may control at least a portion of functions or states associated with at least one component (e.g., the display module (1860), the sensor module (1876), or the communication module (1890)) of the electronic device (1801), for example, on behalf of the main processor (1821) while the main processor (1821) is in an inactive (e.g., sleep) state, or together with the main processor (1821) while the main processor (1821) is in an active (e.g., application execution) state. In one embodiment, the auxiliary processor (1823) (e.g., an image signal processor or a communication processor) may be implemented as a part of another functionally related component (e.g., a camera module (1880) or a communication module (1890)). In one embodiment, the auxiliary processor (1823) (e.g., a neural network processing unit) may include a hardware structure specialized for processing artificial intelligence models. The artificial intelligence models may be generated through machine learning. This learning can be performed, for example, on the electronic device (1801) where the artificial intelligence model is executed, or can be performed through a separate server (e.g., server (1808)). The learning algorithm can include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but is not limited to the examples described above. The artificial intelligence model can include multiple artificial neural network layers.The artificial neural network may be one of a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), a deep Q-network, or a combination of two or more of the above, but is not limited to the examples described above. In addition to, or alternatively to, a hardware structure, an artificial intelligence model may include a software structure.

[0155] The memory (1830) can store various data used by at least one component (e.g., the processor (1820) or the sensor module (1876)) of the electronic device (1801). The data can include, for example, software (e.g., the program (1840)) and input data or output data for commands related thereto. The memory (1830) can include volatile memory (1832) or non-volatile memory (1834).

[0156] The program (1840) may be stored as software in memory (1830) and may include, for example, an operating system (1842), middleware (1844), or an application (1846).

[0157] The input module (1850) can receive commands or data to be used in a component of the electronic device (1801) (e.g., a processor (1820)) from an external source (e.g., a user) of the electronic device (1801). The input module (1850) can include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

[0158] The audio output module (1855) can output audio signals to the outside of the electronic device (1801). The audio output module (1855) can include, for example, a speaker or a receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. In one embodiment, the receiver can be implemented separately from the speaker or as part of the speaker.

[0159] The display module (1860) can visually provide information to an external party (e.g., a user) of the electronic device (1801). The display module (1860) may include, for example, a display, a holographic device, or a projector and a control circuit for controlling the device. In one embodiment, the display module (1860) may include a touch sensor configured to detect a touch, or a pressure sensor configured to measure the intensity of a force generated by the touch.

[0160] The audio module (1870) can convert sound into an electrical signal, or vice versa. According to one embodiment, the audio module (1870) can acquire sound through the input module (1850), output sound through the sound output module (1855), or an external electronic device (e.g., electronic device (1802)) (e.g., speaker or headphone) directly or wirelessly connected to the electronic device (1801).

[0161] The sensor module (1876) can detect the operating status (e.g., power or temperature) of the electronic device (1801) or the external environmental status (e.g., user status) and generate an electrical signal or data value corresponding to the detected status. According to one embodiment, the sensor module (1876) can include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, or an illuminance sensor.

[0162] The interface (1877) may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device (1801) with an external electronic device (e.g., the electronic device (1802)). In one embodiment, the interface (1877) may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

[0163] The connection terminal (1878) may include a connector through which the electronic device (1801) may be physically connected to an external electronic device (e.g., the electronic device (1802)). In one embodiment, the connection terminal (1878) may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (e.g., a headphone connector).

[0164] The haptic module (1879) can convert electrical signals into mechanical stimuli (e.g., vibration or movement) or electrical stimuli that a user can perceive through tactile or kinesthetic sensations. In one embodiment, the haptic module (1879) may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

[0165] The camera module (1880) can capture still images and moving images. In one embodiment, the camera module (1880) may include one or more lenses, image sensors, image signal processors, or flashes.

[0166] The power management module (1888) can manage the power supplied to the electronic device (1801). According to one embodiment, the power management module (1888) can be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

[0167] A battery (1889) may power at least one component of the electronic device (1801). In one embodiment, the battery (1889) may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

[0168] The communication module (1890) may support the establishment of a direct (e.g., wired) communication channel or a wireless communication channel between the electronic device (1801) and an external electronic device (e.g., electronic device (1802), electronic device (1804), or server (1808)), and the performance of communication through the established communication channel. The communication module (1890) may operate independently from the processor (1820) (e.g., application processor) and may include one or more communication processors that support direct (e.g., wired) communication or wireless communication. According to one embodiment, the communication module (1890) may include a wireless communication module (1892) (e.g., a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module (1894) (e.g., a local area network (LAN) communication module, or a power line communication module). Any of these communication modules may communicate with an external electronic device (1804) via a first network (1898) (e.g., a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network (1899) (e.g., a long-range communication network such as a legacy cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (e.g., a local area network or a wide area network)). These various types of communication modules may be integrated into a single component (e.g., a single chip) or implemented as multiple separate components (e.g., multiple chips). The wireless communication module (1892) may use subscriber information (e.g., an international mobile subscriber identity (IMSI)) stored in the subscriber identification module (1896) to identify or authenticate the electronic device (1801) within a communication network such as the first network (1898) or the second network (1899).

[0169] The wireless communication module (1892) can support 5G networks and next-generation communication technologies following the 4G network, such as NR access technology (new radio access technology). NR access technology can support high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimizing terminal power and connecting multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency communications)). The wireless communication module (1892) can support, for example, a high-frequency band (e.g., mmWave band) to achieve a high data transmission rate. The wireless communication module (1892) may support various technologies for securing performance in high-frequency bands, such as beamforming, massive multiple-input and multiple-output (MIMO), full dimensional MIMO (FD-MIMO), array antenna, analog beam-forming, or large scale antenna. The wireless communication module (1892) may support various requirements specified in the electronic device (1801), an external electronic device (e.g., the electronic device (1804)), or a network system (e.g., the second network (1899)). According to one embodiment, the wireless communication module (1892) can support a peak data rate (e.g., 20 Gbps or more) for eMBB implementation, a loss coverage (e.g., 164 dB or less) for mMTC implementation, or a U-plane latency (e.g., 0.5 ms or less for downlink (DL) and uplink (UL), or 1 ms or less for round trip) for URLLC implementation.

[0170] The antenna module (1897) can transmit or receive signals or power to or from an external device (e.g., an external electronic device). In one embodiment, the antenna module (1897) may include an antenna including a radiator formed of a conductor or a conductive pattern formed on a substrate (e.g., a PCB). In one embodiment, the antenna module (1897) may include a plurality of antennas (e.g., an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network, such as the first network (1898) or the second network (1899), may be selected from the plurality of antennas by, for example, the communication module (1890). A signal or power may be transmitted or received between the communication module (1890) and the external electronic device via the at least one selected antenna. In some embodiments, in addition to the radiator, another component (e.g., a radio frequency integrated circuit (RFIC)) may be additionally formed as a part of the antenna module (1897).

[0171] According to various embodiments, the antenna module (1897) may form a mmWave antenna module. In one embodiment, the mmWave antenna module may include a printed circuit board, an RFIC disposed on or adjacent a first side (e.g., a bottom side) of the printed circuit board and capable of supporting a designated high frequency band (e.g., a mmWave band), and a plurality of antennas (e.g., an array antenna) disposed on or adjacent a second side (e.g., a top side or a side side) of the printed circuit board and capable of transmitting or receiving signals in the designated high frequency band.

[0172] At least some of the above components can be interconnected and exchange signals (e.g., commands or data) with each other via a communication method between peripheral devices (e.g., a bus, GPIO (general purpose input and output), SPI (serial peripheral interface), or MIPI (mobile industry processor interface)).

[0173] According to one embodiment, commands or data may be transmitted or received between the electronic device (1801) and an external electronic device (1804) via a server (1808) connected to a second network (1899). Each of the external electronic devices (1802 or 1804) may be the same or a different type of device as the electronic device (1801). According to one embodiment, all or part of the operations executed in the electronic device (1801) may be executed in one or more of the external electronic devices (1802, 1804, or 1808). For example, when the electronic device (1801) is to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device (1801) may, instead of or in addition to executing the function or service itself, request one or more external electronic devices to perform the function or at least a part of the service. One or more external electronic devices that receive the request may execute at least a portion of the requested function or service, or an additional function or service related to the request, and transmit the result of the execution to the electronic device (1801). The electronic device (1801) may process the result as is or additionally and provide it as at least a portion of a response to the request. For this purpose, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used, for example. The electronic device (1801) may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device (1804) may include an Internet of Things (IoT) device. The server (1808) may be an intelligent server utilizing machine learning and / or a neural network.According to one embodiment, an external electronic device (1804) or server (1808) may be included within the second network (1899). The electronic device (1801) may be applied to intelligent services (e.g., smart homes, smart cities, smart cars, or healthcare) based on 5G communication technology and IoT-related technology.

[0174] As described above, an electronic device (e.g., electronic device (200)) may include a display (e.g., display (240)), a communication circuit (e.g., communication circuit (230)), at least one processor (e.g., at least one processor (210)) including a processing circuit, and a memory (e.g., memory (220)) that stores instructions and includes one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to execute the first application to display, using the display, a first user interface (UI) of a first application for a voice over internet protocol (VoIP) call with another electronic device performed using the communication circuit; to display, using the display, a UI object for receiving an input for performing a translation service during the VoIP call with the other electronic device based on an indication for establishing the VoIP call, together with the first UI; to receive an input for the UI object; and to display, using the display, together with the first UI, a second UI of a second application for the translation service based on the input.

[0175] For example, the indication may correspond to a command from the first application to set the audio mode to a communication audio mode.

[0176] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to capture a screen including the first UI, perform optical character recognition (OCR) on at least a portion of the captured screen based on the command, thereby obtaining identification information of a user of the other electronic device included in the first UI, identifying whether data of a source language and a target language set in contact information exist in the electronic device in relation to the identification information, and displaying the UI object together with the first UI based on the identification that the data exists in the electronic device, and refraining from displaying the UI object based on the identification that the data does not exist in the electronic device.

[0177] For example, the UI object may be overlapped or floated on a portion of the first UI.

[0178] For example, the instructions, when individually or collectively executed by the at least one processor, may further cause the electronic device to add another UI object for a translation service during a VoIP call with the other electronic device provided by the second application to a quick panel displayed on the display based on a drag input moving away from an edge area of ​​the display, and to display the second UI, together with the first UI, using the display based on an input to the other UI object in the quick panel.

[0179] For example, the second UI may overlap at least a portion of the first UI, and at least a portion of the second UI may be translucent.

[0180] For example, the second UI may be overlapped on a second area within the first UI that is different from a first area within the first UI that includes UI objects, and the second UI may be opaque.

[0181] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display the second UI superimposed on the first UI while simultaneously displaying the first UI and the third UI of the third application using the display according to a split view, and to maintain the display of the second UI superimposed on the first UI based on execution information of applications of the electronic device while the third application is in a foreground execution state and the first application is in a background execution state.

[0182] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to perform automatic speech recognition (ASR) of a user's utterance of the other electronic device received from the other electronic device using the communication circuit during a VoIP call, and to display, using the display, within the second UI displayed together with the first UI, notification information indicating that the translation service is being performed during the VoIP call with the other electronic device based on performing the earliest ASR operation of the user's utterance of the other electronic device that was first received from the other electronic device using the communication circuit.

[0183] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to refrain from displaying, within the second UI displayed together with the first UI, the results of a translation of an utterance of the user of the electronic device within the VoIP call with the other electronic device before performing the earliest ASR operation of the utterance of the user of the other electronic device that was first received using the communication circuit from the other electronic device.

[0184] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to refrain from performing a translation of an utterance of the user of the electronic device in the VoIP call with the other electronic device before performing the earliest ASR operation of the utterance of the user of the other electronic device that was first received using the communication circuit from the other electronic device, thereby refraining from displaying a result of the translation of the utterance of the user of the electronic device in the VoIP call with the other electronic device.

[0185] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to display, within the second UI displayed together with the first UI, information indicating that the translation service during the VoIP call with the other electronic device begins upon the start of utterance of the user of the other electronic device before performing the earliest ASR operation of the utterance of the user of the other electronic device that was first received using the communication circuit from the other electronic device.

[0186] For example, the instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to initiate the translation service during the VoIP call with the other electronic device based on performing the earliest ASR operation of the user's utterance of the other electronic device that was first received using the communication circuit from the other electronic device.

[0187] The non-transitory computer-readable storage medium described above may store one or more programs. The one or more programs may include instructions that, when executed by an electronic device having a display and a communication circuit, cause the electronic device to execute the first application to display, using the display, a first user interface (UI) of a first application for a voice over internet protocol (VoIP) call with another electronic device performed using the communication circuit; to display, using the display, a UI object that receives an input for performing a translation service during the VoIP call with the other electronic device based on an indication for establishing the VoIP call, together with the first UI; to receive an input for the UI object; and to display, based on the input, a second UI of a second application for the translation service, using the display, together with the first UI.

[0188] For example, the indication may correspond to a command from the first application to set the audio mode to a communication audio mode.

[0189] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to capture a screen including the first UI based on the command, perform optical character recognition (OCR) on at least a portion of the captured screen, thereby obtaining identification information of a user of the other electronic device included in the first UI, identify whether data of a source language and a target language set in contact information exist in the electronic device in relation to the identification information, and display the UI object together with the first UI based on an identification that the data exists, and refrain from displaying the UI object based on an identification that the data does not exist.

[0190] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to add another UI object for a translation service during a VoIP call with the other electronic device provided by the second application to a quick panel displayed on the display based on a drag input moving away from an edge area of ​​the display based on the command, and to display the second UI, together with the first UI, using the display based on an input to the other UI object in the quick panel.

[0191] For example, the one or more programs may include instructions that cause the electronic device, when executed by the electronic device, to display the second UI superimposed on the first UI while simultaneously displaying the first UI and the third UI of the third application using the display according to a split view, and to display the second UI together with the first UI using the display, and to maintain the display of the second UI superimposed on the first UI based on execution information of applications of the electronic device while the third application is in a foreground execution state and the first application is in a background execution state.

[0192] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to perform an automatic speech recognition (ASR) operation of a user's utterance of the other electronic device received from the other electronic device using the communication circuit during a VoIP call, and to display, using the display, within the second UI displayed together with the first UI, information indicating that the translation service during the VoIP call with the other electronic device is started based on performing the earliest ASR operation of the user's utterance of the other electronic device that was first received from the other electronic device using the communication circuit.

[0193] For example, the one or more programs may include instructions that, when executed by the electronic device, cause the electronic device to refrain from displaying, within the second UI displayed together with the first UI, the results of a translation of an utterance of the user of the electronic device within the VoIP call with the other electronic device before performing the earliest ASR operation of the utterance of the user of the other electronic device that was first received using the communication circuit from the other electronic device.

[0194] For example, the one or more programs, when executed by the electronic device, may cause the electronic device to initiate the translation service during the VoIP call with the other electronic device based on performing the fastest ASR operation of the user's utterance of the other electronic device that was first received using the communication circuit from the other electronic device.

[0195] As described above, an electronic device (e.g., electronic device (200)) may include a display (e.g., display (240)), a communication circuit (e.g., communication circuit (230)), at least one processor (e.g., at least one processor (210)) including a processing circuit, and a memory (e.g., memory (220)) that stores instructions and includes one or more storage media. The instructions, when individually or collectively executed by the at least one processor, may cause the electronic device to start a translation service during a first call performed through an application providing the second UI based on receiving, using the communication circuitry, a signal indicating acceptance of a call connection request from the other electronic device, using the communication circuitry, while displaying, using the display, a second UI for a translation service together with a first UI for a first call with the other electronic device performed according to circuit switching using the communication circuitry, and to start a translation service during a second call performed through the application based on performing automatic speech recognition (ASR) of an utterance of a user of the other electronic device initially received using the communication circuitry from the other electronic device, using the communication circuitry, while displaying, using the display, the second UI together with a third UI for a second call with the other electronic device performed according to packet switching using the communication circuitry, in connection with the second call with the other electronic device.

[0196] Electronic devices according to the various embodiments disclosed in this document may take various forms. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to the embodiments of this document are not limited to the aforementioned devices.

[0197] The various embodiments of this document and the terminology used therein are not intended to limit the technical features described in this document to specific embodiments, but should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, similar reference numerals may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the items, unless the context clearly indicates otherwise. In this document, each of the phrases "A or B", "at least one of A and B", "at least one of A or B", "A, B, or C", "at least one of A, B, and C", and "at least one of A, B, or C" can include any one of the items listed together in the corresponding phrase among those phrases, or all possible combinations thereof. Terms such as "first," "second," or "first" or "second" may be used merely to distinguish one component from another, and do not limit the components in any other respect (e.g., importance or order). When a component (e.g., a first component) is referred to as "coupled" or "connected" to another (e.g., a second component), with or without the terms "functionally" or "communicatively," it means that the component can be connected to the other component directly (e.g., wired), wirelessly, or through a third component.

[0198] The term "module" used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit. A module may be an integral component, or a minimum unit or part of such a component that performs one or more functions. For example, according to one embodiment, a module may be implemented in the form of an application-specific integrated circuit (ASIC).

[0199] Various embodiments of the present document may be implemented as software (e.g., a program (1840)) including one or more instructions stored in a storage medium (e.g., an internal memory (1836) or an external memory (1838)) readable by a machine (e.g., an electronic device (1801)). For example, a processor (e.g., a processor (1820)) of the machine (e.g., an electronic device (1801)) may call at least one instruction among the one or more instructions stored from the storage medium and execute it. This enables the machine to operate to perform at least one function according to the at least one called instruction. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' simply means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and the term does not distinguish between cases where data is stored semi-permanently or temporarily on the storage medium.

[0200] According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. The computer program product may be traded as a commodity between a seller and a buyer. The computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read-only memory (CD-ROM)), or may be distributed online (e.g., downloaded or uploaded) via an application store (e.g., Play Store™) or directly between two user devices (e.g., smart phones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or an intermediary server.

[0201] According to various embodiments, each component (e.g., a module or a program) of the above-described components may include one or more entities, and some of the entities may be separated and placed in other components. According to various embodiments, one or more components or operations of the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (e.g., a module or a program) may be integrated into a single component. In such a case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. According to various embodiments, the operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, omitted, or one or more other operations may be added.

Claims

In electronic devices display; communication circuit; At least one processor comprising a processing circuit; and A memory storing instructions, comprising one or more storage media, wherein the instructions, when individually or collectively executed by the at least one processor, Executing the first application to display the first UI (user interface) of the first application for a VoIP (voice over internet protocol) call with another electronic device performed using the above communication circuit, using the display, Based on an indication for establishing a VoIP call, a UI object for receiving an input for performing a translation service during the VoIP call with the other electronic device is displayed using the display together with the first UI, Receive input for the above UI object, and Based on the above input, display the second UI of the second application for the translation service, together with the first UI, using the display. causing the above electronic device, Electronic devices. In claim 1, the indication is: In response to a command from the first application to set the audio mode to communication audio mode, Electronic devices. In claim 1, the instructions, when individually or collectively executed by the at least one processor, Based on the above command, capture the screen including the first UI, By performing OCR (optical character recognition) on at least a portion of the captured screen, user identification information of the other electronic device included in the first UI is obtained, Identify whether data of the source language and target language set in the contact information in relation to the above identification information exists within the electronic device; Based on the identification that the data exists within the electronic device, displaying the UI object together with the first UI, Refrain from displaying the UI object based on the identification that the said data does not exist within the said electronic device; causing the above electronic device, Electronic devices. In claim 2, the UI object, Overlapping or floating on a part of the above first UI, Electronic devices. In claim 2, the instructions, when individually or collectively executed by the at least one processor, Based on the above command, adding another UI object for a translation service during a VoIP call with the other electronic device provided by the second application to the quick panel displayed on the display according to a drag input moving away from the edge area of ​​the display, Based on the input to the other UI object within the quick panel, display the second UI together with the first UI using the display. further causing the above electronic device, Electronic devices. In claim 1, the second UI, Overlapping at least a portion of the above first UI, At least a portion of the above second UI, translucent, Electronic devices. In claim 1, the second UI, Overlapping on a second area within the first UI that is different from a first area within the first UI that includes UI objects, The above second UI is, Opaque, Electronic devices. In claim 1, the instructions, when individually or collectively executed by the at least one processor, While simultaneously displaying the first UI and the third UI of the third application using the display according to the split view: By displaying the second UI overlapping the first UI, the second UI is displayed together with the first UI using the display, While the third application is in a foreground execution state and the first application is in a background execution state, based on execution information of applications of the electronic device, the display of the second UI overlaid on the first UI is maintained. causing the above electronic device, Electronic devices. In claim 1, the instructions, when individually or collectively executed by the at least one processor, During a VoIP call, perform an automatic speech recognition (ASR) operation of a user's speech of the other electronic device received from the other electronic device using the communication circuit; Based on performing the fastest ASR operation of the user's utterance of the other electronic device that is first received using the communication circuit from the other electronic device, displaying notification information indicating that the translation service is being performed during the VoIP call with the other electronic device, using the display, within the second UI displayed together with the first UI. further causing the above electronic device, Electronic devices. In claim 9, the instructions, when individually or collectively executed by the at least one processor, Before performing the fastest ASR operation of the utterance of the user of the other electronic device that is first received using the communication circuit from the other electronic device, refrain from displaying the result of the translation of the utterance of the user of the electronic device in the VoIP call with the other electronic device within the second UI displayed together with the first UI. further causing the above electronic device, Electronic devices. In claim 10, the instructions, when individually or collectively executed by the at least one processor, By refraining from performing a translation of the utterance of the user of the electronic device in the VoIP call with the other electronic device before performing the fastest ASR operation of the utterance of the user of the other electronic device that was first received using the communication circuit from the other electronic device, and thereby refraining from displaying the result of the translation of the utterance of the user of the electronic device in the VoIP call with the other electronic device, further causing the above electronic device, Electronic devices. In claim 9, the instructions, when individually or collectively executed by the at least one processor, Before performing the fastest ASR operation of the utterance of the user of the other electronic device that is first received using the communication circuit from the other electronic device, information indicating that the translation service during the VoIP call with the other electronic device starts upon the start of the utterance of the user of the other electronic device is displayed within the second UI displayed together with the first UI. further causing the above electronic device, Electronic devices. In claim 9, the instructions, when individually or collectively executed by the at least one processor, To start the translation service during the VoIP call with the other electronic device based on performing the fastest ASR operation of the user's speech of the other electronic device that is first received using the communication circuit from the other electronic device, further causing the above electronic device, Electronic devices. In a non-transitory computer-readable storage medium storing one or more programs, the one or more programs, when executed by an electronic device having a display and communication circuit, Executing the first application to display the first UI (user interface) of the first application for a VoIP (voice over internet protocol) call with another electronic device performed using the above communication circuit, using the display, Based on an indication for establishing a VoIP call, a UI object for receiving an input for performing a translation service during the VoIP call with the other electronic device is displayed using the display together with the first UI, Receive input for the above UI object, and Based on the above input, display the second UI of the second application for the translation service, together with the first UI, using the display. comprising instructions causing the electronic device to operate; Computer readable storage medium. In a method of an electronic device having a display and communication circuit, An operation of executing the first application to display a first UI (user interface) of the first application for a VoIP (voice over internet protocol) call with another electronic device performed using the above communication circuit, using the display; An action of displaying, using the display, a UI object that receives an input for performing a translation service during the VoIP call with the other electronic device based on an indication for establishing a VoIP call, together with the first UI; An action for receiving input for the above UI object, and Based on the above input, an action is included to display a second UI of a second application for the translation service, together with the first UI, using the display. method.

Images