Text call method and text call device for mobile terminal

Abstract

The invention discloses a text call method and a text call device for a mobile terminal. The text call method comprises the steps of displaying a text call interface in response to the fact that a text call function is started and a preset application of the mobile terminal is in a call state; in response to the fact that first text data are input into the text call interface, the first text data are converted into first voice data, the first voice data are transmitted to the preset application, and the first text data are only displayed in a first area of the text call interface, the first voice data is sent to the other party of the call by the preset application; and in response to obtaining second voice data from the preset application, converting the second voice data into second text data, and only displaying the second text data in a second area of the text call interface, the second voice data being received by the preset application from the other party of the call.

Description

Technical Field

[0001] The present disclosure generally relates to the field of electronic technology, and more particularly, to a text call method and a text call device for a mobile terminal. Background Art

[0002] With the development and popularization of various smart terminals, audio calls (i.e., VoIP calls, such as WeChat voice calls, QQ voice calls, etc.) as a means of remote communication and collaboration are expanding in modern society. VoIP calls enable people to communicate and collaborate in different locations through real-time audio transmission.

[0003] Currently, in order to solve the problem that users are not convenient to reply to calls with voice in certain scenarios, and to solve the call problems of people with hearing impairments / speech impairments, some applications support text call functions. For example, when user A receives a WeChat voice call from user B in the library and is not convenient to make a sound, user A can turn on the text call function and directly enter text messages in the display interface of the text call application (hereinafter referred to as the text call interface). These text messages will be converted into voice messages and sent to user B, thereby achieving smooth communication in specific scenarios.

[0004] Figure 1 is a diagram showing an example of an existing text call function.

[0005] like Figure 1 As shown, user A of the mobile terminal turns on the text call function in the WeChat voice call interface, and the mobile terminal displays the text call interface. At this time, the voice call is displayed on the text call interface in a floating window. User A enters text 1 ("How's the weather today?") in the text call interface, and text 1 is converted into voice data Voice_data1, and as the input of the WeChat voice call, it is sent to the other party of the call (ie, user B) through WeChat. In this way, user B can hear the corresponding voice content.

[0006] exist Figure 1In the example, the right side box displays the speech of user A, and the left side box displays the speech of user B. After user A inputs and sends text 1 (Text1, "How's the weather today?"), text 2 (Tex2, "How's the weather today?") should not be repeatedly displayed in the left side box (i.e., the area where user B speaks), but only text 3 (Text3, "The weather is nice today") converted from the voice content of user B should be displayed. However, when text 1 is converted into voice data Voice_data1 and transmitted to WeChat, if the voice data Voice_data1 is captured by the text call APP (i.e., the application that performs the text call function), Voice_data1 will be mistakenly recognized as the voice data transmitted by user B to user A through WeChat. In this case, the voice data Voice_data1 will be converted into text 2 through automatic speech recognition (ASR) technology and displayed in the left side box of the text call interface, resulting in a display confusion problem on the text call interface. Summary of the invention

[0007] Embodiments of the present disclosure provide a text call method and a text call device for a mobile terminal, which can effectively solve the display confusion phenomenon on the text call interface.

[0008] According to a first aspect of an embodiment of the present disclosure, a text call method for a mobile terminal is provided, the text call method comprising: in response to a text call function being turned on and a preset application of the mobile terminal being in a call state, displaying a text call interface; in response to first text data being input into the text call interface, converting the first text data into first voice data and transmitting the first voice data to the preset application, and displaying the first text data only in a first area of ​​the text call interface, wherein the first voice data will be sent by the preset application to the other party of the call; in response to obtaining second voice data from the preset application, converting the second voice data into second text data, and displaying the second text data only in a second area of ​​the text call interface, wherein the second voice data is received by the preset application from the other party of the call.

[0009] Optionally, the text call method also includes: in response to the text call function being turned on, registering to monitor audio status events with the operating system of the mobile terminal to monitor the audio status of each application of the mobile terminal; in response to monitoring the audio status of the preset application changing, determining that the preset application is in a call state.

[0010] Optionally, converting the first text data into first voice data and transmitting the first voice data to the preset application includes: transmitting the first voice data only to the preset application, and not transmitting the first voice data to any other application of the mobile terminal.

[0011] Optionally, only the first voice data is transmitted to the preset application via a first dedicated voice data transmission path.

[0012] Optionally, the second voice data is obtained from the preset application through a second dedicated voice data transmission path, or the second voice data is obtained from the preset application through a second dedicated voice data transmission path, and the second voice data is played through a speaker of the mobile terminal.

[0013] Optionally, in response to a mute function for the second voice data being enabled, the second voice data is acquired from the preset application only through a second dedicated voice data transmission path, and the second voice data is not played through a speaker of the mobile terminal.

[0014] Optionally, in response to determining that the preset application is an application that supports the text call function, a first dedicated voice data transmission path is established for transmitting the first voice data to the preset application, and / or in response to determining that the preset application is an application that supports the text call function, a second dedicated voice data transmission path is established for obtaining the second voice data from the preset application.

[0015] Optionally, in response to the unique identification information of the preset application being included in the unique identification information list of applications supporting the text call function, the preset application is determined to be an application supporting the text call function, wherein the unique identification information list of applications supporting the text call function is constructed based on the unique identification information of the application supporting the text call function set by the user.

[0016] Optionally, the step of establishing a first dedicated voice data transmission path for transmitting the first voice data to the preset application includes: creating a first virtual audio device, routing the data stream of the first voice data to the output side of the first virtual audio device, and routing the recording data stream of the preset application to the input side of the first virtual audio device, thereby establishing a first dedicated voice data transmission path.

[0017] Optionally, the step of establishing a second dedicated voice data transmission path for obtaining the second voice data from the preset application includes: creating a second virtual audio device, routing the data stream of the second voice data to the input side of the second virtual audio device, and routing the playback data stream of the preset application to the output side of the second virtual audio device, thereby establishing a second dedicated voice data transmission path.

[0018] Optionally, the text call method also includes: creating an AudioContinuity module to record at least one of the following parameters: package name information of the application that executes the text call method; unique identification information of the application that supports the text call function; whether to exit the text call function; whether to enable the mute function for the second voice data while establishing a first dedicated voice data transmission path dedicated to the first voice data and a second dedicated voice data transmission path dedicated to the second voice data.

[0019] Optionally, the first area represents an area of ​​the text call interface for displaying sent messages, and the second area represents an area of ​​the text call interface for displaying received messages.

[0020] According to a second aspect of an embodiment of the present disclosure, a text call device for a mobile terminal is provided, the text call device comprising: a display module, configured to: in response to a text call function being turned on and a preset application of the mobile terminal being in a call state, display a text call interface; a text data processing module, configured to: in response to first text data being input into the text call interface, convert the first text data into first voice data and transmit the first voice data to the preset application, and display the first text data only in a first area of ​​the text call interface, wherein the first voice data will be sent by the preset application to the other party of the call; a voice data processing module, configured to: in response to obtaining second voice data from the preset application, convert the second voice data into second text data, and display the second text data only in a second area of ​​the text call interface, wherein the second voice data is received by the preset application from the other party of the call.

[0021] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the text call method for a mobile terminal as described above is implemented.

[0022] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory storing a computer program, wherein when the computer program is executed by the processor, the text call method for a mobile terminal as described above is implemented.

[0023] According to the text call method and text call device for a mobile terminal according to the embodiments of the present disclosure, it is possible to ensure that when the text call function is turned on, the text data input by the user is correctly displayed in the user's speech area, and will not be erroneously displayed in the speech area of ​​the other party of the call, thereby avoiding display confusion on the text call interface and improving the user experience.

[0024] Additional aspects and / or advantages of the present general inventive concept will be set forth in part in the following description and in part will be apparent from the description or may be learned through practice of the present general inventive concept. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects and features of exemplary embodiments of the present disclosure will become more apparent from the following description with reference to the accompanying drawings which exemplarily illustrate the embodiments.

[0026] Figure 1 is a diagram showing an example of an existing text call function;

[0027] Figure 2 is a diagram showing an example of an audio processing architecture for executing a text call method for a mobile terminal according to an embodiment of the present disclosure;

[0028] Figure 3 is a flow chart showing a text call method for a mobile terminal according to an embodiment of the present disclosure;

[0029] Figure 4 is a diagram showing an example of turning on a text call function;

[0030] Figure 5 is a diagram illustrating an example of determining one or more applications supporting a text call function;

[0031] Figure 6 is a diagram showing an example of turning on a mute function for second voice data;

[0032] Figure 7 is a diagram showing an example of a text call interface displayed when a VoIP call is connected;

[0033] Figure 8 is a diagram showing another example of an audio processing architecture for executing a text call method for a mobile terminal according to an embodiment of the present disclosure;

[0034] Fig. 9 is a diagram showing an example of a text call disabling interface according to an embodiment of the present disclosure;

[0035] Fig.10 is a block diagram showing a text call device for a mobile terminal according to an embodiment of the present disclosure;

[0036] Fig.11 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same or similar elements, features and structures. However, it is not intended that the present disclosure be limited to specific embodiments by the various embodiments described herein, and it is intended that: the present disclosure covers all modifications, equivalents and / or substitutes of the present disclosure, as long as they are within the scope of the attached claims and their equivalents. The terms and words used in the following specification and claims are not limited to their dictionary meanings, but are only used to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and not for the purpose of limiting the present disclosure defined by the attached claims and their equivalents.

[0038] It should be understood that the singular includes the plural unless the context clearly indicates otherwise.The terms "include", "comprising" and "having" used herein indicate the presence of disclosed functions, operations or elements, but do not exclude other functions, operations or elements.

[0039] For example, the expression "A or B" or "at least one of A and / or B" may indicate A and B, A or B. For example, the expression "A or B" or "at least one of A and / or B" may indicate (1) A, (2) B, or (3) both A and B.

[0040] In various embodiments of the present disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component) or being "coupled" or "connected" to another component (e.g., the second component), the component may be directly connected to the other component or may be connected through another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to another component (e.g., the second component) or being directly coupled to or directly connected to another component (e.g., the second component), there is no other component (e.g., the third component) between the component and the other component.

[0041] The expression "configured to" used in describing various embodiments of the present disclosure may be used interchangeably with expressions such as "suitable for", "having the ability to", "designed to", "suitable for", "manufactured to", and "capable of", for example, depending on the circumstances. The term "configured to" may not necessarily indicate that it is "specially designed to" in terms of hardware. On the contrary, the expression "a device configured to..." in some cases may indicate that the device and another device or part "can..." For example, the expression "a processor configured to perform A, B, and C" may indicate a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit CPU or an application processor (AP)) for performing the corresponding operations by executing at least one software program stored in a memory.

[0042] The terms used herein are to describe certain embodiments of the present disclosure, but are not intended to limit the scope of other embodiments. Unless otherwise noted herein, all terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those skilled in the art. Typically, the terms defined in the dictionary should be considered to have the same meaning as the contextual meaning in the relevant field, and, unless clearly defined herein, should not be understood differently or understood to have an overly formal meaning. In any case, the terms defined in the present disclosure are not intended to be interpreted as excluding embodiments of the present disclosure.

[0043] Refer to the following Figures 2 to 11 A text call method and a text call device for a mobile terminal according to embodiments of the present disclosure are described in detail.

[0044] Figure 2 is a diagram illustrating an example of an audio processing architecture for performing a text call method for a mobile terminal according to an embodiment of the present disclosure.

[0045] Reference Figure 2 The audio processing architecture 200 may include an application layer (Application), an application framework layer (Framework) and a hardware abstraction layer (HAL).

[0046] like Figure 2 As shown in the figure, various applications related to audio processing are all running in the application layer, such as the VoIPAPP and Controller APP shown in the figure. In addition, various applications related to audio processing also include audio players, telephones, sound settings, recorders, etc.

[0047] According to an embodiment of the present disclosure, the text call method for the mobile terminal according to an embodiment of the present disclosure is executed by constructing a Controller APP as a system-level application of the mobile terminal. Here, the Controller APP can be a system-level APP specially developed for executing the text call method, or it can be an original system-level APP of the mobile terminal that executes the text call method.

[0048] The VoIP APP may include an APP Playback module and an APP Recording module. The VoIP APP receives voice data from the other party of the call, and transmits the received voice data as playback data (e.g., audio data played through the speaker of the mobile terminal) to the application framework layer through the APP Playback module. The APP Recording module is used to obtain recording data (e.g., audio data input by the user recorded through the microphone of the mobile terminal) from the application framework layer, and transmit the recording data to the other party of the call through the VoIP APP.

[0049] The Controller APP includes a text input interface (Input words), a text output interface (Output words), a TTS (text-to-speech) module, an ASR (automatic speech recognition) module, an AudioTrack module, and an AudioRecord module. The text input interface is used to receive text data input by a mobile terminal user in the text call interface, the TTS module is used to convert text data into audio data, and the AudioTrack module, as an API (application programming interface), is used to transmit audio data as playback data to the application framework layer. The AudioRecord module, as an API, is used to obtain recording data from the application framework layer, the ASR module is used to convert recording data into text data, and the text output interface is used to display text data on the text call interface.

[0050] The application framework layer is the core component for processing audio data in the operating system of the mobile terminal (for example, but not limited to the Android system). It allows applications to capture, play, process and render audio data to support various audio applications, such as audio players, voice recognition, audio recording, text calls, etc. The application framework layer may include an AudioFlinger module, which is responsible for managing and scheduling the transmission of audio data. The AudioFlinger module is responsible for the transmission of audio data between the VoIP APP and the Controller APP. The AudioFlinger module may further include a playback thread (PlaybackThread) submodule and a recording thread (RecordThread) submodule. The PlaybackThread submodule is responsible for transmitting the playback data from the application layer to the hardware abstraction layer, and the RecordThread submodule is responsible for transmitting the recording data from the hardware abstraction layer to the application layer.

[0051] The hardware abstraction layer is generally responsible for interacting with hardware devices (such as but not limited to speakers, microphones, etc.). The hardware abstraction layer may include a Remote Submix HAL module for implementing the audio data connection between the VoIP APP and the Controller APP. The Remote Submix HAL module may further include an audio pipe (MonoPipe) submodule, each of which may connect a playback data stream with a recording data stream.

[0052] The Controller APP includes the AudioManager module, which is an audio control interface. The Controller APP can set corresponding parameters through the AudioManager module to control whether to provide the audio data streams of the VoIP APP and the Controller APP to the Remote Submix HAL module, and whether to connect the audio data streams of the VoIP APP and the Controller APP.

[0053] The AudioFlinger module in the application framework layer may include two playback thread submodules (PlaybackThread1 and PlaybackThread 2) and two recording thread submodules (RecordThread 1 and RecordThread 2). PlaybackThread 1 is used to transmit the audio data converted from the text data output by the Controller APP to the hardware abstraction layer, and RecordThread 1 is used to transmit the audio data converted from the text data obtained from the hardware abstraction layer to the VoIP APP. PlaybackThread 2 is used to transmit the audio data obtained from the other party of the call output by the VoIP APP to the hardware abstraction layer, and RecordThread 2 is used to transmit the audio data of the other party of the call obtained from the hardware abstraction layer to the Controller APP.

[0054] According to an embodiment of the present disclosure, the application framework layer may further include an AudioPolicy module and an AudioContinuity module, and the AudioFlinger module may further include a SecAudioParamFlinger submodule. The SecAudioParamFlinger submodule is used to parse the parameters set by the AudioManager submodule and provide the parsed parameters to the AudioContinuity module. The AudioContinuity module is specially constructed to store the parameters set by the AudioManager submodule for use by the AudioPolicy module. The AudioPolicy module uses the parameters stored in the AudioContinuity module to control which device and the corresponding HAL layer are selected for the audio data in the mobile terminal. In other words, the AudioPolicy module can select a voice data transmission path for the VoIPAPP and the Controller APP based on the parameters stored in the AudioContinuity module.

[0055] The Remote Submix HAL module in the hardware abstraction layer can use two audio pipe (MonoPipe) submodules (MonoPipe 1 and MonoPipe 2). MonoPipe 1 is used to connect the playback data stream of the Controller APP (i.e., the audio data converted from the text data) with the recording data stream of the VoIP APP, so that the audio data converted from the text data can be sent to the other party of the call. MonoPipe 2 is used to connect the playback data stream of the VoIP APP (i.e., the voice data received from the other party of the call) with the recording data stream of the Controller APP, so that the Controller APP outputs the text data converted from the voice data.

[0056] According to an embodiment of the present disclosure, the AudioTrack module, PlaybackThread 1, MonoPipe 1, RecordThread 1 and APP Recording module may constitute a first dedicated voice data transmission path (in Figure 2 ), which is dedicated to transmitting only the audio data converted from the text data from the text call APP to the VoIP APP; the APP Playback module, PlaybackThread 2, MonoPipe2, RecordThread 2 and AudioRecord module can constitute a second dedicated voice data transmission channel (in Figure 2 The Controller APP is used to transmit the voice data received by the VoIP APP from the other party of the call from the VoIP APP to the Controller APP.

[0057] Figure 3 is a flow chart showing a text call method for a mobile terminal according to an embodiment of the present disclosure. The text call method according to an embodiment of the present disclosure may be executed by the Controller APP as described above.

[0058] Reference Figure 3 In step S301, in response to the text call function being turned on and the preset application (ie, VoIP APP) of the mobile terminal being in a call state, a text call interface is displayed.

[0059] According to an embodiment of the present disclosure, a user of a mobile terminal (ie, user A) can Figure 4 The operation shown turns on the text call function. Figure 4 is a diagram showing an example of turning on the text call function. Figure 4 As shown, in response to the user's operation, the text call function (i.e., Text call) can be Figure 4 The closed state shown on the left in the figure changes to Figure 4 Here, the user's operation may include but is not limited to sliding operation, clicking operation, gesture operation, etc.

[0060] Furthermore, when the text call function is turned on, a text call settings interface (ie, Text call settings) may be entered to determine one or more applications that support the text call function. Figure 5 is a diagram showing an example of determining one or more applications that support a text call function. When the text call function is turned on, at least one VoIP APP that supports the text call function can be selected. Here, in response to the user's selection, WeChat can be determined as an application that supports the text call function. On the other hand, in response to the user's selection, applications such as QQ, Enterprise WeChat (Wecom), DingDing, and Tencent Meeting can be determined as applications that do not support the text call function. Optionally, after determining one or more applications that support the text call function, an APP list (i.e., mPackageList) that supports the text call function can be automatically generated. Optionally, when the text call function is turned on, all VoIP APPs can be automatically selected as applications that support the text call function.

[0061] Optionally, when the text call function is turned on, it can be determined whether to turn on the mute function (Mute other person's voice) for the second voice data obtained from the VoIP APP (ie, the voice data received by the VoIP APP from the other party of the voice call) after entering the text call setting interface. Figure 6 is a diagram showing an example of turning on the mute function. Figure 6 As shown, in response to the user's operation, the mute function can be Figure 6 The closed state shown on the left in the figure changes to Figure 6 When the text call function is turned on, the mute function can be turned on or off. When the mute function is turned off, the voice of the other end or other party (i.e., user B) received during the VoIP call can still be played through the speaker, that is, user A can hear the voice of user B while seeing the text converted from the voice. On the other hand, when the mute function is turned on, user A will not hear the voice of user B, but will only see the text converted from the voice of user B.

[0062] According to an embodiment of the present disclosure, when the text call function is turned on, the Controller APP may register a monitoring audio status event with the operating system of the mobile terminal to monitor the audio status of the application; when the audio status of the preset application is monitored to change, it can be determined that the preset application is in a call state. Here, the application refers to a VoIP APP installed on the mobile terminal, but the present disclosure is not limited to this. In this way, when the text call function is turned on, when the preset application is in a call state, it can be determined based on mPackageList whether the preset application supports the text call function. If the preset application supports the text call function, a text call interface can be displayed, and if the preset application does not support the text call function, a normal VoIP call can continue to be performed. Optionally, when the text call function is turned on, when the preset application supports the text call function, the text call interface can be directly displayed, or the text call interface can be displayed according to the user's selection (for example, touching a specific icon).

[0063] According to an embodiment of the present disclosure, when a preset application enters a call state, in response to a preset operation of the user, the text call function of the preset application may be enabled, thereby displaying a text call interface.

[0064] Figure 7 is a diagram showing an example of a text call interface displayed when a VoIP call is connected. Figure 7 As shown, the user can enter text in the text input box of the text call interface, and when the send button is selected, the text data is converted into voice. Optionally, the text call interface can also be displayed in a floating window form.

[0065] Below, refer to Figure 2 Describes the operations performed in the audio processing architecture 200 when a text call interface is displayed.

[0066] In S1, while displaying the text call interface, the Controller APP can set parameters through the AudioManager module. Specifically, the set parameters may include the package name information (package name) of the Controller APP, the UID of the application that supports the text call function, the audio streaming mode, etc. Here, the audio streaming mode may include: CONTINUITY_APP_NONE, which means stopping the audio streaming and restoring the normal state; CONTINUITY_APP_ALL, which means that the first dedicated voice data transmission path and the second dedicated voice data transmission path will be established respectively, and the audio data stream output by the VoIP APP to the speaker will be cut off (that is, the mute function is turned on), and the audio data stream input from the microphone to the VoIP APP will be cut off; CONTINUITY_APP_ALL_AND_PLAY, which means that the first dedicated voice data transmission path and the second dedicated voice data transmission path will be established respectively, and the audio data stream output by the VoIP APP to the speaker will be retained (that is, the mute function is turned off), and the audio data stream input from the microphone to the VoIP APP will be cut off. That is to say, when the mute function is not turned on, the Controller APP sets the audio streaming mode to CONTINUITY_APP_ALL_AND_PLAY; when the mute function is turned on, the Controller APP sets the audio streaming mode to CONTINUITY_APP_ALL.

[0067] In S2, the SecAudioParamFlinger submodule of the AudioFlinger module parses the parameters set by the AudioManager module. As a non-limiting example, the parameters set by the AudioManager module may include A, B, and C. Here, parameter A indicates that the set parameters are used for the audio streaming mode; parameter B indicates the UID information of the VoIP APP that allows the use of the text call function. Only when the VoIP APP has the UID represented by B can the first dedicated voice data transmission path and the second dedicated voice data transmission path be established to transmit audio data between the VoIP APP and the Controller APP. Parameter C indicates whether the audio streaming mode is CONTINUITY_APP_ALL or CONTINUITY_APP_ALL_AND_PLAY. When parameter C indicates CONTINUITY_APP_ALL, the mute function is turned on, and when parameter C indicates CONTINUITY_APP_ALL_AND_PLAY, the mute function is turned off.

[0068] Furthermore, parameter C is used to limit the audio data transmission between the Controller APP and the VoIP APP that is currently in the VoIP call mode and supports the text call function.

[0069] Next, in S3, the AudioContinuity module records the parameters parsed by the SecAudioParamFlinger submodule. As mentioned above, the AudioManager module provides an API interface for the VoIP APP, which does not have storage capabilities. On the other hand, the AudioPolicy module and the AudioFlinger module are core modules in the audio processing architecture and are not suitable for storing large amounts of data. Therefore, according to an embodiment of the present disclosure, an AudioContinuity module is constructed to store the parameters set by the AudioManager module. In this way, the AudioPolicy module and the AudioFlinger module can easily access the AudioContinuity module and obtain the parameters stored therein, thereby improving the stability of the audio processing architecture and the cleanliness of the code.

[0070] In S4, the AudioPolicy module establishes a first dedicated voice data transmission path based on the parameters stored in the AudioContinuity module. Figure 2 The second dedicated voice data transmission path (shown by the blue line in Figure 2 Here, as an example, S4 is described as being immediately after S3. In fact, the operation of establishing the first dedicated voice data transmission path in S4 can be performed when the Controller APP turns on the AudioTrack module, and the operation of establishing the second dedicated voice data transmission path in S4 can be performed when the Controller APP turns on the AudioRecord module.

[0071] When establishing the first dedicated voice data transmission path, it is possible to first determine whether the running VoIP APP is an APP that supports the voice call function based on the unique identification information of the running VoIP APP (for example, the UID or package name as described above) and the pre-built unique identification information list of VoIP APPs that support the text call function as described above; then, in response to determining that the running VoIP APP is an APP that supports the voice call function, establish the first dedicated voice data transmission path that connects the playback data stream of the audio data (i.e., the first voice data) converted from the text data and the recording data stream of the running VoIP APP. Below, the description is taken as an example of the unique identification information being UID.

[0072] More specifically, the AudioPolicy module can create a virtual audio device 1 in the Remote Submix HAL module based on the parameters obtained from the AudioContinuity module, and route the data stream of the first voice data to the output side of the virtual audio device 1. Then, the UID of the running VoIP APP is matched with the unique identification information list of the VoIP APP that supports the text call function contained in the parameters (for example, but not limited to the mPackageList described above). If the running VoIP APP exists in the unique identification information list of the VoIP APP that supports the text call function, the match is successful; otherwise, the match fails. Through UID matching, it can be ensured that only VoIP APPs that are in a VoIP call and support the text call function can be successfully matched, and other VoIP APPs (including VoIP APPs that are in a VoIP call but do not support the text call function) will not be matched.

[0073] If the match is successful, the AudioPolicy module and the AudioFlinger module can route the recording data stream of the VoIP APP to the input side of the virtual audio device 1 according to the address of the virtual audio device 1 (the address assigned when the virtual audio device 1 is created); if the match fails, no processing will be performed. At this time, an independent first dedicated voice data transmission path connecting the playback data stream of the Controller APP and the recording data stream of the VoIP APP is established. In this way, through the established independent first dedicated voice data transmission path, the Controller APP can only transmit its playback data to the corresponding VoIP APP, and will not transmit its playback data to any other application of the mobile terminal (including the Controller APP itself).

[0074] Optionally, a pair of virtual audio devices may be established for each VoIP APP supporting the text call function, thereby establishing an independent first dedicated voice data transmission path and a second dedicated voice data transmission path for each VoIP APP supporting the text call function, but the present disclosure is not limited thereto.

[0075] On the other hand, when establishing a second dedicated voice data transmission path, it is also possible to first determine whether the running VoIP APP is an APP that supports the voice call function based on the unique identification information of the running VoIP APP and the pre-built unique identification information list of VoIP APPs that support the text call function as described above; then, in response to determining that the running VoIP APP is an APP that supports the voice call function, establish a second dedicated voice data transmission path connecting the playback data stream of the running VoIP APP (i.e., the data stream of the second voice data) and the recording data stream of the Controller APP.

[0076] More specifically, the AudioPolicy module can create a virtual audio device 2 in the Remote Submix HAL module based on the parameters obtained from the AudioContinuity module, and route the recording data stream of the Controller APP to the input side of the virtual audio device 2. Then, as described above, the UID of the running VoIP APP can be matched with the unique identification information list of VoIP APPs supporting the text call function contained in the parameters (such as but not limited to the mPackageList described above). If the running VoIP APP exists in the unique identification information list of VoIP APPs supporting the text call function, the match is successful; otherwise, the match fails. Through UID matching, it can be ensured that only VoIP APPs that are in a VoIP call and support the text call function can be successfully matched, and other VoIP APPs (including VoIP APPs that are in a VoIP call but do not support the text call function) will not be matched.

[0077] If the match is successful, the AudioPolicy module and the AudioFlinger module can route the playback data stream of the VoIP APP to the output side of the virtual audio device 2 according to the address of the virtual audio device 2; if the match fails, no processing will be performed. At this time, an independent second dedicated voice data transmission path connecting the recording data stream of the Controller APP and the playback data stream of the VoIP APP is established. In this way, through the established independent second dedicated voice data transmission path, the Controller APP can obtain the voice data it receives from the VoIP APP.

[0078] Return to reference Figure 3In step S302, in response to the first text data being input into the text call interface, the first text data is converted into first voice data and the first voice data is transmitted to a preset application, and the first text data is displayed only in the first area of ​​the text call interface. The first voice data will be sent by the preset application to the other party of the call (i.e., user B). Here, the first area may refer to an area of ​​the text call interface for displaying a sent message.

[0079] Below, refer to Figure 2 Describes the operations performed in the audio processing architecture 400 when text data is entered in the text call interface.

[0080] In S5-1-a, user A enters text data (for example, "How's the weather today?") in the text call interface. In S5-1-b, the Controller APP converts the input text data into voice data or voice files through the TTS module, and the converted voice data or voice files can be used as the playback data of the Controller APP. In S5-1-c, the Controller APP transmits the playback data to the AudioFlinger module through the AudioTrack module. In S5-1-d, the AudioFlinger module transmits the playback data to the Remote Submix HAL module through the PlaybackThread submodule. In S5-1-e, the Remote Submix HAL module stores the playback data in the MonoPipe submodule of the virtual audio device 1. In S5-1-f, the AudioFlinger module obtains the playback data from the MonoPipe submodule of the virtual audio device 1 through the RecordThread submodule, and transmits the playback data to the App Recording module of the VoIP APP, so that the VoIP APP can send the playback data to user B. In the operation described above, only the voice data converted from the input text data can be transmitted to the preset application through the first dedicated voice data transmission path.

[0081] According to an embodiment of the present disclosure, the Controller APP can convert text data into voice data in the following two ways: (1) converting text data into a voice stream; (2) converting text data into a voice file. Accordingly, when the Controller APP converts text data into a voice stream, the Controller APP can start audio playback to trigger the establishment of a first dedicated voice data transmission path. In the case where the Controller APP converts text data into a voice file, when the Controller APP plays the voice file converted by the TTS module, the establishment of the first dedicated voice data transmission path can be triggered.

[0082] Return to reference Figure 3 In step S303, in response to obtaining the second voice data from the preset application, the second voice data is converted into second text data, and the second text data is displayed only in the second area of ​​the text call interface. Here, the second voice data is received by the preset application from the other party of the call. The second area represents an area of ​​the text call interface for displaying received messages.

[0083] Below, refer to Figure 2 The operations performed in the audio processing architecture 200 are described when a preset application receives voice data from the other party of a call.

[0084] In S5-2-a, the VoIP APP transmits the received voice data (for example, "The weather is nice today.") to the PlaybackThread submodule of the AudioFlinger module through the APP Playback module. In S5-2-b, when the mute function is turned off, the AudioFlinger module transmits the voice data (i.e., playback data) to the Remote Submix HAL module through the PlaybackThread submodule. At the same time, the original playback channel of the speaker is retained, so that the VoIP APP can play the voice data of user B normally through the speaker. In S5-2-c, the Remote Submix HAL module stores the playback data in the MonoPipe submodule of the virtual audio device 2. In S5-2-d, the RecordThread submodule of the AudioFlinger module obtains the playback data from the MonoPipe submodule of the virtual audio device 2, and transmits the playback data to the AudioRecord module of the Controller APP. In S5-2-e, the Controller APP converts the acquired playback data into text data through the ASR module. In S5-2-f, the Controller APP displays the converted text data on the text call interface. In the above operation, since the mute function is turned off, on the one hand, the voice data received by the VoIP APP can be obtained from the VoIP APP through the second dedicated voice data transmission path, and on the other hand, the voice data received by the VoIP APP can be played through the speaker, that is, the voice data received by the VoIP APP is transmitted to the speaker for playback through the first general voice data transmission path (that is, the original playback channel of the speaker).

[0085] exist Figure 2 In the audio processing architecture shown, since the mute function is turned off, the original playback channel of the speaker is retained. Figure 8 is a diagram showing another example of an audio processing architecture for executing a text call method for a mobile terminal according to an embodiment of the present disclosure. Figure 8 In the audio processing architecture shown, the original playback channel of the speaker is disconnected because the mute function is turned on. In this case, the voice data received by the VoIP APP can only be obtained through the second dedicated voice data transmission path, and the voice data received by the VoIP APP is not played through the speaker, that is, the voice data received by the VoIP APP is prohibited from being transmitted to the speaker through the first general voice data transmission path (that is, the original playback channel of the speaker).

[0086] Optionally, the text call method according to the embodiment of the present disclosure may further include the following steps: in response to the preset application exiting the call state, disabling the text call interface. Fig. 9 is a diagram showing an example of a text call disabling interface according to an embodiment of the present disclosure. Fig. 9 , Fig. 9 The left side of shows the enabled text call interface when the text call function is turned on and the preset application is in a call state. Fig. 9 The right side of shows the text call interface disabled in the default application exit call state. When the text call interface is disabled, text data is not allowed to be input on the text call interface. As a non-limiting example, disabling the text call interface may mean that the text call interface is no longer displayed.

[0087] Fig.10 is a block diagram illustrating a text calling device for a mobile terminal according to an embodiment of the present disclosure.

[0088] Reference Fig.10 , the text communication device 1000 may include a display module 1010 , a text data processing module 1020 , and a voice data processing module 1030 .

[0089] Specifically, in response to the text call function being turned on and the preset application of the mobile terminal being in a call state, the display module 1010 may display a text call interface. Alternatively, in response to the text call function being turned on, the text call device 1000 may register a monitoring audio state event with the operating system of the mobile terminal to monitor the audio state of each application of the mobile terminal. In response to monitoring a change in the audio state of the preset application, the text call device 1000 may determine that the preset application is in a call state. In addition, in response to the preset application exiting the call state, the display module 1010 may disable the text call interface.

[0090] In response to the first text data being input into the text call interface, the text data processing module 1020 may convert the first text data into first voice data and transmit the first voice data to a preset application, and display the first text data only in the first area of ​​the text call interface. Here, the first voice data will be sent by the preset application to the other party of the call, and the first area may represent an area of ​​the text call interface for displaying the sent message. Alternatively, the text data processing module 1020 may only transmit the first voice data to the preset application, and not transmit the first voice data to any other application of the mobile terminal. For example, the text data processing module 1020 may only transmit the first voice data to the preset application through the first dedicated voice data transmission path.

[0091] In response to obtaining the second voice data from the preset application, the voice data processing module 1030 may convert the second voice data into second text data, and display the second text data only in the second area of ​​the text call interface. Here, the second voice data is received by the preset application from the other party of the call, and the second area may represent an area of ​​the text call interface for displaying received messages. Alternatively, the voice data processing module 1030 may obtain the second voice data from the preset application through a second dedicated voice data transmission path, or the voice data processing module 1030 may obtain the second voice data from the preset application through a second dedicated voice data transmission path, and play the second voice data through the speaker of the mobile terminal. In addition, in response to the mute function for the second voice data being enabled, the voice data processing module 1030 may obtain the second voice data from the preset application only through the second dedicated voice data transmission path, and not play the second voice data through the speaker transmitted to the mobile terminal.

[0092] The text call method for a mobile terminal according to an embodiment of the present disclosure can be written as a computer program and stored on a computer-readable storage medium. When the computer program is executed by a processor, the text call method for a mobile terminal as described above can be implemented. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. In one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0093] Fig.11 is a block diagram illustrating an electronic device according to an embodiment of the present disclosure.

[0094] Reference Fig.11 , the electronic device 1100 may include a memory 1101 and a processor 1102. The memory 1101 stores a computer program, and when the computer program is executed by the processor 1102, the text call method for the mobile terminal according to the embodiment of the present disclosure is implemented.

[0095] As an example, the electronic device 1100 may be a smart phone, a tablet device, a personal digital assistant, or other mobile electronic device capable of executing the above-mentioned computer program. In the electronic device 1100, the processor 1102 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller or a microprocessor. As an example and not a limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc. The processor 1102 may run instructions or codes stored in the memory 1101, wherein the memory 1101 may also store data. Instructions and data may also be sent and received through a network via a network interface device, wherein the network interface device may use any known transmission protocol. The memory 1101 may be integrated with the processor 1102, for example, a RAM or flash memory is arranged within an integrated circuit microprocessor, etc. In addition, the memory 1101 may include an independent device, such as an external disk drive, a storage array, or other storage devices that can be used by any database system. The memory 1101 and the processor 1102 may be operatively coupled or may be in communication with each other, for example, via an I / O port, a network connection, etc., such that the processor 1102 can read files stored in the memory.

[0096] In addition, the electronic device 1100 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a touch input device, etc.) All components of the electronic device 1300 may be connected to each other via a bus and / or a network.

[0097] According to the text call method and text call device for a mobile terminal according to the embodiments of the present disclosure, it is possible to ensure that when the text call function is turned on, the text data input by the user is correctly displayed in the user's speech area, and will not be erroneously displayed in the speech area of ​​the other party of the call, thereby avoiding display confusion on the text call interface and improving the user experience.

[0098] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0099] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A text call method for a mobile terminal, characterized in that: The text call method comprises: In response to the text call function being turned on and the preset application of the mobile terminal being in a call state, displaying a text call interface; In response to first text data being input into the text call interface, converting the first text data into first voice data and transmitting the first voice data to the preset application, and displaying the first text data only in a first area of ​​the text call interface, wherein the first voice data will be sent by the preset application to the other party of the call; In response to acquiring second voice data from the preset application, the second voice data is converted into second text data, and the second text data is displayed only in a second area of ​​the text call interface, wherein the second voice data is received by the preset application from the other party of the call.

2. The text communication method according to claim 1, wherein: The text call method further includes: In response to the text call function being turned on, registering a monitoring audio status event with an operating system of the mobile terminal to monitor the audio status of each application of the mobile terminal; In response to monitoring a change in the audio state of the preset application, it is determined that the preset application is in a call state.

3. The text communication method according to claim 1, wherein: Converting the first text data into first voice data and transmitting the first voice data to the preset application includes: The first voice data is transmitted only to the preset application, and the first voice data is not transmitted to any other application of the mobile terminal.

4. The text communication method according to claim 1, wherein: Only the first voice data is transmitted to the preset application through the first dedicated voice data transmission path.

5. The text communication method according to claim 1, wherein: Acquire the second voice data from the preset application via a second dedicated voice data transmission path, or, The second voice data is acquired from the preset application through a second dedicated voice data transmission path, and the second voice data is played through a speaker of the mobile terminal.

6. The text communication method according to claim 5, wherein: In response to the mute function for the second voice data being enabled, the second voice data is acquired from the preset application only through the second dedicated voice data transmission path, and the second voice data is not played through the speaker of the mobile terminal.

7. The text communication method according to claim 4, wherein: In response to determining that the preset application is an application supporting a text call function, establishing a first dedicated voice data transmission path for transmitting the first voice data to the preset application, and / or In response to determining that the preset application is an application supporting a text call function, a second dedicated voice data transmission path is established for acquiring the second voice data from the preset application.

8. The text communication method according to claim 7, wherein: In response to the unique identification information of the preset application being included in the unique identification information list of applications supporting the text call function, determining that the preset application is an application supporting the text call function, Wherein, based on the unique identification information of the application supporting the text call function set by the user, a unique identification information list of the application supporting the text call function is constructed.

9. The text communication method according to claim 7, wherein: The step of establishing a first dedicated voice data transmission path for transmitting the first voice data to the preset application comprises: A first virtual audio device is created, the data stream of the first voice data is routed to the output side of the first virtual audio device, and the recording data stream of the preset application is routed to the input side of the first virtual audio device, thereby establishing a first dedicated voice data transmission path.

10. The text communication method according to claim 7, wherein: The step of establishing a second dedicated voice data transmission path for acquiring the second voice data from the preset application comprises: A second virtual audio device is created, the data stream of the second voice data is routed to the input side of the second virtual audio device, and the playback data stream of the preset application is routed to the output side of the second virtual audio device, thereby establishing a second dedicated voice data transmission path.

11. The text communication method according to claim 1, wherein: The text call method further includes: Create an AudioContinuity module to record at least one of the following parameters: package name information of an application that executes the text call method; unique identification information of an application that supports the text call function; whether to exit the text call function; and whether to enable a mute function for the second voice data while establishing a first dedicated voice data transmission path dedicated to the first voice data and a second dedicated voice data transmission path dedicated to the second voice data.

12. The text communication method according to any one of claims 1 to 11, characterized in that: The first area indicates an area of ​​the text call interface for displaying sent messages, and the second area indicates an area of ​​the text call interface for displaying received messages.

13. A text communication device for a mobile terminal, characterized in that: The text communication device comprises: The display module is configured to: in response to the text call function being turned on and the preset application of the mobile terminal being in a call state, display a text call interface; a text data processing module, configured to: in response to first text data being input into the text call interface, convert the first text data into first voice data and transmit the first voice data to the preset application, and display the first text data only in the first area of ​​the text call interface, wherein the first voice data will be sent by the preset application to the other party of the call; The voice data processing module is configured to: in response to obtaining second voice data from the preset application, convert the second voice data into second text data, and display the second text data only in the second area of ​​the text call interface, wherein the second voice data is received by the preset application from the other party of the call.

14. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the text call method for a mobile terminal as claimed in any one of claims 1 to 11 is implemented.

15. An electronic device, characterized in that: The electronic device comprises: processor; A memory storing a computer program, which, when executed by the processor, implements the text call method for a mobile terminal as claimed in any one of claims 1 to 11.

Images