Instant messaging method and device based on AI caption and storage medium

Abstract

This application provides an AI-based instant messaging method, device, and storage medium, relating to the field of terminal technology. When a terminal device and a peer device establish a connection through an instant messaging application, the terminal device can convert user-inputted text into audio data based on the AI ​​captioning application and send the audio data to the peer device via the instant messaging application. This allows the user on the peer device to hear the corresponding sound, solving the problem of users being unable to speak or having language impairments that prevent communication. Furthermore, the terminal device can also convert audio data from the peer device into text based on the AI ​​captioning application and display the text, allowing the user to see the text. This solves the problem of users with hearing impairments or those in noisy environments being unable to hear the sound from the peer device.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of terminals, and in particular to an instant messaging method based on AI subtitles, a device and a storage medium. BACKGROUND

[0002] With the rapid development of terminal technology, various instant messaging applications have emerged in an endless stream. Most instant messaging applications not only support private communication, but also support group communication and remote office scenarios such as task management.

[0003] At present, when a user uses an instant messaging application (such as a conference application) to communicate online, not only can multiple types of data such as voice, video, pictures, text and emoticons be sent, but also files and marked files can be shared in real time, thereby meeting different communication needs. However, when voice communication or video communication is performed using the instant messaging application, the user can be inconvenient to speak or have language dysfunction, resulting in an inability to exchange messages in a timely manner. SUMMARY

[0004] The present application provides an instant messaging method based on AI subtitles, a device and a storage medium, which solves the technical problem that the user cannot exchange messages in a timely manner when voice communication or video communication is performed using an instant messaging application.

[0005] To achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0006] In a first aspect, the present application provides an instant messaging method based on AI subtitles. The method can be applied to a first device. The first device can include a conference application, an AI subtitle application, an AudioFlinger and a first audio interface. The first audio interface is an audio primary interface or an audio remote processing (remote_submix) interface of a hardware abstraction layer (HAL layer). The method can include: the first device displays an interface of the conference application and a window of the AI subtitle application; the first device receives first text input by a user in the window of the AI subtitle application; the first device writes first audio data to the first audio interface through the AudioFlinger, the first audio data being obtained by converting the first text from text to speech (TTS); in response to a recording request of the conference application, the first device reads the first audio data from the first audio interface through the AudioFlinger; and the first device sends the first audio data to a second device through the conference application. The first device and the second device are terminal devices, such as a mobile phone, a PC or a Pad, and the window of the AI subtitle application can be a floating window.

[0007] In the above scheme, the first device can convert the text input by the user into audio data based on the AI caption application, and send the audio data to the second device through the instant messaging application, and the user using the second device can hear the sound corresponding to the audio data, thereby solving the problem that the user cannot communicate messages because the user is inconvenient to speak in the meeting or the user cannot speak due to language dysfunction.

[0008] In a possible implementation, the method can further include: the first device displaying a sending message in a window of the AI caption application, the sending message including the first text.

[0009] In the above scheme, after the first device successfully sends the first audio data, the first device can display the first text, such as "OK, I know", in the window of the AI caption application, so that the user using the first device can know that the audio message corresponding to the first text "OK, I know" has been successfully sent.

[0010] In a possible implementation, the first audio interface is an audio remote processing (remote_submix) interface, the first device further includes a microphone and a media recorder (MediaRecorder), and the microphone is in a prohibited recording state. In response to the recording request of the conference application, the first device reads the first audio data from the first audio interface through AudioFlinger, which can include: in response to a recording request initiated by the media recorder (MediaRecorder) of the conference application, the first device reads the first audio data from the audio remote processing (remote_submix) interface through the recording thread called by AudioFlinger.

[0011] In the above scheme, when the microphone is in the prohibited recording state, the audio remote processing (remote_submix) interface can be used as the first audio interface without considering the problem that the audio remote processing (remote_submix) interface cannot provide audio write and read services for the microphone. In addition, the first device can convert the text input by the user into audio data based on the AI caption application, and mix the audio data with the sound collected by the microphone, and then send an audio message carrying the mixed audio to the second device. Not only does this solve the problem that the user cannot interact messages because the user is inconvenient to speak in the meeting or the user cannot speak due to language dysfunction, but it also allows other meeting participants to hear the background sound of the phone. It should be understood that the mixing of the background sound makes the sound heard by other meeting participants closer to the sound of the real environment in which the user is located.

[0012] In a possible implementation, the first audio interface is an audio primary interface, and the first device further includes a microphone and a media recorder (MediaRecorder), and the microphone is in an allowed recording state. The method can further include: the first device writing second audio data to the audio primary interface, the second audio data being audio data collected by the microphone. Accordingly, in response to the recording request of the conference application, the first device reading the first audio data from the first audio interface by the AudioFlinger can include: in response to the recording request initiated by the media recorder (MediaRecorder) of the conference application, the first device reading mixed data of the first audio data and the second audio data from the audio primary interface by the AudioFlinger calling a recording thread. The first device sending the first audio data to the second device by the conference application can include: the first device sending the mixed data of the first audio data and the second audio data to the second device by the conference application.

[0013] In the foregoing scheme, when the microphone is in the allowed recording state, the performance attribute of the audio remote processing (remote_submix) interface determines that the audio remote processing (remote_submix) interface cannot provide audio write and read services for the microphone, and the performance attribute of the audio primary interface determines that the audio primary interface can provide audio write and read services for the microphone. In this case, the first device can use the audio primary interface to provide audio write and read services for the first audio data and the second audio data.

[0014] In a possible implementation, the first device can further include a second audio interface, and the second audio interface is an audio primary interface or an audio remote processing (remote_submix) interface of a hardware abstraction layer. The method can further include: the first device receiving third audio data from the second device by the conference application; the first device writing the third audio data to the second audio interface by the AudioFlinger; in response to a playback request of an AI caption application, the first device reading the third audio data from the second audio interface by the AudioFlinger; and the first device displaying a receiving message in a window of the AI caption application, the receiving message including second text obtained by performing automatic speech recognition on the third audio data.

[0015] In the foregoing scheme, the first device can convert audio data from the second device into text based on the AI caption application, and display the text, so that a user of the mobile phone can see the text, thereby solving the problem that the user's hearing impairment causes the user to be unable to hear the sound from the other user.

[0016] In a possible implementation, the second audio interface is an audio remote_submix interface, and the first device further includes a speaker, and the speaker is in a state of being prohibited to sound.

[0017] In the above solution, when the speaker is in the state of being prohibited to sound, the audio remote_submix interface can be used as the second audio interface without considering the problem that the audio remote_submix interface cannot provide audio write and read services for the speaker.

[0018] In a possible implementation, the second audio interface is an audio primary interface, and the first device further includes a speaker, and the speaker is in a state of being allowed to sound. The method further includes: reading, by the first device, third audio data from the audio primary interface; and outputting, by the first device, sound corresponding to the third audio data through the speaker.

[0019] In the above solution, when the speaker is in the state of being allowed to sound, the performance attribute of the audio remote_submix interface determines that the audio remote_submix interface cannot provide audio write and read services for the speaker, and the performance attribute of the audio primary interface determines that the audio primary interface can provide audio write and read services for the speaker, and therefore the audio primary interface can be used as the second audio interface. In addition, the first device can display text and output audio at the same time, so that the user can view the text in a noisy environment and listen to the sound in a quiet environment, thereby improving the diversity of the user's information acquisition mode.

[0020] In a possible implementation, the first device further includes an audio manager (AudioManager). The method further includes: based on the state of the microphone and the speaker of the first device, formulating, by the first device, an audio policy for the AudioFlinger through the audio manager (AudioManager). If the audio policy is a first audio policy, the first audio interface and the second audio interface are audio remote_submix interfaces; if the audio policy is a second audio policy, the first audio interface is an audio remote_submix interface, and the second audio interface is an audio primary interface; and if the audio policy is a third audio policy, the first audio interface and the second audio interface are audio primary interfaces.

[0021] In the foregoing solution, the audio manager (AudioManager) is responsible for the policy selection of the audio device switching. The AudioFlinger is an executor of the audio system policy, and is responsible for the management of the audio stream device and the processing and transmission of the audio stream data. The audio manager (AudioManager) controls the audio paths of the microphone and the loudspeaker by instructing the AudioFlinger to indicate different audio policies, so as to meet the audio use requirements of the user in different conference scenarios.

[0022] In a possible implementation, the first device can further include a media player (MediaPlayer), and the writing, by the first device, of the third audio data to the second audio interface through the AudioFlinger can include that the first device invokes the AudioFlinger to write the third audio data to the second audio interface through the media player (MediaPlayer).

[0023] In a possible implementation, the first device can further include a media player (MediaPlayer), and the writing, by the first device, of the first audio data to the first audio interface through the AudioFlinger can include that the first device invokes the AudioFlinger to write the first audio data to the first audio interface through the media player (MediaPlayer).

[0024] In a second aspect, an instant messaging method based on an AI caption is provided. The method can be applied to a first device. The first device can include a conference application and an AI caption application. The method can include: displaying, by the first device, an interface of the conference application; in response to a first operation of a user, displaying, by the first device, a window of the AI caption application on the interface of the conference application; receiving, by the first device, a first text input by the user in the window of the AI caption application; sending, by the first device, first audio data to a second device through the conference application, the first audio data being obtained by converting the first text from text to speech; and displaying, by the first device, a sending message in the window of the AI caption application, the sending message including the first text.

[0025] In the foregoing solution, the first device can convert the text input by the user into audio data based on the AI caption application, and send the audio data to the second device through the instant messaging application. A user of the second device can hear the sound corresponding to the audio data, thereby solving the problem that the user in the conference cannot conveniently speak or cannot communicate messages due to language dysfunction.

[0026] In a possible implementation, the first device can further include a microphone. The method can further include: obtaining, by the first device, second audio data, the second audio data being audio data collected by the microphone. Accordingly, the sending, by the first device, of the first audio data to the second device via the conference application can include: sending, by the first device, mixed audio data of the first audio data and the second audio data to the second device via the conference application.

[0027] In the above solution, the first device can convert the text input by the user into audio data based on the AI caption application, mix the audio data with the sound collected by the microphone, and then send an audio message carrying the mixed audio to the second device. Not only does this solve the problem of users who are in a meeting and cannot speak conveniently, or users who have language disorders and cannot speak, but also enables other participants to hear the background sound of the phone. It should be understood that the mixing of the background sound makes the sound heard by the other participants closer to the sound in the real environment of the user.

[0028] In a possible implementation, the method can further include: receiving, by the first device, third audio data from the second device via the conference application; and displaying, by the first device, a received message in a window of the AI caption application, the received message including second text obtained by performing automatic speech recognition on the third audio data.

[0029] In the above solution, the first device can convert the audio data from the second device into text based on the AI caption application, and display the text, so that the user of the phone can see the text, thereby solving the problem of users who have impaired hearing and cannot hear the sound from the other user.

[0030] In a third aspect, the present application provides a device, which includes units for performing the method of the first aspect or the second aspect. The device can correspond to the AI caption-based instant messaging method described in the first aspect or the second aspect, and the related description of the units in the device is referred to the description of the first aspect or the second aspect. For brevity, it will not be repeated here.

[0031] The method described in the first aspect or the second aspect can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, processing modules or units, display modules or units, and the like.

[0032] Fourthly, this application provides a terminal device including a memory and one or more processors. The memory stores computer program code, which includes computer instructions. When the computer instructions are invoked by the processor, the terminal device executes the AI-based instant messaging method provided in either the first or second aspect.

[0033] Fifthly, this application provides a computer-readable storage medium. The computer-readable storage medium includes computer instructions. When the computer instructions are executed on a terminal device, the terminal device causes the terminal device to perform the AI-based instant messaging method provided by any possible implementation of the first or second aspect.

[0034] Sixthly, this application provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform an AI-based instant messaging method as provided in any possible implementation of the first or second aspect.

[0035] In a seventh aspect, this application provides a chip system. The chip system includes one or more interface circuits and one or more processors. The interface circuits and processors are interconnected via lines. The chip system can be applied to a terminal device including a communication module and a memory. The interface circuits are used to receive signals from the terminal device's memory and send the received signals to the processor, the signals including computer instructions stored in the memory. When the processor invokes the computer instructions, the terminal device can execute an AI-based instant messaging method as provided in any possible implementation of the first or second aspect.

[0036] It is understood that the beneficial effects achieved by the apparatus of the third aspect, the terminal device of the fourth aspect, the computer-readable storage medium of the fifth aspect, the computer program product of the sixth aspect, and the chip system of the seventh aspect can be referred to the beneficial effects of any possible implementation of the first or second aspect, which will not be repeated here. Attached Figure Description

[0037] Figure 1 A schematic diagram of the interface for AI subtitles in a video scene provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of the interface for ordinary subtitles in a conference scenario provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of an interface for enabling AI subtitles provided in an embodiment of this application;

[0040] Figure 4 Three other schematic diagrams illustrating the interfaces for enabling AI subtitles provided in the embodiments of this application;

[0041] Figure 5 One of the scene schematic diagrams using AI subtitles provided for the embodiments of the present application;

[0042] Figure 6 The second scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0043] Figure 7 The third scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0044] Figure 8 The fourth scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0045] Figure 9 The fifth scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0046] Figure 10 The sixth scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0047] Figure 11 The seventh scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0048] Figure 12 The eighth scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0049] Figure 13 The ninth scene schematic diagram using AI subtitles provided for the embodiments of the present application;

[0050] Figure 14 The system architecture schematic diagram of a terminal device provided for the embodiments of the present application;

[0051] Figure 15 The module interaction diagram based on AI subtitles provided for the embodiments of the present application;

[0052] Figure 16 The corresponding specific method flow schematic diagram; Figure 15

[0053] Figure 17 Another module interaction diagram based on AI subtitles provided for the embodiments of the present application;

[0054] Figure 18 The corresponding specific method flow schematic diagram; Figure 17

[0055] Another module interaction diagram based on AI subtitles provided for the embodiments of the present application; Figure 19

[0056] Figure 20 ​​For the purpose of Figure 19 The corresponding specific method flowchart;

[0057] Figure 21 The module interaction diagram corresponding to the parameter distribution strategy of the downlink data provided by the embodiment of the present application;

[0058] Figure 22 For the purpose of Figure 21 The corresponding specific method flowchart;

[0059] Figure 23 The module interaction diagram corresponding to the parameter distribution strategy of the uplink data provided by the embodiment of the present application;

[0060] Figure 24 For the purpose of Figure 23 The corresponding specific method flowchart;

[0061] Figure 25 The hardware structure diagram of the terminal device provided by the embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments.

[0063] In the description of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B. In the description of the present application, "and / or" is only a description of the association relationship between the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can mean that there are three cases of A alone, A and B together, and B alone.

[0064] In the description and claims of the present application, the terms "first" and "second" and the like are used to distinguish different objects or to distinguish different treatments of the same object, not to describe a specific order of the object. For example, the first operation and the second operation are used to distinguish different operations, not to describe a specific order of the operation. In the embodiments of the present application, "a plurality of" means two or more.

[0065] At present, some mobile phones have launched AI caption function. When playing online video on mobile phone, AI caption application can convert the audio stream corresponding to online video into text based on AI technology, and display it in the form of subtitles on the screen. For example, Figure 1(a) of FIG. 1, if a user using a mobile phone has a hearing impairment, the user cannot hear the sound when the mobile phone plays a video advertisement picture 01, causing inconvenience to the user's life. In this case, the user can trigger the mobile phone to pull up an AI caption floating window 02 through a shortcut operation, as shown in (b) of FIG. 1. The AI caption floating window 02 can be used to display the text obtained after the conversion of the audio stream. In this way, even a disabled person with a hearing impairment can obtain specific information of the video advertisement through the AI caption floating window 02. However, the existing AI caption function is limited to video applications and only supports the conversion of an audio stream into text. Figure 1

[0066] In addition to video applications, there is a demand for using AI captions in instant messaging applications. The instant messaging application is a kind of terminal service that allows two or more people to instantly deliver text messages, picture files, voice communication, and video communication based on a network. The instant messaging application can be divided into enterprise instant messaging applications and website instant messaging applications according to the use purpose. For example, the instant messaging application can be a conference application or a voice application. When multiple devices communicate online based on the instant messaging application (such as a conference application), the multiple devices can send each other various types of data such as online voice, online video, video files, picture files, text, and emoticons, and can also share files and mark files in real time, thereby meeting different communication needs of users.

[0067] Exemplarily, as shown in (a) of FIG. 1, a user can click an icon 03 of a conference application in a desktop. The conference application can be a third-party application or a system application. In response to the click operation of the user, the mobile phone displays a conference home page as shown in (b) of FIG. 1. The conference home page includes a join conference 04, a quick conference, and a shared screen, and the like. For example, the user can click the join conference 04, and the mobile phone updates and displays a conference interface as shown in (a) of FIG. 2. Figure 2 Figure 2 Figure 2 ​​​The joining conference interface includes a conference ID 05, a participant name 06, a speaker control 07, a microphone control 08, a camera control 09, a joining conference control 10, and the like. The conference ID 05 is used to input the account of the current conference, for example, the conference ID 05 is “11111111111”. The participant name 06 is used to set the name of the participant, for example, the participant name 06 is “Wang Yiyi”. The speaker control 07 is used to turn on or off the speaker, for example, when the speaker control 07 is slid to the left, the speaker is turned off, or when the speaker control 07 is slid to the right, the speaker is turned on. The microphone control 08 is used to turn on or off the microphone, for example, when the microphone control 08 is slid to the left, the microphone is turned off, or when the microphone control 08 is slid to the right, the microphone is turned on. The camera control 09 is used to turn on or off the camera, for example, when the camera control 09 is slid to the right, the camera is turned on, or when the camera control 09 is slid to the left, the camera is turned off. The joining conference control 10 is used to confirm joining the conference after the user completes the settings. In response to the user's click operation on the joining conference control 10, the phone displays the conference interface 11 as shown in (d) of FIG. 1. Figure 2 The conference interface 11 includes the head portrait of the participant, a mute release control, a video start control, a screen sharing control, a member management control, an end conference control, a subtitle control 12, and the like. It should be noted that the subtitle control 12 is only a functional entrance for the user to input chat content, and does not have the AI subtitle function, such as converting audio stream into text, so it is called ordinary subtitle, not AI subtitle. If the user clicks the subtitle control 12, the phone displays the chat window 13 as shown in (e) of FIG. 1. Figure 2 The user can input the text “no problem” 14 in the chat window 13 through the virtual keyboard, and send the text message “no problem” to other participants. If the user closes the chat window 13, the phone can display the message “no problem” in the form of subtitles 15 in the conference interface 11, so that each participant can see the subtitle message.

[0068] However, when multiple users use an instant messaging application for voice call or video call, the users can not exchange messages in time in any of the following scenarios: scenario 1, one user is in a meeting and the user is inconvenient to talk; scenario 2, one user has language dysfunction and the user cannot talk; scenario 3, one user has hearing impairment and the user cannot hear the sound from another user; scenario 4, the environment where one user is currently located is too noisy and the user cannot hear the sound from another user. It can be seen that there is a demand for AI caption for instant messaging application, but the prior art has not applied AI caption to instant messaging application. In view of these problems, the present application provides an instant messaging method based on AI caption. The method can be applied to a terminal device, which can provide AI caption service when a user uses an instant messaging application. In the case that the terminal device and a peer device have established a connection through an instant messaging application, the terminal device can convert the text input by the user into audio data based on an AI caption application, and send the audio data to the peer device, so that the user using the peer device can hear the sound corresponding to the audio data, thereby solving the problems existing in scenarios 1 and 2 described above. In addition, the terminal device can also convert the audio data from the peer device into text based on the AI caption application, and display the text, so that the user using the terminal device can see the text, thereby solving the problems existing in scenarios 3 and 4 described above.

[0069] In some embodiments, the terminal device described above is also referred to as a terminal or a user equipment (UE). For example, the terminal device can be a personal computer (PC), a mobile phone, a smart screen, a smart television, a tablet computer (Pad), a wearable device, a computer with wireless transceiver function, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, or a wireless terminal in smart home, etc., or can be other devices with display and audio playback functions.

[0070] In the following, the terminal device is taken as an example of a mobile phone, and the description is combined with the following figures. Figures 3 to 13The application scenario of the instant messaging method based on AI caption provided in the application is exemplarily illustrated.

[0071] Scenario example 1

[0072] In some embodiments, the user can turn on the AI caption function through the settings application of the mobile phone.

[0073] Exemplarily, as shown in (a) of FIG. 1, Figure 3 The desktop can include icons of application programs such as honor video, sports health, weather, browser, settings and smart life. The user can trigger the mobile phone to display the operation interface of the application program corresponding to any icon by clicking the icon. For example, the user can click the settings 16.

[0074] After receiving the click operation of the user on the settings 16, the mobile phone displays the settings interface as shown in (b) of FIG. 1. Figure 3 The settings interface includes options such as Bluetooth, wireless local area network, phone, flight mode, health use of mobile phone, smart assistant 17 and auxiliary function. Each option corresponds to a system-level service. The user can trigger the mobile phone to display the service setting interface corresponding to the option by clicking the option. For example, the user can click the smart assistant 17.

[0075] After receiving the click operation of the user on the smart assistant 17, the mobile phone displays the smart assistant interface as shown in (c) of FIG. 1. Figure 3 The smart assistant interface includes YOYO suggestion, negative screen, honor search and honor vision. In addition, the smart assistant interface can also include other smart functions, such as smart voice 18, smart perception, smart screen recognition, etc. The user can trigger the mobile phone to display the setting interface corresponding to the smart function by clicking the smart function. For example, the user can click the smart voice 18.

[0076] After receiving the click operation of the user on the smart voice 18, the mobile phone displays the smart voice interface as shown in (d) of FIG. 1. Figure 3 The smart voice interface provides voice services such as AI caption 19 and voice assistant. Among them, the AI caption 19 is used to set the AI caption, and the voice assistant is used to set the voice assistant. For example, the user can click the AI caption 19.

[0077] After receiving the click operation of the user on the AI caption 19, the mobile phone displays the AI caption interface as shown in (e) of FIG. 1. Figure 3The AI ​​captioning interface is shown in (e). The AI ​​captioning interface includes an AI captioning control 20. When the AI ​​captioning control 20 is slid to the left, the phone disables the AI ​​captioning function; when it is slid to the right, the phone enables the AI ​​captioning function. The AI ​​captioning interface also includes options such as writing mode, font size, desktop shortcut, and background opacity. The desktop shortcut includes an add control 21. When the user clicks add control 21, the phone creates a shortcut on the desktop or the negative one screen. Figure 4 The shortcut icon 29 shown in (d) allows users to quickly bring up the floating window for AI subtitles on their phones by operating the shortcut icon 29.

[0078] like Figure 3 As shown in (f), when the AI ​​caption control 20 is slid to the right, the phone activates the AI ​​caption function and displays the AI ​​caption floating window 22. The bottom area of ​​the floating window 22 includes an input box 23 for editing text, a preset word control 24 for quick reply, a mute control 25 for turning the microphone on or off, and a send control 26 for confirming the text in the input box 23. The top area of ​​the floating window 22 includes a close control 27 for canceling the floating window 22, as well as more options. It should be understood that the floating window 22 is merely an illustrative example and does not limit this application. In actual implementation, the floating window 22 may include more or fewer controls, and the position, size, and style of each control in the floating window 22 may be adjusted.

[0079] In some embodiments, when a user triggers the phone to exit the settings interface and switch to another interface, such as switching from the settings interface to the interface of an instant messaging application, a video application, or a call application, the floating window 22 can be overlaid on the application interface after the switch. That is, the phone will always display the floating window 22 until the user clicks the close control 27.

[0080] In other embodiments, users can also change the position of the floating window 22 on the screen by dragging it, such as moving the floating window 22 to the top or bottom of the screen. Additionally, users can change the size of the floating window 22 by manipulating its borders, such as increasing or decreasing its size.

[0081] In other embodiments, users can also change the transparency of the floating window 22 by manipulating the background opacity option in the AI ​​caption interface. For example, when the background opacity is set to 100%, the background of the floating window 22 is fully visible, thus not obscuring the application interface below the floating window 22. Alternatively, when the background opacity is set to 0%, the background is completely invisible, leaving only the text above the application interface, which essentially does not obscure the application interface below the floating window 22.

[0082] The above embodiments are described by taking the example that the user triggers the phone to start the AI caption function by operating the settings application. The application is not limited thereto. In actual implementation, the user can also trigger the phone to quickly pull up the AI caption floating window 22 in other ways. As an example, as shown in (a) of FIG. 13, when the phone displays the conference interface 11, the user can slide the finger downward from the top of the screen. In response to the sliding operation, the phone displays a control center list as shown in (b) of FIG. 13. Figure 4 Figure 4 The control center list includes WLAN control, Bluetooth control, mobile data control, AI caption control 28, flight mode control, and brightness adjustment control, etc. If the user clicks the AI caption control 28, the phone displays the AI caption floating window 22 above the conference interface 11 as shown in (c) of FIG. 13. As another example, as shown in (d) of FIG. 13, the user can also trigger the phone to quickly pull up the AI caption floating window 22 by clicking the shortcut icon 29 on the desktop. As yet another example, as shown in (e) of FIG. 13, the user can also trigger the phone to quickly pull up the AI caption floating window 22 by saying “start AI caption” to the phone through the voice assistant. Figure 4 Figure 4 Figure 4

[0083] In the above embodiments, the user can pull up the AI caption floating window in multiple ways such as through the settings application, the shortcut icon on the desktop, the AI caption control of the control center, or the voice assistant. In addition to the above ways, the phone can also pull up the AI caption floating window in other ways. In this way, the AI caption can provide the AI caption function for the user during the conference, such as converting the text message to be sent into an audio message, or converting the received audio message into a text message.

[0084] Scenario Example 2

[0085] In some embodiments, after the phone displays the AI caption floating window 22, the user can manually input text in the floating window 22, and the phone converts the text into audio and then sends an audio message carrying the audio to the device used by other conference participants.

[0086] For example, during the conference, when the user is inconvenient to speak, the user can click the input box 23 of the floating window 22 as shown in (a) of FIG. 14. In response to the clicking operation, the phone displays a virtual keyboard 30 in the lower area of the floating window 22 as shown in (b) of FIG. 14. Figure 5 Figure 5 Figure 5 ​​​​​​As shown in (c), in Chinese mode, the user can select and input virtual keys on the virtual keyboard 30, and the phone will display the pinyin corresponding to the selected virtual key in the input box 23, such as "haode". Figure 5 As shown in (d), after editing the text "Okay, I understand," the user clicks the send control 26. In response to the click on the send control 26, the phone converts the text "Okay, I understand" into audio and sends an audio message carrying that audio to the devices used by other participants. On the one hand, as... Figure 5 As shown in (e), after successfully sending the audio message, the phone can display the text "OK, I understand" 32 in the floating window 22, so that the user of the phone can know that the audio message corresponding to the text "OK, I understand" has been successfully sent. On the other hand, after other participants' devices successfully receive the audio message, these devices can play the audio through speakers, earpieces or headphones, or they can convert the audio into the text "OK, I understand" based on AI subtitles and display it.

[0087] In the above embodiments, the mobile phone can convert the text entered by the user into audio data based on the AI ​​captioning application and send the audio data to the other device. The user of the other device can then listen to the audio corresponding to the audio data, thereby solving the problem that users are unable to speak during a meeting or that users with language impairments cannot communicate.

[0088] Scenario Example 3

[0089] Example 2 above illustrates the process of converting text entered by a user in a floating window into audio and sending an audio message carrying that audio directly to the devices used by other participants. In some embodiments, the mobile phone can convert the text entered by the user in the floating window into audio, mix it with the sound captured by the microphone, and then send an audio message carrying the mixed audio to the devices used by other participants.

[0090] For example, such as Figure 6 As shown in (a), the mute control 25 in the floating window 22 is in mute mode, indicating that the microphone is in a muted recording state, meaning the microphone will not collect sound. If a user wants other participants to hear the background noise of their environment, they can click the mute control 25; or, when multiple users are using the same phone to participate in the meeting but one user is unable to speak, that user can click the mute control 25. After receiving the user's click on the mute control 25, the mute control 25 will... Figure 6 The mute form shown in (a) is changed to as follows: Figure 6The non-mute mode shown in (b) indicates that the phone's microphone is also enabled. As an example, such as... Figure 6 As shown in (b), after the microphone is turned on, the phone can also display a message "Microphone is turned on" on the screen; as another example, the phone can also not display a message.

[0091] like Figure 6 As shown in (e), the microphone can record the background noise from the phone's local memory to obtain audio 2. Meanwhile, as... Figure 6 As shown in (d), the user can type the text "OK, I understand" using the virtual keyboard and then click the send control 26. The phone can first convert the text "OK, I understand" into audio 1, and then mix audio 1 and audio 2 to obtain audio 3. Then, the phone sends an audio message carrying audio 3 to the devices used by other participants. On the one hand, as... Figure 6 As shown in (f), after successfully sending the audio message carrying audio 3, the mobile phone can display the text "OK, I understand" 34 in the floating window 22, so that the user of the mobile phone can know that audio 3 has been successfully sent. On the other hand, after other participants' devices successfully receive the audio message carrying audio 3, these devices can play audio 3 through speakers, earpieces or headphones, or they can extract "OK, I understand" from audio 3 based on AI captions and display it in text form.

[0092] In other embodiments, the meeting interface 11 also includes a mute control 33, which allows users to activate or deactivate the microphone by operating the mute control 33 on the meeting interface 11, thereby changing the audio mode of the AI ​​captioning application.

[0093] For example, such as Figure 6 As shown in (c), the mute control 33 in the meeting interface 11 is in mute mode, indicating that the microphone is in a muted recording state. If a user wants other participants to hear the background noise of their environment, the user can click the mute control 33. After receiving the user's click operation on the mute control 33, the mute control 33 will... Figure 6 The mute mode shown in (c) is changed to as follows: Figure 6 The non-mute mode shown in (b) is configured, and the microphone is enabled. Referring to the description of the above embodiment, the mobile phone can convert the text entered by the user through the virtual keyboard into audio, mix it with the sound captured by the microphone, and then send an audio message carrying the mixed audio to the devices used by other participants, which will not be elaborated here.

[0094] In other embodiments, referring to the description of the above embodiments, the conference interface 11 and the floating window 22 are respectively provided with a mute control, and the mute control includes a mute form and a non-mute form. In order to avoid logical conflicts caused by user operations, before the user triggers the phone to pull up the floating window 22 through setting applications, desktop shortcuts, AI caption controls of control centers, voice assistants and the like, the phone can detect the state of the microphone and set the display form of the mute control 25 in the floating window 22 according to the state of the microphone. Specifically, if the microphone is in the off state, the mute control 25 in the floating window 22 is set to the mute form as shown in (a) of Figure 6 , indicating that the microphone is in the prohibited recording state; if the microphone is in the on state, the mute control 25 in the floating window 22 is set to the non-mute form as shown in (b) of Figure 6 , indicating that the microphone is in the allowed recording state. It should be understood that, as shown in (c) of Figure 6 , the setting of the microphone has been completed before entering the conference, so after entering the conference, the display form of the mute control 33 is consistent with the state of the microphone. Finally, the display forms of the mute control 25 and the mute control 33 are consistent with the state of the microphone.

[0095] In other embodiments, after pulling up the floating window 22, if the phone receives user operations on the mute control 33 of the conference interface 11 or receives user operations on the mute control 25 of the floating window 22, the phone can notify the conference application, the AI caption application to change the display form of the mute control, such as setting both mute controls to the non-mute form, so that the display of the mute control 33 of the conference interface 11 or the mute control 25 of the floating window 22 remains consistent, facilitating the user to understand the true state of the microphone at the current time.

[0096] In other embodiments, after pulling up the floating window 22, the phone can notify the conference application to set the mute control 33 of the conference interface 11 to an unavailable state (i.e., not allowing the user to turn on or turn off the microphone through operations on the mute control 33), and only the mute control 25 of the floating window 22 is in an available state (i.e., allowing the user to turn on or turn off the microphone through operations on the mute control 25). In this case, the user can only operate the mute control 25 of the floating window 22, but cannot operate the mute control 33 of the conference interface 11, thereby avoiding logical conflicts caused by user operations.

[0097] In the above embodiments, the mobile phone can convert the text entered by the user into audio data based on an AI captioning application, mix this audio data with the sound captured by the microphone, and then send an audio message carrying the mixed audio to the devices used by other participants. This not only solves the problems of users being unable to speak during a meeting or users with language impairments being unable to interact with messages, but also allows other participants to hear the background noise from the mobile phone. The inclusion of background noise makes the sound heard by other participants closer to the sound of the user's real environment.

[0098] Scenario Example 4

[0099] In some embodiments, after the floating window 22 displaying AI subtitles on the mobile phone, if the mobile phone receives an audio message sent by a device used by another participant, the mobile phone can convert the audio message into a text message through the AI ​​subtitle application and display the text message in the floating window 22.

[0100] For example, such as Figure 2 As shown in (a), speaker 1 and speaker 2 are located on the top and bottom bezels of mobile phone 1. The conference interface 11 also includes a playback control 35, which is in the form of an earpiece, indicating that speaker 1 and speaker 2 are muted. If mobile phone 1 receives audio data 1 from mobile phone 2, then mobile phone 1 uses an AI captioning application to convert audio data 1 into the text "Departing at 9:00 tomorrow" and displays the text "Departing at 9:00 tomorrow" in the floating window 22. Since speaker 1 and speaker 2 are muted at this time, mobile phone 1 cannot play audio data 1 through speaker 1 and speaker 2.

[0101] like Figure 7 As shown in (b), the user can click the playback control 35 on the conference interface 11. In response to this click, the mobile phone 1 changes the playback control 35 as follows: Figure 7 The speaker configuration is shown in (c) in the image, and a prompt message "Sound will be played from the speaker" is displayed on the screen. This prompt message informs the user that speaker 1 and speaker 2 have been successfully turned on by phone 1. After speaker 1 and speaker 2 are turned on, if phone 1 receives audio data 2 from phone 2, then phone 1 uses an AI captioning application to convert the audio data 2 into the text "Gather at the company entrance," and displays it as shown in the image. Figure 7 The floating window 22 shown in (d) displays the text "Gather at the company entrance" 38. Additionally, since speakers 1 and 2 are currently in the playback-enabled state, therefore... Figure 7 As shown in (d), mobile phone 1 can also play the audio "Gather at the company entrance" through speaker 1 and speaker 2.

[0102] It should be noted that the aboveFigure 7 is exemplarily illustrated by taking the example of opening the loudspeaker in the conference by the user operation, which does not limit the present application. Referring to Figure 7 (c), if the user slides the microphone control 08 to the right in the joining conference interface, the microphone is in the open state when the phone 1 enters the conference interface 11, and the playback control 35 is in the form of the loudspeaker as shown in Figure 2 (c). In this case, the phone 1 can not only display the text corresponding to the received audio data, but also play the audio data through the loudspeaker.

[0103] In other embodiments, the user can also click the playback control 35 as shown in Figure 7 (c), so that the playback control 35 changes from the loudspeaker form to the earpiece mode as shown in Figure 7 (b). In this case, the loudspeaker is prohibited to output the audio data from the phone 2.

[0104] In the above embodiments, the phone can convert the audio data from the opposite device into text based on the AI caption application and display the text, so that the user using the phone can see the text, solving the problem that the user cannot hear the sound from the user of the other party due to impaired hearing. In addition, the phone can display the text and output the audio, so that the text is viewed in a noisy environment and the sound is listened to in a quiet environment, thereby improving the diversity of the user's information acquisition method.

[0105] Scenario Example 5

[0106] The above scenario example 4 is exemplarily illustrated by taking the example of controlling the opening or closing of the loudspeaker through the operation of the playback control 35 of the conference interface 11. In some embodiments, the phone can also control the opening or closing of the loudspeaker through the pressing operation of the “volume +” button and the “volume -” button arranged on the side frame.

[0107] As an example, the user can press the “volume +” button 39 as shown in Figure 7 (a). In response to the pressing operation of the “volume +” button 39 by the user, if the loudspeaker 1 and the loudspeaker 2 are not opened, the phone 1 opens the loudspeaker 1 and the loudspeaker 2, and displays the prompt information “the sound will be played from the loudspeaker” 37 as shown in Figure 8 (b) in the screen; if the loudspeaker 1 and the loudspeaker 2 are already opened, the phone 1 increases the volume of the loudspeaker 1 and the loudspeaker 2. In the case that the phone 1 has opened the loudspeaker 1 and the loudspeaker 2, if the phone 1 receives the audio data 2 from the phone 2, the phone 1 converts the audio data 2 into the text “meet at the company gate” through the AI caption application, and displays the text as shown in Figure 8As shown in (c) of FIG. 22, the phone 1 displays the text "Meet at the company gate" 38 in the floating window 22. Since the speaker 1 and the speaker 2 are in the sound playing enabled state at this moment, the phone 1 can also play the audio "Meet at the company gate" through the speaker 1 and the speaker 2.

[0108] As another example, as shown in (d) of FIG. 22, the user can press the "Volume - " button 40. In response to the user's pressing operation on the "Volume - " button 40, if the speaker 1 and the speaker 2 have been turned on, the phone 1 turns off the speaker 1 and the speaker 2, and displays the prompt information "Sound will be turned off" 41 as shown in (e) of FIG. 22. In the case where the phone 1 has turned off the speaker 1 and the speaker 2, if the phone 1 receives the audio data 2 from the phone 2, the phone 1 converts the audio data 2 into the text "Meet at the company gate" through the AI caption application, and displays the text "Meet at the company gate" 38 in the floating window 22 as shown in (f) of FIG. 22. Since the speaker 1 and the speaker 2 are in the sound playing disabled state at this moment, the phone 1 does not need to play any audio through the speaker 1 and the speaker 2. Figure 8 Figure 8 Figure 8

[0109] In the above embodiment, when the user presses the "Volume +" button 39, it is explained that the user can want to increase the volume, at which time the speaker can be turned on, and the audio from other devices can be input through the speaker, and the text corresponding to the audio data can be displayed through the floating window 22; when the user presses the "Volume - " button 40, it is explained that the user can want to decrease the volume, at which time the speaker can be turned off, and the audio from other devices can be disabled to be input through the speaker, and only the text corresponding to the audio data can be displayed through the floating window 22.

[0110] Scenario Example 6

[0111] In some embodiments, the floating window 22 can also include a sound playing control. The user can control the turning on or turning off of the speaker through the operation of the sound playing control.

[0112] As an example, as shown in (a) of FIG. 23, the sound playing control 42 is in the prompt turning on form, indicating that the speaker is in the sound playing disabled state, i.e., the speaker will not output sound. The user can press the sound playing control 42. In response to the user's pressing operation on the sound playing control 42, the phone 1 turns on the speaker 1 and the speaker 2, and displays the prompt information "Sound will be turned on" 43 in the screen as shown in (b) of FIG. 23. Figure 8 Figure 9 ​​​​The prompt message "Sound will be played from the speaker" 37 shown in (b) indicates that the speaker is in a play-enabled state, meaning the speaker can output sound. With phone 1 already turned on (1) and speaker 2 already on, if phone 1 receives audio data (2) from phone 2, then phone 1 uses an AI captioning application to convert the audio data (2) into the text "Gather at the company entrance," and displays it as shown in (b). Figure 9 As shown in (c), the text "Gather at the company entrance" is displayed in the floating window 22. Since speakers 1 and 2 are in the playback enabled state at this time, mobile phone 1 can also play the audio "Gather at the company entrance" through speakers 1 and 2.

[0113] As another example, such as Figure 9 As shown in (d), the playback control 42 is in a closed state, indicating that the speaker is in the playback enabled state, that is, the speaker can output sound. The user can press the playback control 42. In response to the user's pressing operation of the playback control 42, the phone 1 turns off speaker 1 and speaker 2, and displays as shown on the screen. Figure 9 The prompt message "Sound will be turned off" 41 shown in (e) is displayed, and the playback control 42 is changed to an on prompt, indicating that the speaker is in a muted state, meaning the speaker will not output sound. With phone 1 already turned off 1 and speaker 2, if phone 1 receives audio data 2 from phone 2, then phone 1 uses an AI captioning application to convert the audio data 2 into the text "Gather at the company entrance," and displays it as shown in (e). Figure 9 As shown in (f), the text "Gather at the company entrance" is displayed in the floating window 22. Since speakers 1 and 2 are currently muted, phone 1 does not need to play any audio through speakers 1 and 2.

[0114] In the above embodiments, the floating window 22 may further include a playback control 42. Users can activate or deactivate the phone's speaker by operating the playback control 42. Specifically, when the speaker is on, the phone can input audio from other devices through the speaker and display text corresponding to the audio data through the floating window 22; when the speaker is off, the phone can prevent audio from other devices from being output through the speaker and only display text corresponding to the audio data through the floating window 22.

[0115] It should be noted that the above scenario examples 4, 5, and 6 are merely illustrative. In actual implementation, the speaker can be turned on or off in other ways. As another example, phone 1 can also automatically turn the speaker on or off based on the current volume. For instance, when phone 1 detects that the current system volume is greater than or equal to a preset volume, it turns on the speaker. In this way, the phone can not only input audio from other devices through the speaker, but also display text corresponding to the audio data through the floating window 22. When phone 1 detects that the current system volume is less than the preset volume, it turns off the speaker. In this way, the phone prohibits the output of audio from other devices through the speaker, and only displays text corresponding to the audio data through the floating window 22.

[0116] Scenario Example 7

[0117] In some embodiments, the floating window 22 may also include a preset word control 24. Users can quickly reply with preset words to other devices by operating the preset word control 24.

[0118] For example, such as Figure 9 As shown in (a), the floating window 22 may also include a preset word control 24. During the meeting, the user can click on the preset word control 24. In response to the click operation on the preset word control 24, the mobile phone displays as shown in (a). Figure 10 The preset word box is shown in (b) above. The preset word box includes multiple preset words, such as "I'm busy now, will reply later," "No problem," "Currently being processed urgently," etc. For example, the user selected "No problem" based on the meeting context.43 Figure 10 As shown in (c), the phone displays the text "No problem" in input box 23. Figure 10 As shown in (d), after the user confirms that the text in input box 23 is correct, they click the send control 26. In response to the click on send control 26, the phone converts the text "No problem" into audio and sends an audio message carrying that audio to the devices used by other participants. On the one hand, as... Figure 10 As shown in (e), after successfully sending the audio message, the phone can display the text "No problem" 44 in the floating window 22, so that the user knows that the audio message corresponding to the text "Okay, I understand" has been successfully sent. On the other hand, after other participants' devices successfully receive the audio message, these devices can play the audio through speakers, earpieces or headphones, or they can convert the audio into the text "No problem" based on an AI captioning application and display it.

[0119] As an example, a mobile phone can be pre-set with a preset word library, which contains a large number of preset words and preset sentences. These preset words and preset sentences can be pre-set by the user, pushed by the server, or come with the mobile phone system itself. In a meeting scenario, the AI subtitle application can perform semantic analysis on the received audio data and display multiple preset words corresponding to the semantic analysis results in the preset word box.

[0120] In the above embodiment, since the AI subtitle application provides the preset word control 24 in the floating window 22, the user can quickly reply with preset words to other devices by operating the preset word control 24, thus solving the problem of low efficiency in inputting text through the virtual keyboard.

[0121] Scenario Example 8

[0122] In some embodiments, the AI subtitle application also provides a translation function.

[0123] Exemplarily, as shown in (a) of Figure 10 , the floating window 22 further includes a more options 45. When the user wants to use more functions provided by the AI subtitle application, the user can click on the more options 45. In response to the click operation on the more options 45, as shown in (b) of Figure 11 , the mobile phone displays controls such as font, quick reply, and language type 46. For example, the user can click on the language type 46, and the mobile phone displays multiple language options as shown in (c) of Figure 11 , such as Chinese-English 47, Chinese-German, Chinese-French, Chinese-Russian, Chinese-Korean, and Chinese-Japanese, etc. If the user uses Chinese and other participants use English, then the user can select Chinese-English 47. As shown in (d) of Figure 11 , the user can enter the Chinese word "很好" in the input box 23 and then click on the send control 26. The AI subtitle application can translate the Chinese word "很好" into the English word "excellent", convert the English word "excellent" into audio, and then send a message carrying the audio "excellent" to the devices used by other participants. On the one hand, as shown in (e) of Figure 11 , after successfully sending the audio message, the mobile phone can display the text "很好" 48 in the floating window 22, so that the user can know that the audio message corresponding to the text "很好" has been successfully sent. On the other hand, after the devices used by other participants successfully receive the audio message, these devices can play the audio "excellent" through the speaker, earpiece, or headphones.

[0124] In other embodiments, if other participants' devices send audio in the first language to the mobile phone, the mobile phone can first convert the audio in the first language into text in the first language, then convert the text in the first language into text in the second language, and then display the text in the second language in the floating window 22.

[0125] As an example, suppose the user selects "Chinese-English" as the audio language, where "Chinese" represents the language used by the mobile phone user and "English" represents the language used by other participants. Figure 11 As shown, mobile phone 2 sends the audio "After workdining together" to mobile phone 1. Mobile phone 1 can first convert the audio "After work dining together" into the text "After work dining together", then convert the text "After work dining together" into the text "After get off work dinner together", and then display the text "After get off work dinner together" 49 in the floating window 22.

[0126] As another example, suppose the user selects "English-Chinese" as the audio language, where "English" represents the language used by the mobile phone user, and "Chinese" represents the language used by other participants. Figure 12 As shown, mobile phone 2 sends the audio "After get off work dinner together" to mobile phone 1. Mobile phone 1 can first convert the audio "After get off work dinner together" into the text "After get off work dinner together", then convert the text "After work dinner together" into the text "After get off work dining together", and then display the text "After work dining together" 50 in the floating window 22.

[0127] In the above embodiments, when participants use different languages, the AI-based subtitle application can not only translate languages ​​but also convert text and audio, thereby improving the communication efficiency of participants.

[0128] It should be noted that the above examples illustrate AI-powered captioning applications for enabling conferencing applications and do not limit the scope of this application. In actual implementation, AI captioning applications can also enable audio and video applications, such as converting audio from audio and video applications into text and displaying it as captions. Furthermore, AI captioning applications can also be used with call applications, such as converting received audio from a call application into text and displaying it; or converting user-inputted text into audio and sending it to other devices.

[0129] To better understand the above scenario examples, we will first illustrate the system architecture of the terminal device, and then illustrate the AI-based instant messaging method in conjunction with the system architecture.

[0130] The software system of the aforementioned terminal device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of the terminal device. Figure 13 This is a schematic diagram of the system architecture of the terminal device provided in the embodiments of this application.

[0131] like Figure 14 As shown, terminal devices can adopt a layered architecture, dividing the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the software layers of the software structure are divided from top to bottom as follows: application (APP) layer, application framework (FW) layer, system library (FWKLIB), hardware abstraction layer (HAL) layer, and kernel layer. This software architecture runs on top of the hardware layers, which may include speakers, microphones, USB interfaces, displays, and sensors, etc.

[0132] The application layer, also known as the application package layer, can include a series of application packages. For example, the application layer can include AI captioning applications, speech processing applications, conferencing applications, call applications, game applications, video applications, and settings applications. These applications can be system applications or third-party applications. The AI ​​captioning application can provide AI captioning functionality. The speech processing application can provide text-to-speech (TTS) functionality and automatic speech recognition (ASR) functionality. As an example, TTS and ASR functionality can be combined into a speech processing application. As another example, TTS and ASR functionality can be provided by two separate applications; that is, the TTS application and the ASR application are two separate applications. As yet another example, TTS and ASR functionality can also be combined into an AI captioning application. When these application packages are run, they can access the various service modules provided by the application framework layer through the application programming interface (API) and execute corresponding intelligent business logic.

[0133] The application framework layer provides APIs and a programming framework for applications in the application layer. The application framework layer includes some predefined functions. For example... Figure 14As shown, the application framework layer can include a media player (e.g., MediaPlayer), a media recorder (e.g., MediaRecorder), an audio data entity (e.g., AudioFlinger), and an audio manager (e.g., AudioManager). The audio manager can include an audio service (AudioService) and an audio policy (AudioPolicy) module. The AudioFlinger and the AudioPolicy are two basic services of the audio system: the AudioPolicy is a maker of the audio system policy, responsible for policy selection of audio device switching, volume adjustment policy, etc.; the AudioFlinger is an executor of the audio system policy, responsible for management of the audio stream device and processing and transmission of the audio stream data, and thus the AudioFlinger is also called an engine of the audio system.

[0134] Exemplarily, after the conference application receives the audio data from the opposite end device, the conference application can issue the audio data to the media player. The media player can send the audio data to the AudioFlinger for audio processing. The AudioFlinger can obtain the audio policy corresponding to the audio data from the audio manager. The conference application can invoke the AudioService and the AudioPolicy to set the corresponding audio policy for the audio data at runtime. The AudioFlinger can process the audio data from the conference application according to the audio policy made by the AudioPolicy, such as mixing, resampling, and sound effect setting of the audio data, etc.

[0135] The system library can include a plurality of functional modules, such as a surface manager, media libraries, a two-dimensional (2D) graphics engine, a three-dimensional graphics processing library, etc. In the system library, the Android Runtime includes a core library and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system. The virtual machine is used to perform functions such as management of the object life cycle, stack management, thread management, security and exception management, and garbage collection, etc.

[0136] The hardware abstraction layer has a standard interface implemented by hardware vendors. For example, the hardware abstraction layer can include an audio HAL, a display HAL, and the like. In addition, the hardware abstraction layer can also include a playback thread (DuplicatingThread), a mixing thread (MixerThread), a recording thread (RecordThread), and the like. The AudioFlinger calls these threads, and can send processed audio data to the audio HAL, which sends the audio data to the corresponding audio output device (such as a speaker, a wired earphone, or a Bluetooth earphone, and the like) for playing. As shown in FIG. 8, according to different audio output devices, the Audio HAL can be divided into an audio primary interface, a USB interface, an audio remote submix interface, an A2dp interface, and the like. The AudioFlinger can send audio data to different audio interfaces in the Audio HAL according to the audio policy formulated by the AudioPolicy. For example, the AudioFlinger can call the audio primary interface to output audio data to the speaker of the mobile phone; call the USB interface to output audio data to the wired earphone connected to the mobile phone; call the audio remote submix interface to output audio data to the remote device; and call the A2dp interface to output audio data to the Bluetooth earphone connected to the mobile phone. In addition, the AudioFlinger can also call these threads to read audio data from the audio interface. Figure 14

[0137] The kernel layer is a layer between hardware and software, and belongs to the bottom part of the Android system. The kernel layer can include various driver interfaces, such as a speaker driver, a microphone driver, a USB driver, a display driver, and the like.

[0138] It should be noted that, although the embodiments of the present application are described by taking the Android system as an example, the basic principles are also applicable to terminal devices based on the iOS or Windows operating system.

[0139] It can be understood that, in order to implement the instant messaging method based on the AI caption in the embodiments of the present application, the terminal device includes hardware and / or software modules corresponding to each function. The algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented in the form of hardware or a combination of hardware and computer software. Whether a certain function is implemented in hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in combination with the embodiments.

[0140] ​The following describes three instant messaging methods based on AI captioning with conference application as an example and three embodiments. It should be understood that the instant messaging method can also be applied to other instant messaging applications.

[0141] Embodiment One

[0142] Embodiment One provides a first audio strategy: the local device does not support collecting local sound through the microphone and transmitting it to the opposite device, but can only convert the text input in the floating window of the AI captioning application into audio data through the TTS function and send it to the opposite device; the local device also does not support playing audio data from the opposite device through the speaker, but can only convert the audio data from the opposite device into text through the ASR function and display it in the floating window of the AI captioning application.

[0143] Exemplarily, Figure 14 a module interaction diagram based on AI captioning is shown, Figure 15 a corresponding specific method flowchart is shown. Figure 16

[0144] As Figure 15 shown, in the instant messaging process of mobile phone 1 and mobile phone 2, four data streams are involved in mobile phone 1.

[0145] 1. The data stream shown in line 1.

[0146] The user of mobile phone 1 inputs text in the floating window 22, and the AI captioning application calls the speech processing application to convert the text into audio data. Then, the AI captioning application sends the audio data to AudioFlinger through the media player (MediaPlayer). AudioFlinger calls the playback thread (DuplicatingThread), which in turn calls the mixing thread (MixerThread), and sends the audio data to the audio remote processing (remote_submix) interface, such as the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0147] It should be noted that the above playback thread (DuplicatingThread) is a bus thread responsible for audio playback. The playback thread (DuplicatingThread) can call the mixing thread (MixerThread) and the recording thread (RecordThread), etc. Among them, the mixing thread (MixerThread) can be responsible for mixing processing of audio from other devices, and the recording thread can be responsible for mixing processing of locally recorded audio and audio obtained through TTS.

[0148] ​2. Data flow as shown in line 2.

[0149] Generally, when the microphone is in an open state, the microphone records the local audio, and then the microphone driver sends the audio data recorded by the microphone to the audio primary interface. The AudioFlinger calls the recording thread (RecordThread) to read the audio data from the audio primary interface, and sends the audio data to the conference application through the media recorder (MediaRecorder). However, as shown in Figure 15 When the data path / data flow as shown in line 2 is closed, the microphone is in a closed state, so that the conference application cannot obtain the local audio collected by the microphone.

[0150] 3. Data flow as shown in line 3.

[0151] The conference application can call the AudioFlinger and the recording thread (RecordThread) through the media recorder (MediaRecorder) to read data from the audio remote processing (remote_submix) interface, such as the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0152] 4. Data flow as shown in line 4.

[0153] The conference application can send the audio data from the phone 2 to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger calls the playback thread (DuplicatingThread) and the mixing thread (MixerThread) to send the audio data to the audio remote processing (remote_submix) interface, such as the MonoPipe interface of the audio remote processing (remote_submix) interface. Then, the AI caption application can read the audio data from the MonoPipe interface through the AudioFlinger. After that, the AI caption application calls the speech processing application to convert the audio data into text. After that, the AI caption application displays the converted text in the floating window 22. However, as shown in Figure 15 Because the speaker driver path is closed, the speaker is in a closed state, so the speaker driver cannot read the audio data from the MonoPipe interface, and cannot output the sound corresponding to the audio data through the speaker.

[0154] As shown in Figure 16 The method is applied to the phone 1, and the method includes A1 to A12. It should be noted that, due to the limited size of the drawings, Figure 16The duplicating thread, the mixer thread, and the recording thread are not shown.

[0155] A1. The conference application receives the audio data a from the mobile phone 2.

[0156] The mobile phone 2 can be one of the terminal devices that establishes a conference link with the mobile phone 1.

[0157] In actual implementation, the mobile phone 1 can establish conference links with multiple devices, i.e., multi-person conference. At a certain moment, the conference application of the mobile phone 1 can receive multiple audio data, such as the audio data from the mobile phone 2 and the audio data from the mobile phone 3. The mobile phone 1 processes the multiple audio data in a manner similar to the processing of the audio data a, which can be referred to the descriptions of A2 to A6 below, and thus is not described herein.

[0158] A2. The conference application sends the audio data a to the AudioFlinger through the media player.

[0159] A3. The AudioFlinger invokes the duplicating thread and the mixer thread to send the audio data a to the audio remote processing (remote_submix) interface.

[0160] For example, the audio data a is sent to the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0161] It should be noted that the audio primary (primary) interface and the audio remote processing (remote_submix) interface have different properties and support different data types. The speaker and the microphone usually use the audio primary (primary) interface and cannot use the audio remote processing (remote_submix) interface. In the first embodiment, since the local device does not support playing the audio data from the opposite device by the speaker, the audio primary (primary) interface can not be used and the audio remote processing (remote_submix) interface can be used to process the remote data, such as the audio data a.

[0162] A4. The audio remote processing (remote_submix) interface writes the audio data a.

[0163] A5. The AI caption application reads the audio data a from the MonoPipe interface through the AudioFlinger.

[0164] As an example, an AI captioning application can read audio data from the MonoPipe interface at preset intervals. For instance, the AI ​​captioning application can send a read command to AudioFlinger every N milliseconds. If the MonoPipe interface has stored new audio data, such as audio data 'a', then AudioFlinger returns audio data 'a' to the AI ​​captioning application. If the MonoPipe interface has not stored new audio data, then AudioFlinger returns empty data to the AI ​​captioning application, or does not return any data.

[0165] In step A6, the AI ​​captioning application sends a request to the speech processing application to convert audio data 'a' into text. In response to this request, the speech processing application converts audio data 'a' into text 'A' and returns text 'A' to the AI ​​captioning application.

[0166] The aforementioned speech processing application provides ASR and TTS services. In the A6 above, the speech processing application can convert audio data 'a' into text based on the ASR service.

[0167] A7, the AI ​​captioning application displays the text "A" in a floating window.

[0168] like Figure 7 As shown in (d), after the AI ​​captioning application calls the voice processing application to convert the audio data a into the text "Gather at the company entrance", the text "Gather at the company entrance" can be displayed in the floating window 22 38.

[0169] When the conferencing application on phone 1 receives multiple audio data simultaneously, the AI ​​captioning application on phone 1 can convert each audio data into a separate text message and display each text message in a floating window 22. Each text message carries an identifier for the device that sent the audio data, a user identifier, or an account identifier. This allows the user to distinguish the source of each text message.

[0170] A8, the AI ​​captioning application receives the text B entered by the user in the floating window.

[0171] like Figure 5 (a) to Figure 5 As shown in (e), the user of mobile phone 1 can enter the text "OK, I understand" in the input box 23 of the floating window 22 and click the send control 26, so that mobile phone 1 can perform the following A9 to A12.

[0172] A9, the AI ​​captioning application sends a request to the speech processing application to convert text B into audio data. In response to the request, the speech processing application converts text B into audio data b and returns the audio data b to the AI ​​captioning application.

[0173] A10, the AI caption application sends the audio data b to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger calls the playback thread (DuplicatingThread) and the mixing thread (MixerThread) to send the audio data to the audio remote processing (remote_submix) interface.

[0174] For example, the MonoPipe interface sends the audio data b to the audio remote processing (remote_submix) interface.

[0175] A11, the conference application calls the recording thread (RecordThread) through the media recorder (MediaRecorder) and the AudioFlinger to read the audio data b from the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0176] A12, the conference application sends the audio data b to the mobile phone 2 through the network.

[0177] For example, the mobile phone 1 converts the text "OK, I know" into audio, sends a message carrying the audio "OK, I know" to the mobile phone 2, and the mobile phone 2 can play the audio "OK, I know" through the speaker, earpiece or earphone. In addition, as shown in (f) of FIG. 1, the mobile phone 1 can display the text "OK, I know" 32 in the floating window 22, so that the user using the mobile phone can know that the audio message corresponding to the text "OK, I know" has been successfully sent. Figure 5

[0178] In the above embodiment, if the user of the mobile phone 1 does not want to hear the sound from the mobile phone 2 and does not want to send the local sound (such as background sound) to the mobile phone 2, the user of the mobile phone 1 can turn off the microphone and the speaker by operation. In this case, the microphone and the speaker are in the off state, and neither local recording nor playing sound from the mobile phone 2 can be performed. However, the AI caption of the mobile phone 1 supports converting the audio from the mobile phone 2 into text for display, and also supports converting the locally input text into audio for sending to the opposite end. That is, the user of the mobile phone 1 can communicate with the user of the mobile phone 2 only through text. It should be noted that for the specific implementation of the user turning off the microphone and the speaker, please refer to the description of the above scene example 3 to scene example 6, which will not be repeated here.

[0179] Embodiment Two

[0180] ​Embodiment two provides a second audio strategy: the local device does not support collecting local sound through a microphone and transmitting to the opposite device, but can only convert the text input in the floating window of the AI caption application into audio data through a TTS function and send to the opposite device; the local device supports both playing audio data from the opposite device through a loudspeaker and converting audio data from the opposite device into text through an ASR function and displaying in the floating window of the AI caption application.

[0181] Exemplarily, Figure 17 Another AI caption-based module interaction diagram is shown, Figure 18 corresponding to Figure 17 a specific method flow diagram.

[0182] As Figure 17 shown, in the instant messaging process of the mobile phone 1 and the mobile phone 2, four data streams are involved in the mobile phone 1.

[0183] 1. The data stream as shown in line type 1.

[0184] 2. The data stream as shown in line type 2.

[0185] 3. The data stream as shown in line type 3.

[0186] For the implementation of the data stream as shown in line type 1, the data stream as shown in line type 2, and the data stream as shown in line type 3, reference can be made to the specific description of the above-mentioned embodiment one, which will not be repeated here.

[0187] 4. The data stream as shown in line type 4.

[0188] The conference application can send the audio data from the mobile phone 2 to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger calls the playback thread (DuplicatingThread) and the mixing thread (MixerThread) to send the audio data to the audio primary interface, and the audio primary interface writes the audio data. On the one hand, the AI caption application can read the audio data from the audio primary interface through the AudioFlinger and the mixing thread (MixerThread), and then call the speech processing application to convert the audio data into text, and then display the converted text in the floating window 22. On the other hand, the loudspeaker driver can also read the audio data from the audio primary interface and play the corresponding sound through the loudspeaker.

[0189] As Figure 18As shown, the method is applied to the mobile phone 1, and the method comprises B1 to B13. It should be noted that, due to the limited size of the drawing, Figure 18 The duplicating thread (DuplicatingThread), the mixer thread (MixerThread) and the recording thread (RecordThread) are not shown.

[0190] B1, the conference application receives the audio data a from the mobile phone 2.

[0191] B2, the conference application sends the audio data a to the AudioFlinger through the media player (MediaPlayer).

[0192] B3, the AudioFlinger calls the duplicating thread (DuplicatingThread) and the mixer thread (MixerThread) to send the audio data a to the audio primary interface.

[0193] Referring to the description of the above embodiment one, the attributes of the audio primary interface and the audio remote processing (remote_submix) interface are different, and the supported data types are also different. The loudspeaker and the microphone usually use the audio primary interface, and cannot use the audio remote processing (remote_submix) interface. In the embodiment two, since the local device supports playing the audio data from the opposite device by the loudspeaker, in this case, the audio primary interface can be used to process the remote data, such as the audio data a.

[0194] B4, the audio primary interface writes the audio data a.

[0195] B5, the AI caption application reads the audio data a from the audio primary interface through the AudioFlinger.

[0196] As an example, the AI caption application can read the audio data from the audio primary interface according to a preset period. For example, the AI caption application can issue a reading instruction to the AudioFlinger every N milliseconds. If the audio primary interface stores new audio data, such as the audio data a, the AudioFlinger returns the audio data a to the AI caption application. If the audio primary interface does not store new audio data, the AudioFlinger returns empty data to the AI caption application, or does not return data.

[0197] B6, the AI caption application sends a request to convert the audio data a into text to the speech processing application. In response to the request, the speech processing application converts the audio data a into text A and returns the text A to the AI caption application.

[0198] B7, the AI caption application displays the text A in the floating window.

[0199] For the specific implementation of B6 and B7, please refer to the description of A6 and A7 of Embodiment I, which will not be repeated here.

[0200] B8, the audio primary interface sends the audio data a to the speaker driver.

[0201] The speaker driver drives the speaker to output the sound corresponding to the audio data a.

[0202] It should be noted that the execution order of B5-B7 and B8 is not specifically limited in the present application. For example, B5-B7 and B8 can be executed synchronously, so that the screen displays the text A corresponding to the audio data a at the same time as the speaker plays the sound corresponding to the audio data a.

[0203] B9, the AI caption application receives the text B input by the user in the floating window.

[0204] B10, the AI caption application sends a request to convert the text B into audio data to the speech processing application. In response to the request, the speech processing application converts the text B into audio data b and returns the audio data b to the AI caption application.

[0205] B11, the AI caption application sends the audio data b to the AudioFlinger through the MediaPlayer. The AudioFlinger calls the playback thread (DuplicatingThread) and the mixing thread (MixerThread) to send the audio data to the audio remote processing (remote_submix) interface.

[0206] For example, the audio data b is sent to the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0207] B12, the conference application calls the recording thread (RecordThread) through the MediaRecorder and the AudioFlinger to read data from the audio remote processing (remote_submix) interface, such as reading the audio data b from the MonoPipe interface of the audio remote processing (remote_submix) interface.

[0208] B13, the conference application sends the audio data b to the phone 2 through the network.

[0209] For the specific implementation of B9-B13, please refer to the description of A8-A12 of Embodiment Two, which will not be repeated here.

[0210] In the above embodiment, if the user of the phone 1 wants to hear the sound from the phone 2, but does not want to send the local sound (such as background sound) to the phone 2, the user of the phone 1 can operate to turn off the microphone and turn on the speaker. In this case, the microphone cannot make local recording, and the speaker can play the sound from the phone 2. In addition, the AI caption supports converting the audio from the phone 2 into text for display, and also supports converting the locally input text into audio and sending it to the opposite end.

[0211] Embodiment Three

[0212] Embodiment Three provides a third audio strategy: the local device supports both collecting local sound through the microphone and converting the text input in the floating window of the AI caption application into audio data through the TTS function; the local device supports both playing the audio data from the opposite end device through the speaker and converting the audio data from the opposite end device into text and displaying it in the floating window of the AI caption application through the ASR function.

[0213] Exemplarily, Figure 19 Another AI caption-based module interaction diagram is shown, Figure 20 corresponding to Figure 19 the specific method flow diagram.

[0214] As Figure 19 shown, in the instant messaging process between the phone 1 and the phone 2, four data streams are involved inside the phone 1.

[0215] 1. The data stream shown in line type 1.

[0216] The user of the phone 1 inputs text in the floating window 22, and the AI caption application calls the speech processing application to convert the text into audio data. Then, the AI caption application sends the audio data to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger calls the mixer thread (MixerThread) to send the audio data to the audio primary interface.

[0217] 2. The data stream shown in line type 2.

[0218] The microphone is in an open state. The microphone records the local audio. The microphone driver sends the audio data recorded by the microphone to the audio primary interface.

[0219] Thus, the RecordThread can obtain two types of audio data from the primary audio interface (i.e., audio data obtained through the data stream shown in line 1 and audio data obtained through the data stream shown in line 2), and mix these two types of audio data to obtain mixed data.

[0220] 3. Data flow as shown in line type 3.

[0221] Conferencing applications can call AudioFlinger through MediaRecorder to read mixed data from the RecordThread.

[0222] 4. Data flow as shown in line type 4.

[0223] The conferencing application can send audio data from mobile phone 2 to AudioFlinger via MediaPlayer. AudioFlinger calls the MixerThread to send the audio data to the primary audio interface, which then writes the audio data. On one hand, the AI ​​captioning application can read the audio data from the primary audio interface through AudioFlinger and call the speech processing application to convert the audio data into text, which is then displayed in the floating window 22. On the other hand, the speaker driver can also read the audio data from the primary audio interface and play the corresponding sound through the speaker. It should be noted that the primary audio interface of the MixerThread is a write interface, while the primary audio interface of the recording thread is a read interface; these two primary audio interfaces are of different types.

[0224] like Figure 20 As shown, this method is applied to mobile phone 1, and includes C1 to C14. It should be noted that due to the limited size of the attached figures, Figure 20 The playback thread (DuplicatingThread), mixing thread (MixerThread), and recording thread (RecordThread) are not shown.

[0225] C1, the conferencing application receives audio data a from mobile phone 2.

[0226] C2, the conferencing application sends audio data 'a' to AudioFlinger via MediaPlayer.

[0227] C3, AudioFlinger invokes the mixer thread (MixerThread) to send the audio data a to the audio primary interface.

[0228] Referring to the description of the above embodiment one, the attributes of the audio primary interface and the audio remote_submix interface are different, and the supported data types are also different. The loudspeaker and the microphone usually use the audio primary interface, and cannot use the audio remote_submix interface. In embodiment three, since the local device supports both playing the audio data from the opposite device by the loudspeaker and collecting the local sound by the microphone, in this case, the audio primary interface can be used to store the audio data from the opposite device and the audio data collected by the microphone.

[0229] C4, the audio primary interface writes the audio data a.

[0230] C5, the AI caption application reads the audio data a from the audio primary interface through AudioFlinger.

[0231] For the specific implementation of C5, refer to the description of B5 of embodiment two, which will not be repeated here.

[0232] C6, the AI caption application sends a request to the speech processing application to convert the audio data a into text. In response to the request, the speech processing application converts the audio data a into text A and returns the text A to the AI caption application.

[0233] C7, the AI caption application displays the text A in the floating window.

[0234] C8, the audio primary interface sends the audio data a to the loudspeaker driver.

[0235] The loudspeaker driver drives the loudspeaker to output the sound corresponding to the audio data a.

[0236] For the specific implementation of C1-C8, refer to the description of B1-B8 of embodiment two, which will not be repeated here.

[0237] C9, the AI caption application receives the text B input by the user in the floating window.

[0238] C10, the AI caption application sends a request to the speech processing application to convert the text B into audio data. In response to the request, the speech processing application converts the text B into audio data b and returns the audio data b to the AI caption application.

[0239] C11, the AI caption application sends the audio data b to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger calls the mixer thread (MixerThread) to send the audio data to the audio primary interface.

[0240] C12, the microphone driver sends the audio data c recorded by the microphone to the audio primary interface.

[0241] C13, the conference application calls the record thread (RecordThread) to read the mixed data of the audio data b and the audio data c through the media recorder (MediaRecorder) and the AudioFlinger.

[0242] For example, the conference application can send a read data instruction to the record thread (RecordThread) through the media recorder (MediaRecorder) and the AudioFlinger every preset time interval. For example, if the record thread (RecordThread) reads the audio data b and the audio data c from the audio primary interface in a certain period, the record thread (RecordThread) mixes the audio data b and the audio data c to obtain the audio data d, and then returns the audio data d to the conference application through the AudioFlinger and the media recorder (MediaRecorder).

[0243] C14, the conference application sends the mixed data of the audio data b and the audio data c to the mobile phone 2 through the network.

[0244] In the above embodiment, if the user of the mobile phone 1 wants to hear the sound from the mobile phone 2 and wants to send the local sound (such as background sound) to the mobile phone 2, the user of the mobile phone 1 can open the microphone and the speaker by operation. In this case, the microphone can record the local sound, and the speaker can play the sound from the mobile phone 2. In addition, the AI caption supports converting the audio from the mobile phone 2 into text for display, and also supports converting the locally input text into audio and sending it to the opposite end.

[0245] The above three embodiments introduce three audio strategies. As an implementation manner, the mobile phone can pre-set one audio strategy, and the mobile phone can directly use this audio strategy during the conference, such as the first audio strategy. As another implementation manner, the mobile phone can pre-set three audio strategies, and formulate the use mode of the three audio strategies, so that only one audio strategy can be used at a certain time, thereby avoiding logical conflicts caused by simultaneous use of multiple audio strategies.

[0246] The following will be described in combination withFigures 21 to 24 introduce the specific use of the three audio strategies.

[0247] Exemplarily, Figure 21 a module interaction diagram corresponding to the parameter distribution strategy of the downlink data (data from the mobile phone 2) is shown, Figure 22 a specific method flowchart diagram corresponding to the parameter distribution strategy of the downlink data is shown. Figure 21

[0248] As shown in Figure 21 and Figure 22 , in the instant messaging process between the mobile phone 1 and the mobile phone 2, the audio manager of the mobile phone 1 can formulate a parameter distribution strategy corresponding to the downlink data. Specifically, as shown in (c) of Figure 7 , the playback control 35 is in the form of a loudspeaker, or as shown in (a) of Figure 8 , the user presses the “volume +” button 39, or as shown in (b) of Figure 9 , the playback control 42 is in the form of a prompt to close, in these cases, the loudspeaker is in an open state, and the audio manager of the mobile phone 1 sends parameter 1 to the AudioFlinger; as shown in (a) of Figure 7 , the playback control 35 is in the form of an earpiece, or as shown in (c) of Figure 8 , the user presses the “volume -” button 40, or as shown in (e) of Figure 9 , the playback control 42 is in the form of a prompt to open, in these cases, the loudspeaker is in a closed state, and the audio manager of the mobile phone 1 sends parameter 2 to the AudioFlinger.

[0249] After the conference application of the mobile phone 1 receives the audio data a from the mobile phone 2, the conference application of the mobile phone 1 can send the audio data a to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger executes the distribution strategy corresponding to the downlink parameter at the current moment. If the downlink parameter at the current moment is parameter 1, A1 to A7 are executed, that is, the local device does not support playing the audio data from the opposite device by the loudspeaker, but can only convert the audio data from the opposite device into text through the ASR function and display it on the floating window of the AI caption application; if the downlink parameter at the current moment is parameter 2, B1 to B7 are executed, that is, the local device supports playing the audio data from the opposite device by the loudspeaker and also supports converting the audio data from the opposite device into text through the ASR function and displaying it on the floating window of the AI caption application. For specific implementation modes of A1 to A7 and B1 to B7, reference can be made to the description of the above embodiments, which will not be described here.

[0250] Exemplarily, Figure 23 ​The module interaction diagram corresponding to the parameter distribution strategy of the uplink data (data sent to the mobile phone 2) is shown, Figure 24 The module interaction diagram corresponding to the parameter distribution strategy of the uplink data (data sent to the mobile phone 2) is shown, Figure 23 The specific method flow diagram corresponding to the parameter distribution strategy of the uplink data (data sent to the mobile phone 2) is shown.

[0251] As shown in Figure 23 and Figure 24 , in the instant messaging process of the mobile phone 1 and the mobile phone 2, the audio manager of the mobile phone 1 can formulate a parameter distribution strategy corresponding to the uplink data. Specifically, as shown in (a) of Figure 6 , the mute control 25 is in a mute form, or as shown in (c) of Figure 6 , the mute control 33 is in a mute form, in these cases, the microphone is in a closed state, and the audio manager of the mobile phone 1 sends the parameter 3 to the AudioFlinger; as shown in (b) of Figure 6 , the playback control 35 and the mute control 33 are in a non-mute form, in these cases, the speaker is in an open state, and the audio manager of the mobile phone 1 sends the parameter 4 to the AudioFlinger.

[0252] The user of the mobile phone 1 inputs text in the floating window 22. The AI caption application calls the speech processing application to convert the text into audio data. Then, the AI caption application sends the audio data to the AudioFlinger through the media player (MediaPlayer). The AudioFlinger executes the distribution strategy corresponding to the uplink parameter at the current time. If the uplink parameter at the current time is the parameter 3, A8 to A12 are executed, that is, the local device does not support collecting local sound through the microphone and transmitting it to the opposite device, but can only convert the text into audio data through the TTS function and send it to the opposite device; if the uplink parameter at the current time is the parameter 4, C9 to C14 are executed, that is, the local device supports collecting local sound through the microphone and also supports converting text into audio data through the TTS function. For specific implementation modes of A8 to A12 and C9 to C14, reference can be made to the description of the above embodiments, which will not be described here.

[0253] According to the description of the above embodiments, the downlink parameter includes parameter 1 and parameter 2, and the uplink parameter includes parameter 3 and parameter 4. Generally, the audio manager sends a set of parameters to the AudioFlinger, and the set of parameters includes a downlink parameter and an uplink parameter. If the set of parameters includes parameter 1 and parameter 3, the audio policy at this moment is referred to as the first audio policy in the first embodiment described above; if the set of parameters includes parameter 2 and parameter 3, the audio policy at this moment is referred to as the second audio policy in the second embodiment described above; if the set of parameters includes parameter 2 and parameter 4, the audio policy at this moment is referred to as the third audio policy in the third embodiment described above. It should be understood that in addition to the first audio policy, the second audio policy and the third audio policy, there can be a fourth audio policy including parameter 1 and parameter 4, and the implementation of the fourth audio policy can refer to the description of the above embodiments, which will not be described here.

[0254] In the above embodiments, the audio manager can formulate different audio policies by sending different uplink and downlink parameters to the AudioFlinger, thereby realizing the control of the audio path of the microphone and the audio path of the loudspeaker, and meeting the audio use requirements of the user in different conference scenarios.

[0255] Figure 25 A hardware structure schematic diagram of a terminal device provided by an embodiment of the present application is shown.

[0256] Taking a mobile phone as an example, as shown in Figure 25 the mobile phone 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charge management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a loudspeaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0257] The processor 110 can include one or more processing units, for example: the processor 110 can include an application processor (AP), a modem processor, a controller, a memory, a baseband processor, etc.

[0258] The wireless communication function of the mobile phone 100 can be realized by the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor, and the baseband processor, etc., so that the mobile phone 100 interacts with other devices to exchange instant messaging data, such as voice, video, pictures, text, and emoticon, etc.

[0259] The mobile phone 100 realizes the display function through the GPU, the display screen 194, and the application processor, etc. The display screen 194 is used to display content, such as displaying a video call interface, a voice call interface, a subtitle box, a control center, a setting application interface, etc.

[0260] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to realize the expansion of the storage capacity of the mobile phone 100. The external memory card communicates with the processor 110 through the external memory interface 120 to realize the data storage function. For example, the subtitle text obtained based on the AI subtitle application is saved in the external memory card.

[0261] The mobile phone 100 can realize the audio function through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the earphone interface 170D, and the application processor, etc. Among them, the audio module 170 is used to convert digital audio information into an analog audio signal output, and is also used to convert an analog audio input into a digital audio signal; the speaker 170A, also known as a "loudspeaker", is used to convert an audio electrical signal into an acoustic signal, and the user can listen to the call request message, instant voice message, etc. from the opposite device through the speaker 170A; the receiver 170B, also known as a "earpiece", is used to convert the audio electrical signal from the opposite device into an acoustic signal; the microphone 170C, also known as a "microphone", "sound collector", is used to convert the sound signal collected locally into an electrical signal, and the user can make a sound by putting his mouth close to the microphone 170C during the instant messaging process, and input the sound signal into the microphone 170C; the earphone interface 170D is used to connect a wired earphone to convert the audio electrical signal from the opposite device into an acoustic signal.

[0262] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different component arrangements. The components shown can be realized in hardware, software, or a combination of software and hardware.

[0263] The embodiment of the present application further provides a terminal device, comprising a processor coupled with a memory, and the processor is used for executing computer programs or instructions stored in the memory, so that the terminal device implements the method in the above-mentioned embodiments.

[0264] The embodiment of the present application further provides a computer readable storage medium, which stores computer instructions; when the computer readable storage medium runs on a computer, the computer executes the method as shown above. The computer instructions can be stored in the computer readable storage medium, or transmitted from one computer readable storage medium to another computer readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center through wired (for example, coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (for example, infrared, wireless, microwave, etc.). The computer readable storage medium can be any available medium that can be accessed by a computer or data storage device including one or more servers, data centers, etc. integrated with one or more media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk or a magnetic tape), an optical medium or a semiconductor medium (for example, a solid state disk (SSD)) and the like.

[0265] The embodiment of the present application further provides a computer program product, which comprises computer program codes, and when the computer program codes run on a computer, the computer executes the method in the above-mentioned embodiments.

[0266] The embodiment of the present application further provides a chip coupled with a memory, and the chip is used for reading and executing computer programs or instructions stored in the memory to execute the method in the above-mentioned embodiments. The chip can be a general-purpose processor or a special-purpose processor. It should be noted that the chip can be implemented by using one or more field programmable gate arrays (FPGA), programmable logic devices (PLD), controllers, state machines, gate logic, discrete hardware components, any other suitable circuit or any combination of circuits capable of performing various functions described throughout the present application.

[0267] The terminal device, the computer readable storage medium, the computer program product and the chip provided by the embodiments of the present application are all used for executing the method provided above, and thus the beneficial effects that can be achieved thereby can refer to the beneficial effects corresponding to the method provided above, which will not be described herein again.

[0268] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0269] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, and any change or replacement within the technical scope disclosed in the present application should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An AI caption-based instant messaging method, characterized in that, The method is applied to a first device, the first device comprising a conference application, an AI caption application, a voice processing application, an audio data entity AudioFlinger, an audio main interface and an audio remote processing interface, and the method comprises: The first device displays an interface of the conference application and a window of the AI caption application; The first device receives first text input by a user in the window of the AI caption application; The first device converts the first text into first audio data by invoking the voice processing application through the AI caption application; When the microphone is in a forbidden recording state, the first device writes the first audio data to the audio remote processing interface by invoking the AudioFlinger through the AI caption application; the first device reads the first audio data from the audio remote processing interface by invoking the AudioFlinger through the conference application; and the first device sends the first audio data to a second device through the conference application; When the microphone is in an allowed recording state, the first device writes the first audio data to the audio main interface by invoking the AudioFlinger through the AI caption application; the first device writes second audio data collected through the microphone to the audio main interface; the first device reads mixed audio data of the first audio data and the second audio data from the audio main interface by invoking the AudioFlinger through the conference application; and the first device sends the mixed audio data to the second device through the conference application.

2. The method of claim 1, wherein, The method further comprises: The first device displays a sending message in the window of the AI caption application, the sending message comprising the first text.

3. The method of claim 1, wherein, The method further comprises: The first device receives third audio data from the second device through the conference application; The first device writes the third audio data to a second audio interface through the AudioFlinger, the second audio interface being the audio main interface or the audio remote processing interface; In response to a playback request of the AI caption application, the first device reads the third audio data from the second audio interface through the AudioFlinger; The first device displays a receiving message in the window of the AI caption application, the receiving message comprising second text, the second text being obtained by performing automatic speech recognition on the third audio data.

4. The method of claim 3, wherein, The second audio interface is the audio remote processing interface, and the first device further comprises a loudspeaker, the loudspeaker being in a forbidden sound emission state.

5. The method of claim 3, wherein, The second audio interface is the audio main interface, and the first device further comprises a loudspeaker, the loudspeaker being in an allowed sound emission state. The method further comprises: The first device reads the third audio data from the audio main interface; The first device outputs sound corresponding to the third audio data through the loudspeaker.

6. The method of claim 3, wherein, An audio interface used for writing the first audio data is referred to as a first audio interface; the first device further comprises an audio manager; the method further comprises: Based on the status of the microphone and the speaker of the first device, the first device formulates an audio policy for the AudioFlinger through the audio manager; Wherein, if the audio policy is a first audio policy, the first audio interface and the second audio interface are the audio remote processing interface; if the audio policy is a second audio policy, the first audio interface is the audio remote processing interface, and the second audio interface is the audio main interface; if the audio policy is a third audio policy, the first audio interface and the second audio interface are the audio main interface.

7. The method of claim 3, wherein, The first device further comprises a media player; the first device writes the third audio data to the second audio interface through the AudioFlinger, comprising: The first device calls the AudioFlinger through the media player to write the third audio data to the second audio interface.

8. The method according to any one of claims 1 to 7, characterized in that, The first device further comprises a media player; the first device writes the first audio data to the first audio interface through the AudioFlinger, comprising: The first device calls the AudioFlinger through the media player to write the first audio data to the first audio interface.

9. A terminal device, comprising: comprises a processor, and a memory coupled to the processor; Wherein, the memory stores instructions, and the processor invokes the instructions to enable the terminal device to perform the instant messaging method based on AI subtitles according to any one of claims 1 to 8.

10. A chip, characterized by The chip is coupled to the memory, and the chip is configured to read and execute a computer program stored in the memory to implement the instant messaging method based on AI subtitles according to any one of claims 1 to 8.

11. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and when the computer program runs on the electronic device, the electronic device performs the instant messaging method based on AI subtitles according to any one of claims 1 to 8.

Images