Subtitle management method and electronic equipment

Abstract

The invention discloses a subtitle management method and electronic equipment, and relates to the technical field of electronic equipment, and the subtitle management method comprises the steps that the electronic equipment displays a first area in a first interface in response to an operation that a user starts artificial intelligence AI subtitles; when the electronic equipment detects that the audio playing state is changed, if the currently played audio of the electronic equipment meets the preset condition, the electronic equipment obtains application information of a target application of the currently played audio, and subtitles corresponding to the currently played audio and the application information of the target application are displayed in the first area. In the application, the electronic equipment can obtain the source of the currently played audio and display the application information of the currently played audio, so that automatic classification of the subtitles is realized, the information effectiveness of the subtitles is improved, and the user experience of using the AI subtitles is optimized.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of electronic devices, and in particular to a subtitle management method and an electronic device. Background Art

[0002] In scenarios where some users use electronic devices to play videos and audios, in order to have a better audio-visual experience, users can turn on the artificial intelligence (AI) subtitle function of the electronic device. The function of AI subtitles is to convert the audio and video resources played by the electronic device into voice-to-subtitle text. When the electronic device plays video or audio, the subtitles corresponding to the video or audio can be displayed on the display interface.

[0003] In the scenario where different applications switch between playing videos or audios, AI subtitles will continue to generate subtitles for the played videos or audios. The existing technology cannot distinguish the AI ​​subtitles of the played videos or audios before and after the switch. All AI subtitles of the played videos or audios are displayed together. Users cannot confirm which subtitles belong to a certain played video or audio, reducing the content effectiveness of the AI ​​subtitles. Summary of the invention

[0004] The embodiment of the present application provides a subtitle management method and an electronic device. When the electronic device detects a change in the audio playback state, when the currently playing audio meets a preset condition, the application information of the application corresponding to the currently playing audio can be obtained. Based on the application information, the subtitles generated by the currently playing audio and the application information are displayed in the first area of ​​the first interface of the electronic device, thereby improving the effectiveness of the subtitle content. Displaying the application information of the subtitle source on the display interface can optimize the user's experience of browsing subtitles.

[0005] In order to achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions.

[0006] In a first aspect, a subtitle management method is provided, the method comprising:

[0007] The electronic device displays a first area on a first interface in response to a first operation by the user; the first operation is an operation of turning on artificial intelligence AI subtitles.

[0008] When the electronic device detects a change in the audio playback state, if the audio currently being played by the electronic device meets a preset condition, the electronic device obtains application information of a target application corresponding to the currently playing audio, wherein the target application corresponding to the currently playing audio is an application holding the audio focus.

[0009] The electronic device displays subtitles corresponding to the currently playing audio and application information of the target application in the first area; the application information includes at least one of an application name and an application icon of the target application.

[0010] In the present application, when the electronic device detects a change in the audio playback state, when the currently playing audio meets a preset condition, the application information of the application corresponding to the currently playing audio can be obtained, and based on the application information, the subtitles generated by the currently playing audio and the application information are displayed in the first area of ​​the first interface of the electronic device, thereby improving the effectiveness of the subtitle content. Displaying the application information of the subtitle source on the display interface can optimize the user's experience of browsing subtitles.

[0011] In a possible implementation of the first aspect, the electronic device saves the subtitles corresponding to the currently playing audio to a storage space corresponding to the application information of the target application in response to the user's operation of saving subtitles according to the application information of the target application.

[0012] In the present application, the electronic device can classify and process the subtitles generated by the currently playing audio based on the application information, thereby realizing the automatic classification and storage of the subtitles. The subtitle management and storage does not rely on the manual operation of the user, thus simplifying the subtitle management and storage process and effectively improving the utilization rate of the effective information of the subtitles.

[0013] In another possible implementation of the first aspect, the audio currently played by the electronic device meets the preset condition, including:

[0014] The playback channel of the currently playing audio is a media channel, and the type of the currently playing audio does not belong to the preset noise type, and the state of the media player currently playing the audio is the playing state; wherein the preset noise type includes at least one of system background sound, system keyboard tapping sound, alarm ringtone, and incoming call ringtone.

[0015] In the present application, the electronic device determines the validity of the currently playing audio by determining that the playback channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the state of the media player currently playing the audio is a playback state. If the currently playing audio is valid, it is considered that the application currently playing the audio is the target application holding the audio focus, so that the application information of the target application holding the audio focus can be obtained, avoiding the situation of erroneous acquisition of application information.

[0016] In another possible implementation manner of the first aspect, the audio playback state change includes any one of the following:

[0017] The electronic device changes from not playing audio to playing audio; the source of the currently playing audio of the electronic device changes from a first application to a second application; the playback volume of the currently playing audio of the electronic device increases or decreases; the playback state of the currently playing audio of the electronic device changes from playing to paused, or the playback state of the currently playing audio changes from paused to playing.

[0018] In the present application, any change in the audio playback state of the electronic device will trigger the electronic device to judge the validity of the currently playing audio. This is because the change in the audio playback state may involve the switching of the audio source, and it is necessary to reconfirm the validity of the currently playing audio, so as to obtain the application information of the switched audio source.

[0019] In another possible implementation manner of the first aspect, the method further includes:

[0020] When the electronic device receives the first operation from the user, the electronic device initializes the audio instance object of the currently playing audio; the audio instance object is used to describe the session information of the currently playing audio, and the session information includes the source, playing volume and playing status of the currently playing audio.

[0021] Before the electronic device detects a change in the audio playing state, the electronic device rewrites a preset audio state changing method; the preset audio state changing method is used to detect a change in the audio playing state of the currently playing audio.

[0022] In the present application, the electronic device can monitor the audio playback status by overriding the preset audio status change method, and when a change in the audio playback status is detected, it can call back to obtain the audio instance object of the currently playing audio, thereby determining the validity of the currently playing audio.

[0023] In another possible implementation of the first aspect, when the electronic device detects a change in the audio playback state, if the audio currently being played by the electronic device meets a preset condition, the electronic device obtains application information of a target application currently playing the audio, including:

[0024] When the electronic device detects a change in the audio playing state, the electronic device calls an audio state change method to obtain an audio instance object of the currently playing audio.

[0025] The electronic device determines whether the currently playing audio meets the preset conditions based on the session information of the currently playing audio described by the audio instance object. If the currently playing audio meets the preset conditions, the electronic device obtains the application information of the target application of the currently playing audio.

[0026] In the present application, the electronic device calls an overwritten preset audio state change method, and when a change in the audio playback state is detected, obtains the audio instance object of the currently playing audio, thereby judging the validity of the currently playing audio. When it is determined that the currently playing audio meets the preset conditions, that is, the currently playing audio is valid audio, the source application of the currently playing audio is determined to be the target application holding the audio focus, so that the application information of the target application can be obtained, thereby realizing the classification management of the subtitles generated by the currently playing audio based on the application information.

[0027] In another possible implementation of the first aspect, the electronic device determines whether the currently played audio meets a preset condition according to session information of the currently played audio described by the audio instance object, including:

[0028] The electronic device obtains the playback channel of the currently playing audio, the audio type of the currently playing audio, and the state of the media player currently playing the audio according to the session information of the currently playing audio.

[0029] If the playing channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the state of the media player currently playing the audio is a playing state, the electronic device determines that the currently playing audio meets the preset conditions;

[0030] If the playback channel of the currently playing audio is not a media channel, or the type of the currently playing audio belongs to any one of the preset noise types, or the state of the media player currently playing the audio is not a playing state, the electronic device determines that the currently playing audio does not meet the preset conditions.

[0031] In the present application, the electronic device determines the validity of the currently playing audio by determining that the playback channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the state of the media player currently playing the audio is a playback state. If the currently playing audio is valid, it is considered that the application currently playing the audio is the target application holding the audio focus, so that the application information of the target application holding the audio focus can be obtained, avoiding the situation of erroneous acquisition of application information.

[0032] In another possible implementation of the first aspect, the electronic device obtains application information of a target application that is currently playing audio, including:

[0033] The electronic device uses the reflection mechanism to call the preset method of obtaining the application identification of the audio instance object to obtain the identification of the target application currently playing the audio; the electronic device obtains the application information of the target application based on the identification of the target application.

[0034] In the present application, the electronic device obtains the corresponding application identifier in the audio instance object based on the reflection mechanism. The application corresponding to the audio instance object is a determined application holding the audio focus. The electronic device obtains the application identifier of the application holding the audio focus and can obtain the corresponding application information based on the application identifier of the application, thereby avoiding the problem of obtaining the application information of the application that does not hold the audio focus.

[0035] In another possible implementation manner of the first aspect, the method further includes:

[0036] In response to a user's opening operation on a preset application, the electronic device displays a recommendation control of an auxiliary function corresponding to the preset application in the second interface.

[0037] Among them, the recommended control is used to provide an entry for opening auxiliary functions, and the auxiliary functions include at least one of AI subtitles, screenshots, and full-screen notes.

[0038] The electronic device can determine the correspondence between the preset application and the auxiliary function according to the frequency of opening the auxiliary function when the user runs the application. For example, if the frequency of opening the auxiliary function 1 when running the application 1 is greater than the threshold, it is considered that the auxiliary function 1 can be used as a recommended function of the application 1. In this case, when the electronic device receives the user's operation to open the application 1, in response to the operation, the recommendation control of the auxiliary function 1 is displayed on the current display interface of the application 1, so as to provide the user with an opening entry for the auxiliary function 1.

[0039] In the present application, the electronic device can determine the correspondence between the application and the auxiliary function according to the frequency with which the user uses the corresponding auxiliary function in different applications, so that when receiving the user's request to open the application, the recommended control for opening the auxiliary function is displayed, making it more convenient for the user to open the corresponding auxiliary function in the running application, thereby optimizing the user's usage experience.

[0040] In another possible implementation manner of the first aspect, the method further includes:

[0041] In response to the user's full-screen zoom-in operation on the first area, the electronic device updates the display size of the first area from a default size to a full-screen size to form a second area.

[0042] The second area includes application information of at least one target application and subtitles corresponding to the currently played audio of each target application.

[0043] In this application, the second area after the first area is fully enlarged displays all subtitles generated after the AI ​​subtitle function is turned on. Users can browse all subtitle contents by swiping up or down. At the same time, the source of subtitles is also displayed in the second area, which allows users to classify the source of subtitle content more intuitively and clearly.

[0044] In a second aspect, an electronic device is provided, which includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes a method as described in any one of the above-mentioned first aspects.

[0045] According to a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on an electronic device, the electronic device can execute any one of the methods according to the first aspect.

[0046] According to a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute any one of the methods according to the first aspect.

[0047] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor, the processor being used to call a computer program in a memory to execute the method of the first aspect.

[0048] It can be understood that the beneficial effects that can be achieved by the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect provided above can refer to the beneficial effects in the first aspect and any possible design method thereof, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0050] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 A flowchart of a subtitle management method provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of enabling the AI ​​subtitle function provided in an embodiment of the present application;

[0053] Figure 5 A schematic diagram of another embodiment of the present application for enabling the AI ​​subtitle function;

[0054] Figure 6 A schematic diagram of an electronic device displaying application information of a target application and subtitles generated by the target application provided by an embodiment of the present application;

[0055] Figure 7 A flowchart of another subtitle management method provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of an electronic device provided in an embodiment of the present application showing a first area magnified in full screen;

[0057] Fig. 9 A flowchart of another subtitle management method provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] In the description of the embodiments of the present application, the terms used in the following embodiments are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification and the appended claims of the present application, the singular expressions "a", "said", "above", "the" and "this" are intended to also include such expressions as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the following embodiments of the present application, "at least one", "one or more" refer to one or more (including two). The term "and / or" is used to describe the association relationship of associated objects, indicating that three relationships can exist; for example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are a kind of "or" relationship.

[0061] References to "one embodiment" or "some embodiments" etc. described in this specification mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Thus, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. The term "connection" includes direct connection and indirect connection, unless otherwise specified. "First" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated.

[0062] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0063] With the development of electronic equipment technology, electronic devices can provide users with artificial intelligence (AI) subtitle functions. When users turn on the AI ​​subtitle function of electronic devices, the electronic devices can convert the audio and video played by the application into subtitle text, and synchronously display the subtitles corresponding to the audio and video. Users can also choose to save the subtitles to a specified storage space for later viewing. The emergence of the AI ​​subtitle function provides users with a better audio-visual experience.

[0064] Generally, the AI ​​subtitle function will generate and display subtitles for the audio and video played by the foreground running application. If there are multiple foreground running applications in the electronic device, based on the "audio focus mechanism" of the electronic device, the AI ​​subtitle function can obtain the application holding the audio focus, and generate and display subtitles according to the audio and video played by the application. Among them, the "audio focus mechanism" of the electronic device means that when there are multiple foreground running applications playing multiple audio and video in the electronic device, the electronic device can determine that only one application holds the audio focus from multiple foreground running applications according to preset rules. Optionally, the preset rules may include, for different applications, according to the priority of the pre-set application, determining the application with the highest priority as the application holding the audio focus; or, the preset rules may also include, according to the frequency of use of the application, determining the application with the highest frequency of use as the application holding the audio focus, etc. For the audio and video played by the application that does not hold the audio focus, the electronic device can reduce the impact of the application that does not hold the audio focus on the application that holds the audio focus by reducing its playback volume. If there is only one foreground running application in the electronic device playing audio and video, then the foreground running application is the application holding the audio focus.

[0065] In the scenario where the user actually uses the AI ​​subtitle function of the electronic device, the user may switch applications. For example, when the user watches a TV series through a video application, at this time, the only application running in the foreground is the video application, and the video application is the application holding the audio focus. The user turns on the AI ​​subtitle function, and the AI ​​subtitle function generates and displays the subtitles of the TV series based on the TV series played by the video application. For example, in an example scenario, if the user is interrupted by an online conference application (or an audio and video call of a communication application) while watching a TV series, the foreground running application is switched from the video application to the online conference application (or communication application), and accordingly, the application holding the audio focus is also switched from the video application to the online conference application (or communication application). After displaying the subtitles of the TV series, the AI ​​subtitle function will then generate and display the subtitles corresponding to the audio and video played by the online conference application (or communication application) based on the audio and video played by the online conference application (or communication application).

[0066] When the user wants to save subtitles, the generated subtitles include the subtitle content corresponding to the audio and video played by multiple applications such as video applications, online conference applications (or communication applications). In this case, if the user wants to save subtitles according to different applications, the user needs to manually split the subtitles based on the subtitle content and save them separately. Alternatively, the user can save the subtitles generated by the TV series played by the video application in time when the video application is interrupted, and turn off the AI ​​subtitle function. After switching to the online conference application (or communication application), turn on the AI ​​subtitle function to generate and save the subtitles generated by the audio and video played by the online conference application (or communication application).

[0067] Regardless of which of the above methods is used, the process for users to obtain and save subtitles corresponding to different applications is cumbersome, which reduces the user experience of using the AI ​​subtitle function. At the same time, since the generated subtitles include subtitle content of audio and video played by multiple applications, the information effectiveness of the generated subtitles is greatly reduced.

[0068] An embodiment of the present application provides a subtitle management method. When a user turns on the AI ​​subtitle function, the application information of the application holding the audio focus can be identified and obtained. While generating and displaying subtitles for the audio and video played by the application holding the audio focus, the application information of the application holding the audio focus is displayed. This can improve the effectiveness of the AI ​​subtitle content and optimize the user experience of using AI subtitles.

[0069] Furthermore, when a user operation to save subtitles is received, the subtitles of the application can be classified and stored according to the application information of the application, thereby realizing automatic classification storage of subtitles based on different applications without relying on manual operations of the user, thereby optimizing the user experience of using AI subtitles and improving the information validity of the stored subtitles.

[0070] The subtitle management method provided in the embodiment of the present application can be applied to an electronic device. Figure 1 The electronic device 100 shown may be a portable computer (such as a mobile phone), a tablet computer, a laptop computer, a personal computer (PC), a wearable electronic device (such as a smart watch), an augmented reality (AR) device, a virtual reality (VR) device, a car computer, or other devices. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0071] Please refer to Figure 1, which shows a structural block diagram of an electronic device (such as electronic device 100) provided in an embodiment of the present application. Among them, the electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, a communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, an earphone interface 370D, a sensor module 380, a display screen 394, etc.

[0072] The structure shown in the embodiment of the present invention does not limit the electronic device 100. It may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0073] The processor 310 may include one or more processing units. For example, the processor 310 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors.

[0074] The controller can be a decision maker that directs the various components of the electronic device 100 to work in coordination according to the instructions. It is the nerve center and command center of the electronic device 100. The controller generates an operation control signal according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.

[0075] The processor 310 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a high-speed cache memory, which can store instructions or data that the processor 310 has just used or cyclically used. If the processor 310 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0076] In some embodiments, the processor 310 may include an interface. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM interface, and / or a USB interface, etc.

[0077] The interface connection relationship between the modules shown in the embodiment of the present invention is only for illustrative purposes and does not constitute a structural limitation on the electronic device 100. The electronic device 100 may adopt different interface connection methods in the embodiment of the present invention, or a combination of multiple interface connection methods.

[0078] The charging management module 340 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 may receive charging input from a wired charger through the USB interface 330. In some wireless charging embodiments, the charging management module 340 may receive wireless charging input through a wireless charging coil of the electronic device 100. While the charging management module 340 is charging the battery 342, it may also power the electronic device 100 through the power management module 341.

[0079] The power management module 341 is used to connect the battery 342, the charging management module 340 and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, and supplies power to the processor 310, the internal memory 321, the external memory interface 320, the display screen 394, the camera 393, and the communication module 360. The power management module 341 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some embodiments, the power management module 341 can also be set in the processor 310. In some embodiments, the power management module 341 and the charging management module 340 can also be set in the same device.

[0080] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the communication module 360, the modem and the baseband processor.

[0081] Antenna 1 is used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, a cellular network antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in combination with a tuning switch.

[0082] The modem may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 370A, a receiver 370B, etc.), or displays an image or video through a display screen 394. In some embodiments, the modem may be an independent device. In some embodiments, the modem may be set in the same device independently of the processor 310 and other functional modules.

[0083] The communication module 360 ​​can provide a communication processing module for wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The communication module 360 ​​can be one or more devices integrating at least one communication processing module. The communication module 360 ​​receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 310. The communication module 360 ​​can also receive the signal to be sent from the processor 310, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 1.

[0084] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the communication module 360, so that the electronic device 100 can communicate with the network and other devices through wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global satellite positioning system (satellite based augmentation systems, SBAS), a global navigation satellite system (global navigation satellite system, GLONASS), a BeiDou navigation satellite system (BeiDou navigation satellite system, BDS), a Quasi-Zenith satellite system (Quasi-Zenith satellite system, QZSS) and / or a satellite based augmentation system (satellite based augmentation systems, SBAS).

[0085] The electronic device 100 implements the display function through a GPU, a display screen 394, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 310 may include one or more GPUs, which execute program instructions to generate or change display information.

[0086] The display screen 394 is used to display images, videos, etc. For example, when the AI ​​subtitle function is turned on in the electronic device, the display screen 394 can display the subtitles corresponding to the audio and video played by the application holding the audio focus. In the embodiment of the present application, while displaying the subtitles, the display screen 394 can also display the application information of the application holding the audio focus, for example, the application information can be the name, icon, etc. of the application.

[0087] Optionally, in some embodiments, when the AI ​​subtitle function is not enabled in the electronic device, when the preset application plays audio and video, the display screen 394 may display a function recommendation control on the current display interface of the application. The function recommendation control may provide the user with an entry to enable the AI ​​subtitle function, and the user may enable the AI ​​subtitle function through the function recommendation control, so that the display screen 394 may display the subtitles corresponding to the audio and video played by the application holding the audio focus. The preset application may be determined according to the type of application, frequency of use, etc.

[0088] In some embodiments, if the electronic device supports multiple applications holding audio focus, when the AI ​​subtitle function is turned on, the display screen 394 can display multiple pop-up windows, floating display interfaces and other interface controls. In the multiple pop-up windows and floating display interfaces, the subtitles corresponding to the audio and video played by the application holding the audio focus are displayed respectively.

[0089] The display screen 394 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 394, where N is a positive integer greater than 1.

[0090] The external memory interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 310 through the external memory interface 320 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0091] The internal memory 321 can be used to store computer executable program codes, which include instructions. The processor 310 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 321. The memory 321 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the memory 321 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, other volatile solid-state storage devices, a universal flash storage (UFS), etc.

[0092] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 370, the speaker 370A, the receiver 370B, the microphone 370C, the headphone jack 370D, and the application processor.

[0093] In some embodiments, the audio module 370 may determine, according to preset rules, an application that can hold the audio focus among one or more applications that play audio and video. Exemplarily, the number of audio focuses is set to one. If the electronic device currently has only one application that plays audio and video, then the audio module 370 determines that the application that is currently playing audio and video is the application that holds the audio focus. If the electronic device currently has multiple applications that play audio and video, then the audio module 370 may determine, according to preset rules, an application that holds the audio focus from multiple applications that are currently playing audio and video. The preset rules may include, for different applications, determining the application with the highest priority as the application that holds the audio focus according to a preset priority of the application; or, the preset rules may also include determining, according to the frequency of use of the application, the application with the highest frequency of use as the application that holds the audio focus, etc.

[0094] Exemplarily, the number of audio focuses can be set to N. If the electronic device currently has only one application playing audio or video, the audio module 370 determines that the application currently playing audio or video is the application holding the audio focus. If the electronic device currently has N+M applications playing audio or video, the audio module 370 can determine N applications holding the audio focus from the N+M applications currently playing audio or video according to preset rules. The preset rules may include, for different applications, determining the N applications with the highest priority as the applications holding the audio focus according to a preset application priority; or, the preset rules may also include determining the N applications with the highest usage frequency as the applications holding the audio focus according to the frequency of use of the application, etc.

[0095] The audio module 370 can transmit application information of the application holding the audio focus to the processor 310. The processor 310 can determine the application holding the audio focus based on the application information of the application holding the audio focus, and perform operations such as transmission of instructions such as generation and display of AI subtitles and processing of data based on the audio and video played by the application.

[0096] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplified.

[0097] Figure 2 1 is a software structure diagram of the electronic device 100 of an embodiment of the present invention. The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, The system is divided into four layers, from top to bottom: application layer, application framework layer, Runtime (Android runtime) and system libraries and kernel layer.

[0098] The application layer can include a series of application packages.

[0099] like Figure 2 As shown, the application package may include subtitle application, camera, gallery, calendar, phone, map, WLAN, Bluetooth, music and other applications.

[0100] In this embodiment, the user can enable the AI ​​subtitle function of the electronic device by starting the subtitle application. Alternatively, in some embodiments, the user can enable the AI ​​subtitle function through the system setting interface of the electronic device.

[0101] In some embodiments, if the electronic device turns on the AI ​​subtitle function, when audio and video are detected, the subtitles corresponding to the audio and video being played can be displayed on the display interface. At the same time, the application information of the application that plays the audio and video can also be displayed on the display interface. For example, the application information can be the application icon, name and other information.

[0102] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0103] like Figure 2 As shown, the application framework layer may include an audio manager, a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0104] In this embodiment, the audio manager is used to control and access the system audio model and volume of the electronic device. For example, after the AI ​​subtitle function is turned on, the audio manager can detect the audio playback status of the electronic device, and confirm whether to switch between audio and video based on the audio playback status. That is, the audio manager can identify the application currently holding the audio focus based on the audio playback status.

[0105] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0106] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0107] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0108] The phone manager is used to provide communication functions of the electronic device 100. For example, the management of call status (including answering, hanging up, etc.).

[0109] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0110] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0111] The runtime includes the core library and the virtual machine. The runtime is responsible for The core library consists of two parts: one is the function that the Java language needs to call, and the other is The core library.

[0112] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0113] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as openGL ES), 2D graphics engine (such as SGL), etc.

[0114] The surface manager is used to manage the display subsystem and provide fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of multiple common audio and video formats, as well as static image files. The media library can support multiple audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc. The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, synthesis, and layer processing. The 2D graphics engine is a drawing engine for 2D drawing.

[0115] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0116] The following describes a subtitle management method provided by an embodiment of the present application in combination with a specific scenario. Figure 3 A flowchart of a subtitle management method is provided, including:

[0117] S201: The electronic device receives an operation from a user to enable an AI subtitle function, and displays a first area on a first interface.

[0118] The entry of the AI ​​subtitle function can be set in the function item of the system setting of the electronic device. The user enters the system setting interface of the electronic device and turns on the operation of the AI ​​subtitle function from the system setting interface. Figure 4 (a) shows a schematic diagram of a system setting interface of an electronic device, wherein the system setting interface includes multiple function options, such as "application", "battery", "storage", "security", "privacy", "healthy use of mobile phone", "smart assistant", "card package", "accessibility", "account", "system update", etc. The entrance to the AI ​​subtitle function can be set in the "smart assistant", and optionally, the entrance to the AI ​​subtitle function can also be set in other function options.

[0119] Taking the example that the entrance to the AI ​​subtitle function can be set in the "Smart Assistant", if the electronic device receives a user's selection operation for the "Smart Assistant" function item 101, in response to the operation, the electronic device displays the display interface 102 of the "Smart Assistant". Figure 4 (b) shows a display interface 102 of a "smart assistant", wherein the display interface 102 of the "smart assistant" may include a function item of "AI subtitles" and other function items belonging to the smart assistant, such as "voice assistant", etc. If the electronic device receives a user's selection operation for the "AI subtitles" function item, in response to the operation, the electronic device displays a detailed interface 103 of "AI subtitles".

[0120] For example, reference Figure 4 (c) shows a detailed interface 103 of "AI subtitles". The detailed interface 103 of "AI subtitles" may include an introduction to AI subtitles, a switch control 1031 for the AI ​​subtitle function, and other functional items. For example, other functional items may include "writing mode", "language", "subtitle font size", "panel transparency", etc. Among them, "writing mode" can be used to select the mode of subtitles or dialogues. The "language" function item may include the language for generating subtitles, such as Chinese, English, Japanese, Korean, etc.; or the "language" function item may include the translation language in dialogue mode, such as Chinese to English, English to Chinese, Chinese to Japanese, Japanese to Chinese, Chinese to Korean, Korean to Chinese, etc. The "subtitle font size" function item can select the size of the subtitle font, for example, the font size may include large font size, larger font size, moderate font size, smaller font size, small font size, etc. "Panel transparency" can select the transparency of the panel where the subtitles are located, and the default value of transparency is 0%. The transparency can be adjusted to any integer value between 0% and 100%.

[0121] If the electronic device receives a user's on-off operation of the switch control 1031 for the AI ​​subtitle function, in response to the operation, the electronic device displays the first area. The first area is used to display the generated subtitles and some functional options related to the subtitles. In this case, the first interface is the current "AI subtitle" details interface 103.

[0122] In some embodiments, the default display size of the first area is smaller than the full screen size. The default display size may be determined based on the size of the display screen of the electronic device (full screen size), for example, the default display size is 1 / N times the full screen size. Alternatively, the default display size may be a preset fixed value.

[0123] In this embodiment, the first area can also be understood as a display panel for subtitles. Figure 4 In (c), adjusting the panel transparency is to adjust the transparency of the first area. Figure 4 (e) shows a schematic diagram of the first area 105. The first area 105 includes subtitle content, a zoom-in control 1051 for the first area, a zoom-out control 1052 for the first area, a close control 1054 for the first area, and a control 1053 for switching between a subtitle mode and a dialogue mode.

[0124] If the electronic device is not currently playing any audio or video, no subtitle content may be displayed in the first area; or, the "subtitle content" display area in the first area may display some characters to indicate that subtitles are waiting to be generated. Figure 4 In the "subtitle content" display area in (e), ellipsis is displayed with a static or flashing dynamic effect.

[0125] In some embodiments, if the electronic device receives a user's selection operation of the zoom-in control 1051 for the first area, the electronic device may respond to the operation and fully zoom in on the first area. If the electronic device receives a user's selection operation of the zoom-out control 1052 for the first area, the electronic device may respond to the operation by minimizing the first area to the left edge / right edge of the display screen; or, minimizing the first area to the bottom of the display screen; or, minimizing the first area to a specified position of the display screen. If the electronic device receives a user's selection operation of the close control 1054 for the first area, the electronic device may respond to the operation by not displaying the first area and exiting the AI ​​subtitle function.

[0126] In response to the user's selection operation of the zoom control 1051 for the first area, the electronic device zooms in the first area in full screen. Figure 4 (f), wherein the display area 106 is a schematic diagram of the first area 105 after full-screen enlargement. The enlarged display area 106 may include a storage control 1061, an exit enlargement control 1062, a language selection control 1063, and a function selection control 1064. If the electronic device receives a user's selection operation on the storage control 1061, the electronic device may, in response to the operation, display a list item of at least one preset storage space, and based on the user's selection of the list item of the storage space, the electronic device may store the generated subtitles in the storage space indicated by the list item. If the electronic device receives a user's selection operation on the exit enlargement control 1062, the electronic device may, in response to the operation, exit the full-screen enlargement and restore the default size display of the first area (reference Figure 4 (e)). If the electronic device receives a selection operation of the language selection control 1063 by the user, the electronic device can respond to the operation by displaying a language selection interface and switching the subtitle language or the dialogue language. If the electronic device receives a selection operation of the function selection control 1064 by the user, the electronic device can respond to the operation by displaying a function selection details interface and selecting and implementing a corresponding function in the function selection details interface.

[0127] Optionally, in some embodiments, when the electronic device receives a user's on-off operation of the switch control 1031 for the AI ​​subtitle function, the electronic device may further display permission reminder information about the AI ​​subtitle function before displaying the first area 105. Figure 4 (d), in response to the user's turning on the switch control 1031 of the AI ​​subtitle function, the electronic device displays permission reminder information 104 in the details interface 103 of "AI subtitles" to remind the user that turning on the AI ​​subtitles function may cause access to user privacy. For example, the permission reminder information may include "Do you allow "Smart Assistant" to record / project your screen? If allowed, this application will record / project all the content displayed on your screen. Please pay attention to protect personal sensitive information, such as audio, passwords, payment information, photos, information, etc." At the same time, the permission reminder information 104 may also include "Prohibit" controls and "Allow" controls. When the user selects the "Allow" control, the electronic device can obtain the recording / projection permission of the screen to realize the AI ​​subtitles function.

[0128] Optionally, in another implementation, the entrance to the AI ​​subtitle function can also be set in the floating ball. Among them, the floating ball can be displayed in the main interface or application interface of the electronic device. The first interface can be the current display interface of the electronic device, such as the main interface of the electronic device, or the display interface of the application currently running on the electronic device. The user selects the floating ball to enter the function item list interface of the floating ball, and selects the "AI subtitle" function item in the function item list interface to enable the AI ​​subtitle function.

[0129] For example, reference Figure 5 (a), taking the first interface as the main interface as an example, a floating ball 201 is displayed in the main interface of the electronic device, and the floating ball 201 can provide a function entrance of a preset function. The electronic device can respond to the user's operation of selecting the floating ball 201 and display a preset function list interface 202 of the floating ball 201 in the main interface. Figure 5 (b), the preset functions include AI subtitles, screenshots, etc., and the preset function list interface 202 includes function items "AI subtitles", "screenshots", etc. In response to the user's selection operation of the function item "AI subtitles", the electronic device displays the first area 203 in the main interface, referring to Figure 5 (c). The first area is used to display the generated subtitles and some functional options related to the subtitles. Among them, the controls or contents included in the first area 203 can refer to Figure 4 The relevant introduction of the first area 105 in (e) is not repeated here.

[0130] Optionally, the size of the first area can be a fixed size, or the size of the first area can be an adjustable size. For example, the user can adjust the height and width of the first area according to the actual usage scenario. When the length of the generated subtitle content exceeds the width of the display area of ​​the first area, the subtitles can be automatically wrapped to continue to be generated. If the length of the subtitle content exceeds the area of ​​the first area, the first area can hide the first generated subtitles in the first area according to the sliding display mechanism, and continue to display the newly generated subtitles in the first area.

[0131] It should be noted that after the first area 105 is fully enlarged, all subtitles generated after the AI ​​subtitle function is turned on are displayed in the first area after full screen enlargement, and the user can browse all subtitle contents by swiping up or down.

[0132] In some embodiments, in response to the user's selection operation of the function item "AI subtitles", the electronic device can minimize the floating ball 201 to the side area or other designated position of the display screen in addition to displaying the first area 203. Figure 5 (c) shows the display mode of the floating ball 201.

[0133] S202: If the electronic device detects that the audio state has changed and the currently playing audio meets a preset condition, obtain application information of a target application corresponding to the currently playing audio.

[0134] The target application corresponding to the currently playing audio is the application holding the audio focus. The audio played by the application holding the audio focus is the currently playing audio, that is, the audio being played after the electronic device detects a change in the audio state.

[0135] In this embodiment, the audio state changes in various situations. For example, the audio playback source changes, from application A playing the audio to application B playing the audio. For example, the audio playback state changes, from not playing the audio to playing the audio, or from playing the audio to playing pause / stopping the audio. For example, the audio volume changes, the volume of the playing audio increases or decreases. For example, the audio output device changes, from headphone output to speaker output, etc.

[0136] When the electronic device detects any of the above changes in the audio state, the electronic device further determines whether the currently playing audio meets the preset conditions. If the currently playing audio meets the preset conditions, the application information of the target application is obtained. In this embodiment, the target application refers to the application that holds the audio focus. When the electronic device detects any of the above changes in the audio state, the electronic device can obtain the identifier of the application that currently holds the audio focus. For example, the identifier can be the user identification (uid) of the application. The electronic device obtains the application information of the application based on the uid of the application. Exemplarily, the application information may include information such as the icon and name of the application.

[0137] Optionally, in some embodiments, the electronic device detects whether the currently playing audio meets a preset condition to determine the validity of the playing audio, thereby determining the application holding the audio focus and obtaining application information of the target application (the application holding the audio focus).

[0138] The electronic device further detecting that the currently played audio meets the preset condition may include:

[0139] In one case, the electronic device determines whether the current audio playback channel is a media playback channel. The electronic device can obtain the playback channel information of the currently played audio, and if the audio playback channel is a media playback channel, the electronic device determines that the currently played audio is a valid audio.

[0140] In another case, the electronic device determines whether the currently played audio is a preset noise. The electronic device can obtain the type of the played audio to determine whether the currently played audio is the preset noise. The preset noise includes system prompt sound, screen keyboard tapping sound, alarm ringtone, incoming call ringtone, etc. If the currently played audio does not belong to the preset noise, the electronic device determines that the currently played audio is valid audio.

[0141] In another case, the electronic device determines whether the current playing state of the media player is the playing state. If the current playing state of the media player is the playing state, the electronic device determines that the currently playing audio is the valid audio.

[0142] In other cases, the electronic device can determine whether the currently playing audio is the audio played by the application holding the audio focus, that is, determine whether the audio played by the application holding the audio focus has changed. If it has changed, determine that the currently playing audio meets the preset conditions, and obtain application information of the new application holding the audio focus.

[0143] When the electronic device determines that all of the above conditions are met, it determines that the currently played audio is valid audio, and the application playing the valid audio is the application holding the audio focus. In this case, the electronic device obtains application information of the application holding the audio focus.

[0144] In some embodiments, if the electronic device determines that the currently playing audio does not meet at least one of the above conditions, it determines that the currently playing audio does not meet the preset conditions. For example, the playback channel of the currently playing audio is not a media playback channel; or, the currently playing audio belongs to the preset noise; or, the current playback status of the media player is not played; or, the playback channel of the currently playing audio is not a media playback channel, and the currently playing audio belongs to the preset noise, etc. The electronic device determines that the currently playing audio is invalid audio, the currently playing audio does not meet the preset conditions, does not generate and display AI subtitles for the currently playing audio, and does not perform application information acquisition operations.

[0145] After the electronic device acquires the application information of the target application, the electronic device may display the application information in the first area.

[0146] S203: The electronic device displays application information of the target application and subtitles generated by the target application in the first area.

[0147] In this embodiment, the electronic device determines that the played audio is valid audio, determines that the application holding the audio focus is the target application based on the played audio, obtains application information of the target application, and applies the application information in the first area. The application information may include at least one of the name and icon of the target application. For example, refer to Figure 6 As shown, Figure 6 A schematic diagram of an electronic device displaying application information of a target application and subtitles generated by the target application is provided. Figure 6 In the example, the target application is a video application, and the video application plays video 1. After determining that the audio output by video 1 is valid audio, the electronic device obtains application information of the video application and displays the icon 401 of the video application in the first area 400. At the same time, the electronic device generates the corresponding subtitles based on video 1 and displays them synchronously in the first area 400. Optionally, the time when the subtitles are started to be generated can also be displayed in the first area.

[0148] Optionally, if the user wants to save subtitles, refer to Figure 7 After executing S201-S203, the subtitle management method provided in this embodiment may further include:

[0149] S204: When the electronic device receives an operation from the user to save subtitles, the electronic device saves subtitles of the target application according to application information of the target application.

[0150] Exemplarily, when the electronic device receives an operation from the user to save subtitles, the electronic device performs corresponding classification and storage of subtitles according to the application information of the target application. Exemplarily, after acquiring the application information of the target application, the electronic device establishes a storage space with the application information of the target application, establishes a corresponding relationship between the subtitles generated after acquiring the application information and the application information, and upon receiving the operation to save subtitles, stores the subtitles in the storage space corresponding to the application information. Alternatively, after acquiring the application information of the target application, the electronic device writes the application information into the storage space. Upon receiving the operation to save subtitles, the generated subtitles are stored after the application information. Exemplarily, after acquiring the application information of the target application, the electronic device writes the application information into the memo of the electronic device. Optionally, the application information of the target application can be written into the memo as a title. Upon receiving the operation to save subtitles, the electronic device writes the generated subtitles into the memo under the title of the application information of the target application to achieve classification of the generated subtitles.

[0151] In some embodiments, if the change in playing audio is from application 1 playing audio to application 2 playing audio. When application 1 plays audio, the first area displays application information of application 1. When the electronic device determines that the audio played by application 2 is valid audio, it obtains application information of application 2, and updates the content of the display area of ​​the application information displayed in the first area from the application information of application 1 to the application information of application 2.

[0152] After switching between the audio of application 1 and application 2, the first area is enlarged and displayed, and the first area can simultaneously display the application information of application 1, the start time of generating the subtitles of application 1, the subtitle content of application 1, the application information of application 2, the start time of generating the subtitles of application 2, and the subtitle content of application 2. Figure 8 As shown, Figure 8 The first area includes an application icon 501 of application 1, the start time of generating subtitles of application 1, and the subtitle content of application 1, and an application icon 502 of application 2, the start time of generating subtitles of application 2, and the subtitle content of application 2. The first area also includes a storage control 503. When the electronic device receives the user's operation on the storage control 503, the electronic device can classify and store the corresponding subtitles according to the application information in the first area. For example, the subtitle content of application 1 is saved to the storage space / directory / title of application 1, and the subtitle content of application 2 is saved to the storage space / directory / title of application 2, and so on.

[0153] It should be noted that the first area displayed in full screen magnification can display the application information and subtitle content of all applications generated after the AI ​​subtitle function is turned on. Users can view the subtitle content by sliding down or up. The first area that is not fully magnified (default size) displays part of the subtitle content generated in real time.

[0154] In some embodiments, if the electronic device detects that the audio status has changed, the audio playback status changes to not playing, and there is no audio playback for a preset period of time, even if the user's operation to save subtitles is not received, the electronic device can directly store the corresponding subtitles according to the application information of the application, thereby realizing automatic classification storage of subtitles.

[0155] In the subtitle management method provided in the embodiment of the present application, the electronic device does not obtain the application information of the application running in the foreground. Instead, the electronic device obtains the application information of the application holding the audio focus when a change in the audio state is detected and the audio state meets a preset condition. The acquisition of application information based on holding the audio focus is more accurate than directly acquiring the application information of the application running in the foreground, and avoids the problem that the application holding the audio focus cannot obtain the application information of the application when it runs in the background.

[0156] In some embodiments, the subtitle management method provided by the embodiment of the present application is further introduced in combination with various modules in the electronic device, wherein the electronic device includes an AI subtitle module, an audio manager and an application management module, and the audio manager includes an audio management module and an audio playback configuration module. Fig. 9 As shown, including:

[0157] S301. The AI ​​subtitle module sends a registration request to the audio management module in response to the user's operation of turning on the AI ​​subtitle function.

[0158] The registration request is used to instruct the audio management module to perform operations such as registering the audio instance object of the system audio model.

[0159] S302: The audio management module obtains an audio instance object.

[0160] In some embodiments, the audio management module obtains the audio instance object of the system audio model corresponding to the audio and video currently playing by the electronic device.

[0161] S303: The audio management module registers an instance.

[0162] The audio management module calls the AudioPlaybackCallback method to register an abstract class instance. The abstract class instance is used to receive notifications of audio playback status updates for the currently playing audio or video.

[0163] S304: The audio management module registers audio playback monitoring.

[0164] The audio management module calls the registerAudioPlaybackCallback method to register and monitor the playback status of the currently playing audio and video.

[0165] S305. The audio management module rewrites the audio state changing method.

[0166] When registering and initializing the abstract class instance in the audio management module, the audio state change method (onPlaybackConfigChanged) can be overridden. The purpose of overriding the audio state change method (onPlaybackConfigChanged) is to call back the onPlaybackConfigChanged method when the audio state of the currently playing audio and video is detected to change, and return the instance object in the preset attribute list, where the preset attribute list can be set to AudioPlaybackConfiguration and the instance object is AudioAttribute.

[0167] S306: When the audio playback configuration module detects that the audio state has changed, it obtains an instance object of the audio state change.

[0168] After the audio management module rewrites the onPlaybackConfigChanged method, the audio playback configuration module can call the rewritten onPlaybackConfigChanged method. For example, when the audio playback configuration module detects that the audio state of the currently playing audio and video has changed, the onPlaybackConfigChanged method is called back, the instance object AudioAttribute in the AudioPlaybackConfiguration attribute list is returned, and the validity of the audio is judged based on the instance object AudioAttribute.

[0169] The instance object AudioAttribute is used to describe the session information of audio playback. For example, the session information of audio playback may include the source of audio playback, audio volume, audio playback status, etc. The source of audio playback refers to which application the audio playback comes from, or from the system of the electronic device. The audio playback status includes the states of play, pause, stop, etc. The audio volume is the playback decibel data.

[0170] In some embodiments, running application 1 to play audio, or switching to application 2 to play audio while playing audio of application 1 will change the audio state of the currently playing audio. At this time, onPlaybackConfigChanged will return the changed audio state callback of the currently playing audio to the instance object AudioAttribute.

[0171] Therefore, the audio playback configuration module can obtain the instance object AudioAttribute to obtain relevant information about the audio state of the currently playing audio.

[0172] S307: The audio playback configuration module obtains the playback channel of the currently playing audio.

[0173] After obtaining the instance object AudioAttribute, the audio playback configuration module can determine the audio validity based on the audio state of the currently playing audio described by the instance object AudioAttribute.

[0174] In some embodiments, the audio playback configuration module can call the getUsage method to obtain the playback channel of the currently playing audio, and based on the obtained playback channel of the currently playing audio, determine whether it is a media playback channel (USAGE_MEDIA). When it is determined that the playback channel of the currently playing audio is a media playback channel, the audio playback configuration module executes the judgment of the second condition, that is, executes S308.

[0175] S308: The audio playback configuration module obtains the audio type of the currently playing audio.

[0176] The audio playback configuration module can call the getContentType method to obtain the audio type of the currently playing audio, and based on the obtained audio type of the currently playing audio, determine whether it is a preset noise (CONTENT_TYPE_SONIFICATION). Among them, the preset noise can include the tapping sound of the mobile phone screen keyboard, system notification sound, alarm ringtone, incoming call ringtone, etc. If the audio type of the currently playing audio does not belong to any of the preset noises, the audio playback configuration module executes the judgment of the third condition, that is, executes S309.

[0177] S309: The audio playback configuration module obtains the playback status of the media player.

[0178] The audio playback configuration module calls the getPlayerState method in the AudioPlaybackConfiguration instance object through the Java reflection mechanism to obtain the playback state of the media player currently playing the audio. If the playback state of the media player is PLAYER_STATE, execute S310.

[0179] The audio playback configuration will only execute the operation of obtaining the application information of the application when it is determined that the media player is in the playback state, thus avoiding the redundant operation of obtaining the application information when the application is opened but the audio is not played.

[0180] S310. The audio playback configuration module obtains the uid of the application that plays the audio, and transmits the uid of the application to the application management module.

[0181] The audio playback configuration module uses the reflection mechanism to call the preset getClientUid method in the AudioPlaybackConfiguration instance object to obtain the uid of the target application currently playing the audio. The uid of the application is a unique identifier assigned to the application when it is installed. When determining that the currently playing audio is valid audio, the application (source) that is currently playing the audio can be determined to be the application that holds the audio focus (target application).

[0182] S311. The application management module obtains application information of the application according to the uid of the application.

[0183] The audio playback configuration module sends the uid of the target application to the application management module, and the application management module uses the uid to call a preset method to obtain application information such as the application package name, application name, and application icon of the target application.

[0184] Among them, the preset methods include android.content.pm.ApplicationInfo and android.content.pm.PackageManager.

[0185] The application management module can call the getPackageManager().getNameForUid(uid), getApplicationInfo(), and loadIcon() methods in android.content.pm.PackageManager to obtain application information such as the application package name, application name, and application icon.

[0186] Among them, include:

[0187] S312. The application management module obtains the application package name of the application according to the uid of the application.

[0188] The application management module obtains the application package name of the application through getPackageManager().getNameForUid(uid).

[0189] S313. The application management module obtains the application name of the application according to the uid of the application.

[0190] The application management module obtains the application name of the application through getPackageManager().getApplicationInfo().

[0191] S314. The application management module obtains the application icon of the application according to the uid of the application.

[0192] The application management module obtains the application icon through the getPackageManager().loadIcon() method.

[0193] S315. The application management module returns the application information of the application to the audio playback configuration module.

[0194] S316: The audio playback configuration module returns the application information of the application to the audio management module.

[0195] S317. The audio management module returns the application information of the application to the AI ​​subtitle module.

[0196] S318. The AI ​​subtitle module displays the subtitles and application information of the application.

[0197] S319. When the AI ​​subtitle module receives an operation from the user to save subtitles, the subtitles are saved according to the application information of the application.

[0198] In some embodiments, if the electronic device detects that the audio status has changed, the audio playback status changes to not playing, and there is no audio playback for a period of time, even if the user does not receive an operation to save subtitles, the electronic device can directly store the corresponding subtitles according to the application information of the application, thereby realizing automatic classification storage of subtitles.

[0199] When the user turns on the AI ​​subtitles function, the application information of the application holding the audio focus can be identified and obtained. While generating and displaying the subtitles of the audio and video played by the application holding the audio focus, the application information of the application holding the audio focus is displayed. When receiving the user's operation to save subtitles, the subtitles of the application can be stored by application classification based on the application information of the application holding the audio focus, realizing automatic classification storage of subtitles based on different applications, without relying on manual operations of the user, optimizing the user experience of using AI subtitles, and improving the information validity of the stored subtitles. Whether it is the first time to turn on the AI ​​subtitles function, or when turning on the AI ​​subtitles function, during the switching process of multiple audio and video playback applications, the electronic device can promptly obtain and display the application information of the application currently holding the audio focus when it detects a change in the audio status, and can also effectively classify the subtitles when saving the subtitles, thereby improving the information validity of the subtitles.

[0200] In some embodiments, the electronic device may also obtain one or more applications whose usage frequency is greater than a preset threshold according to the frequency of the user using the AI ​​subtitle function. When the user opens these applications, the entrance to the AI ​​subtitle function is recommended to the user through a floating window or a pop-up window, so that the user can more conveniently open the AI ​​subtitle function when using these applications.

[0201] In an embodiment of the present application, the electronic device can display a recommended entry for the AI ​​subtitle function in applications with higher frequency based on the frequency with which the user turns on the AI ​​subtitle function when running the application, making it more convenient for the user to turn on the AI ​​subtitle function and optimizing the user experience.

[0202] Correspondingly, for other preset auxiliary functions of electronic devices, such as screenshot, magnifying glass, full-screen note-taking and other auxiliary functions, electronic devices can provide users with corresponding entrances to auxiliary functions in floating windows or pop-up windows when users open the corresponding applications, based on the frequency of users using the corresponding auxiliary functions in different applications, thereby optimizing users' experience of using the auxiliary functions of electronic devices.

[0203] For example, when the frequency of starting preset function 1 when running application 1 is greater than a threshold, it is considered that preset function 1 can be used as a recommended function for application 1. In this case, when the electronic device receives an operation from the user to start application 1, it responds to the operation and displays a recommendation control for preset function 1 on the current display interface of application 1, so as to provide the user with an entry to start preset function 1.

[0204] In an embodiment of the present application, the electronic device can determine the correspondence between applications and preset functions based on the frequency with which users use corresponding auxiliary functions in different applications, so that when receiving a request from a user to open an application, a recommended control for opening the preset function is displayed, making it more convenient for users to open the corresponding preset function in the running application, thereby optimizing the user's usage experience.

[0205] In the above embodiments, when the application holding the audio focus is taken as one application, the AI ​​subtitle function can generate and display subtitle content corresponding to the audio and video played by the application holding the audio focus in the first area.

[0206] Optionally, in some embodiments, the application holding the audio focus may also be an application specified by the user, that is, the user may select an application from multiple applications that play audio and video as the application holding the audio focus. The AI ​​subtitle function generates and displays the subtitle content corresponding to the audio and video played by the application holding the audio focus in the first area, as well as the application information of the application holding the audio focus.

[0207] Optionally, in some embodiments, if the electronic device supports multiple audio focuses, that is, if the electronic device detects that multiple applications holding audio focuses are running in the foreground or background, the electronic device can display multiple first areas in the first interface. Each first area corresponds to an application holding audio focus, and each first area displays the subtitles generated by the audio and video played by the corresponding application holding audio focus. At the same time, each first area displays the application information of the corresponding application holding audio focus.

[0208] Optionally, in some embodiments, the application holding the audio focus may also be an application specified by the user, that is, the user may select multiple applications from multiple applications that play audio and video as the application holding the audio focus. The AI ​​subtitle function generates and displays the subtitle content corresponding to the audio and video played by the corresponding application holding the audio focus in each first area, as well as the application information of the corresponding application holding the audio focus.

[0209] Optionally, in some embodiments, the technical solution of displaying the source of subtitles (application icons) on the first interface can help users classify and organize subtitle manuscripts of different applications in scenarios such as online multinational meetings, life entertainment, etc. that require constant switching of multiple applications, which is convenient and quick.

[0210] In an embodiment of the present application, the AI ​​subtitle function can generate and display subtitles based on one or more applications holding audio focus, which can meet the user's needs of browsing subtitles in multiple applications at the same time in complex scenarios and optimize the user experience.

[0211] Fig.10 A possible structural diagram of the electronic device involved in the above embodiments is shown. Fig.10 The electronic device 1000 shown includes a processor 1001 , a display screen 1002 , and a storage module 1003 .

[0212] Among them, the processor 1001 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like. The storage module 1003 may be a memory.

[0213] For example, the processor 1001 may be Figure 1 The processor 310 shown; the display screen 1002 can be as shown Figure 1 The display screen 394 shown, the storage module 1003 can be as follows Figure 1 The internal memory 321 shown. The electronic device provided in the embodiment of the present application can be Figure 1 An electronic device 100 is shown.

[0214] The present application also provides a chip system (eg, a system on a chip (SoC)), such as Fig.11 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 can be interconnected by lines. For example, the interface circuit 702 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 702 can be used to send signals to other devices (such as a processor 701 or a camera of an electronic device). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send the instructions to the processor 701. When the instruction is executed by the processor 701, the electronic device can perform the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0215] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device, the electronic device executes each function or step executed by the electronic device 100 in the above-mentioned method embodiment.

[0216] The embodiment of the present application further provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the electronic device 100 in the above method embodiment. For example, the computer may be the above electronic device 100.

[0217] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0218] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0219] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0220] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0221] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0222] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A subtitle management method, characterized in that: The method comprises: The electronic device displays a first area on a first interface in response to a first operation of the user; the first operation is an operation of turning on artificial intelligence AI subtitles; When the electronic device detects a change in the audio playback state, if the audio currently being played by the electronic device meets a preset condition, the electronic device obtains application information of a target application corresponding to the currently playing audio; the target application corresponding to the currently playing audio is an application holding the audio focus; The electronic device displays subtitles corresponding to the currently playing audio and application information of the target application in the first area; the application information includes at least one of an application name and an application icon.

2. The method according to claim 1, characterized in that The method further comprises: In response to the user's operation of saving subtitles, the electronic device saves the subtitles corresponding to the currently playing audio to the storage space corresponding to the application information of the target application according to the application information of the target application.

3. The method according to claim 1, characterized in that The electronic device currently playing audio meets the preset condition including: The playing channel of the currently playing audio is a media channel, and the type of the currently playing audio does not belong to the preset noise type, and the state of the media player of the currently playing audio is a playing state; The preset noise type includes at least one of system background sound, system keyboard tapping sound, alarm ringtone, and incoming call ringtone.

4. The method according to claim 1 or 3, characterized in that: The audio playback status change includes any of the following: The electronic device changes from not playing audio to playing audio; The source of the currently playing audio of the electronic device is changed from the first application to the second application; Increase or decrease the volume of the currently playing audio of the electronic device; The playing state of the currently playing audio of the electronic device is changed from playing to paused, or the playing state of the currently playing audio is changed from paused to playing.

5. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: When the electronic device receives the first operation of the user, the electronic device initializes the audio instance object of the currently playing audio; the audio instance object is used to describe the session information of the currently playing audio, and the session information includes the source, playing volume and playing status of the currently playing audio; Before the electronic device detects a change in the audio playing state, the electronic device rewrites a preset audio state changing method; the preset audio state changing method is used to detect a change in the audio playing state of the currently playing audio.

6. The method according to claim 5, characterized in that When the electronic device detects a change in the audio playing state, if the audio currently being played by the electronic device meets a preset condition, the electronic device obtains application information of a target application of the currently playing audio, including: When the electronic device detects a change in the audio playing state, the electronic device calls the preset audio state change method to obtain the audio instance object of the currently playing audio; The electronic device determines, according to the session information of the currently playing audio described by the audio instance object, whether the currently playing audio satisfies the preset condition; If the audio currently being played by the electronic device meets a preset condition, the electronic device obtains application information of a target application of the currently playing audio.

7. The method according to claim 6, characterized in that The electronic device determines, according to the session information of the currently played audio described by the audio instance object, whether the currently played audio satisfies the preset condition, including: The electronic device acquires, according to the session information of the currently playing audio, a play channel of the currently playing audio, an audio type of the currently playing audio, and a state of a media player of the currently playing audio; If it is simultaneously satisfied that the playing channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the state of the media player of the currently playing audio is a playing state, the electronic device determines that the currently playing audio meets the preset condition; If the playback channel of the currently playing audio is not the media channel, or the type of the currently playing audio belongs to any one of the preset noise types, or the state of the media player of the currently playing audio is not the playing state, the electronic device determines that the currently playing audio does not meet the preset condition.

8. The method according to claim 6, characterized in that The electronic device obtains application information of the target application currently playing audio, including: The electronic device uses a reflection mechanism to call a preset method for obtaining an application identifier of the audio instance object to obtain an identifier of the target application currently playing the audio; The electronic device obtains application information of the target application based on the identifier of the target application.

9. The method according to any one of claims 1 to 8, characterized in that The method further comprises: In response to the user's opening operation of a preset application, the electronic device displays a recommendation control of an auxiliary function corresponding to the preset application in a second interface; the recommendation control is used to provide an opening entry for the auxiliary function; the auxiliary function includes at least one of AI subtitles, screenshots, and full-screen notes.

10. The method according to any one of claims 1 to 9, characterized in that The display size of the first area is smaller than the full screen size of the electronic device, and the method further includes: In response to a user's full-screen zoom operation on the first area, the electronic device zooms in the display size of the first area to a full-screen size to form a second area; The second area includes application information of at least one target application and subtitles corresponding to the currently playing audio of each target application.

11. An electronic device, characterized in that: The electronic device includes a memory, a display screen and one or more processors; the memory and the display screen are coupled to the processor; the memory stores computer program code, and the computer program code includes computer instructions, and when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 10.

Images