Subtitle management method and electronic device

Abstract

The application discloses a subtitle management method and an electronic device, and relates to the technical field of electronic devices, and comprises the following steps: in response to a user starting an artificial intelligence (AI) subtitle operation, an electronic device displays a first area on a first interface. When the electronic device detects a change in an audio playing state, if the current audio played by the electronic device meets preset conditions, the electronic device acquires application information of a target application of the current audio, and displays the subtitles corresponding to the current audio and the application information of the target application in the first area. In the application, the electronic device can acquire the source of the current audio and display the application information thereof, realizes automatic classification of subtitles, improves the information effectiveness of the subtitles, and optimizes the experience of using the AI subtitles by a user.

Description

Technical Field

[0001] This application relates to the field of electronic device technology, and in particular to a subtitle management method and an electronic device. Background Technology

[0002] In scenarios where users play videos and audio on electronic devices, they can enable the artificial intelligence (AI) subtitle function on their electronic devices for a better audio-visual experience. The AI ​​subtitle function is to convert the audio and video resources played by the electronic device into subtitle text, and display the corresponding subtitles on the display interface when the electronic device plays video or audio.

[0003] In scenarios where different applications switch between playing videos or audio, AI subtitles continuously generate subtitles for the played videos or audio. Existing technology cannot distinguish between AI subtitles for videos or audio played before and after the switch. All AI subtitles for videos or audio that have been played are displayed together, making it impossible for users to identify which subtitles belong to a particular video or audio, thus reducing the effectiveness of AI subtitle content. Summary of the Invention

[0004] This application provides a subtitle management method and an electronic device. When the electronic device detects a change in audio playback status, and if the currently playing audio meets preset conditions, it can obtain application information of the application corresponding to the currently playing audio. Based on the application information, the subtitles generated by the currently playing audio and the application information are displayed in a first area of ​​the first interface of the electronic device, improving the effectiveness of the subtitle content. Displaying the application information of the subtitle source on the display interface can optimize the user's subtitle browsing experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions.

[0006] Firstly, a subtitle management method is provided, which includes:

[0007] The electronic device responds to the user's first operation by displaying the first area on the first interface; the first operation is to enable AI subtitles.

[0008] When an electronic device detects a change in audio playback status, if the currently playing audio meets preset conditions, the electronic device obtains the application information of the target application corresponding to the currently playing audio, wherein the target application corresponding to the currently playing audio is the application that holds the audio focus.

[0009] The electronic device displays subtitles corresponding to the currently playing audio and application information of the target application in the first area; the application information includes at least one of the following: the application name and the application icon of the target application.

[0010] In this application, when the electronic device detects a change in the audio playback status, and if the currently playing audio meets preset conditions, it can obtain the application information of the application corresponding to the currently playing audio. Based on the application information, it displays the subtitles generated by the currently playing audio and the application information in the first area of ​​the first interface of the electronic device, thereby improving the effectiveness of the subtitle content. Displaying the application information of the subtitle source on the display interface can optimize the user's subtitle browsing experience.

[0011] In one possible implementation of the first aspect, the electronic device, in response to the user's operation to save subtitles, saves the subtitles corresponding to the currently playing audio to the storage space corresponding to the application information of the target application, based on the application information of the target application.

[0012] In this application, the electronic device can classify and process the subtitles generated from the currently playing audio based on application information, thereby realizing the automated classification and storage of subtitles. The subtitle management and storage does not rely on manual operation by the user, simplifying the subtitle management and storage process and effectively improving the utilization rate of effective subtitle information.

[0013] In another possible implementation of the first aspect, the preset conditions for the electronic device currently playing audio include:

[0014] The playback channel for the currently playing audio is a media channel, and the type of the currently playing audio does not belong to the preset noise type. Furthermore, the media player currently playing the audio is in playback mode. The preset noise type includes at least one of the following: system background sound, system keyboard typing sound, alarm clock ringtone, and incoming call ringtone.

[0015] In this application, the electronic device determines the validity of the currently playing audio by considering that the playback channel of the currently playing audio is a media channel, that the type of the currently playing audio does not belong to a preset noise type, and that the state of the media player playing the currently playing audio is the playback state. If the currently playing audio is valid, then the application playing the currently playing audio is considered to be the target application holding the audio focus, thereby obtaining the application information of the target application holding the audio focus and avoiding the situation of incorrectly obtaining application information.

[0016] In another possible implementation of the first aspect, the audio playback state change includes any of the following:

[0017] The electronic device changes from not playing audio to playing audio; the source of the currently playing audio on the electronic device changes from a first application to a second application; the playback volume of the currently playing audio on the electronic device increases or decreases; the playback status of the currently playing audio on the electronic device changes from playing to paused, or the playback status of the currently playing audio changes from paused to playing.

[0018] In this application, any change in the audio playback state of the electronic device will trigger the electronic device to determine the validity of the currently playing audio. This is because the change in the audio playback state may involve a switch of the audio source, which requires reconfirmation of the validity of the currently playing audio in order to obtain the application information of the switched audio source.

[0019] In another possible implementation of the first aspect, the method further includes:

[0020] When an electronic device receives the user's first operation, it initializes an audio instance object for the currently playing audio. The audio instance object is used to describe the session information of the currently playing audio, including the source of the currently playing audio, the playback volume, and the playback status.

[0021] Before the electronic device detects a change in the audio playback state, the electronic device rewrites the preset audio state change method; the preset audio state change method is used to detect changes in the audio playback state of the currently playing audio.

[0022] In this application, an electronic device can monitor the audio playback state by overriding a preset audio state change method, and when a change in the audio playback state is detected, it can call back to obtain the audio instance object of the currently playing audio, thereby determining the validity of the currently playing audio.

[0023] In another possible implementation of the first aspect, when the electronic device detects a change in the audio playback state, if the currently playing audio meets preset conditions, the electronic device obtains application information of the target application of the currently playing audio, including:

[0024] When an electronic device detects a change in audio playback status, it calls the audio status change method to obtain the audio instance object of the currently playing audio.

[0025] The electronic device determines whether the currently playing audio meets preset conditions based on the session information of the currently playing audio described by the audio instance object. If the currently playing audio meets the preset conditions, the electronic device obtains the application information of the target application of the currently playing audio.

[0026] In this application, the electronic device calls a rewritten preset audio state change method. When a change in audio playback state is detected, it obtains the audio instance object of the currently playing audio and then judges the validity of the currently playing audio. When it is determined that the currently playing audio meets the preset conditions, that is, the currently playing audio is valid audio, it determines that the source application of the currently playing audio is the target application holding the audio focus. In this way, the application information of the target application can be obtained, and the subtitles generated by the currently playing audio can be classified and managed based on the application information.

[0027] In another possible implementation of the first aspect, the electronic device determines whether the currently playing audio meets preset conditions based on the session information of the currently playing audio described by the audio instance object, including:

[0028] The electronic device obtains the playback channel, audio type, and media player status of the currently playing audio based on the session information of the currently playing audio.

[0029] If the playback channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the media player of the currently playing audio is in the playback state, the electronic device determines that the currently playing audio meets the preset conditions.

[0030] If the playback channel of the currently playing audio is not a media channel, or if the type of the currently playing audio belongs to any of the preset noise types, or if the media player of the currently playing audio is not in a playback state, the electronic device determines that the currently playing audio does not meet the preset conditions.

[0031] In this application, the electronic device determines the validity of the currently playing audio by considering that the playback channel of the currently playing audio is a media channel, that the type of the currently playing audio does not belong to a preset noise type, and that the state of the media player playing the currently playing audio is the playback state. If the currently playing audio is valid, then the application playing the currently playing audio is considered to be the target application holding the audio focus, thereby obtaining the application information of the target application holding the audio focus and avoiding the situation of incorrectly obtaining application information.

[0032] In another possible implementation of the first aspect, the electronic device obtains application information of the target application currently playing the audio, including:

[0033] The electronic device uses reflection to call the preset method for obtaining the application identifier of the audio instance object to obtain the identifier of the target application currently playing the audio; based on the identifier of the target application, the electronic device obtains the application information of the target application.

[0034] In this application, the electronic device obtains the application identifier corresponding to the audio instance object based on the reflection mechanism. The application corresponding to the audio instance object is a specific application that holds the audio focus. The electronic device obtains the application identifier of the application that holds the audio focus, and can obtain the corresponding application information based on the application identifier, thus avoiding the problem of obtaining the application information of applications that do not hold the audio focus.

[0035] In another possible implementation of the first aspect, the method further includes:

[0036] In response to the user's activation of a preset application, the electronic device displays recommended controls for the accessibility functions corresponding to the preset application on a second interface.

[0037] The recommended control is used to provide an entry point for enabling accessibility functions, which include at least one of AI captions, screenshots, and full-screen notes.

[0038] Electronic devices can determine the correspondence between preset applications and accessibility functions based on the frequency with which users activate accessibility functions when running applications. For example, if the frequency with which accessibility function 1 is activated when running application 1 is greater than a threshold, then accessibility function 1 is considered a recommended function of application 1. In this case, when the electronic device receives the user's operation to activate application 1, it responds by displaying a recommendation control for accessibility function 1 on the current display interface of application 1, providing the user with an entry point to activate accessibility function 1.

[0039] In this application, the electronic device can determine the correspondence between applications and accessibility functions based on the frequency with which users use corresponding accessibility functions in different applications. When the user opens an application, the device can display a recommended control for opening the accessibility function, making it more convenient for the user to open the corresponding accessibility function while running the application and optimizing the user experience.

[0040] In another possible implementation of the first aspect, the method further includes:

[0041] In response to the user's full-screen zoom operation on the first area, the electronic device updates the display size of the first area from the default size to the full-screen size, forming the second area.

[0042] The second area includes application information for at least one target application and subtitles corresponding to the currently playing audio of each target application.

[0043] In this application, the second area, after the first area is enlarged to full screen, displays all the subtitles generated after the AI ​​subtitle function was enabled. Users can browse all the subtitle content by swiping up or down. At the same time, the source of the subtitles is also displayed in the second area, which makes it more intuitive and clear for users to classify the source of the subtitle content.

[0044] In a second aspect, an electronic device is provided, comprising a memory, a display screen, and one or more processors; the memory, the display screen, and the processors are coupled together; the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of the first aspects above.

[0045] Thirdly, a computer-readable storage medium is provided that stores instructions which, when executed on an electronic device, cause the electronic device to perform any of the methods described in the first aspect.

[0046] Fourthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to perform the method described in any one of the first aspects above.

[0047] Fifthly, embodiments of this application provide a chip, the chip including a processor, the processor being configured to invoke a computer program in memory to execute the method as described in the first aspect.

[0048] It is understood that the beneficial effects of the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, the computer program product described in the fourth aspect, and the chip described in the fifth aspect can be referred to the beneficial effects of the first aspect and any of its possible design embodiments, which will not be repeated here. Attached Figure Description

[0049] Figure 1 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0050] Figure 2 A software structure block diagram of an electronic device provided in an embodiment of this application;

[0051] Figure 3 A flowchart illustrating a subtitle management method provided in an embodiment of this application;

[0052] Figure 4 This is a schematic diagram illustrating the enabling of AI subtitle functionality, provided as an embodiment of this application.

[0053] Figure 5 This is a schematic diagram illustrating another way to enable AI subtitle functionality, as provided in an embodiment of this application.

[0054] Figure 6 A schematic diagram illustrating an electronic device displaying application information of a target application and subtitles generated by the target application, provided in an embodiment of this application;

[0055] Figure 7 A flowchart illustrating another subtitle management method provided in this application embodiment;

[0056] Figure 8 This is a schematic diagram of a first area of ​​a fully enlarged display on an electronic device, provided in an embodiment of this application.

[0057] Figure 9 A flowchart illustrating another subtitle management method provided in this application embodiment;

[0058] Figure 10 This is a schematic diagram of the structure of another electronic device provided in an embodiment of this application;

[0059] Figure 11 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0060] In the description of the embodiments of this application, the terminology used in the following embodiments is for the purpose of describing specific embodiments only and is not intended to be a limitation of this application. As used in the specification and appended claims of this application, the singular expressions "a," "the," "the," "the," and "this" are intended to also include expressions such as "one or more," unless the context clearly indicates otherwise. It should also be understood that in the following embodiments of this application, "at least one" and "one or more" refer to one or more (including two). The term "and / or" is used to describe the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can indicate: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0061] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized. The term "connection" includes direct connections and indirect connections, unless otherwise stated. "First" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated.

[0062] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0063] With the advancement of electronic device technology, electronic devices can now offer users AI-powered subtitle functionality. When a user enables this feature, the device can convert the audio and video being played into spoken text and display the corresponding subtitles simultaneously. Users can also choose to save the subtitles to a specified storage space for later viewing. The emergence of AI subtitle functionality provides users with a superior audiovisual experience.

[0064] Generally, AI captioning functions generate and display captions for audio and video played by foreground applications. If multiple foreground applications are running on an electronic device, based on the device's "audio focus mechanism," the AI ​​captioning function can identify the application holding the audio focus and generate and display captions based on the audio and video played by that application. The "audio focus mechanism" refers to the ability of an electronic device to determine a unique application holding the audio focus when multiple foreground applications are playing multiple audio and video clips, based on preset rules. Optionally, the preset rules may include determining the application with the highest priority as the application holding the audio focus based on a pre-set application priority; or, the preset rules may include determining the application with the highest usage frequency as the application holding the audio focus. For audio and video played by applications that do not hold the audio focus, the electronic device can reduce the impact of these applications on the audio focus of the application by lowering their playback volume. If only one foreground application is playing audio and video, then that foreground application is the application holding the audio focus.

[0065] In scenarios where users actually use the AI ​​subtitle function on electronic devices, there may be situations where users switch applications. For example, when a user is watching a TV series through a video application, the only application running in the foreground is the video application, which holds the audio focus. When the user activates the AI ​​subtitle function, the AI ​​subtitle function generates and displays subtitles based on the TV series being played in the video application. For another example, in one scenario, if the user is interrupted while watching a TV series by an online meeting application (or an audio / video call in a communication application), the foreground application switches from the video application to the online meeting application (or communication application). Correspondingly, the application holding the audio focus also switches from the video application to the online meeting application (or communication application). After displaying the subtitles for the TV series, the AI ​​subtitle function will then generate and display subtitles corresponding to the audio and video played in the online meeting application (or communication application).

[0066] When a user wants to save subtitles, the generated subtitles may include subtitle content from multiple applications, such as video apps, online conferencing apps (or communication apps), etc. In this case, if the user wants to save subtitles separately for each application, they need to manually split the subtitles and save them individually based on the subtitle content. Alternatively, the user can save the subtitles generated by the TV series being played in the video app when the video app is interrupted, and then turn off the AI ​​subtitle function. After switching to the online conferencing app (or communication app), the AI ​​subtitle function can be turned on again to generate and save subtitles generated by the audio and video played in the online conferencing app (or communication app).

[0067] Regardless of the method mentioned above, the process of users obtaining and saving subtitles corresponding to different applications is cumbersome, which reduces the user experience of using AI subtitle functions. At the same time, since the generated subtitles include subtitle content from multiple applications playing audio and video, the effectiveness of the generated subtitle information is greatly reduced.

[0068] This application provides a subtitle management method. When a user enables the AI ​​subtitle function, the method can identify and obtain the application information of the application holding the audio focus. While generating and displaying subtitles for the audio and video played by the application holding the audio focus, the method also displays the application information of the application holding the audio focus. This can improve the effectiveness of AI subtitle content and optimize the user experience of using AI subtitles.

[0069] Furthermore, upon receiving a user's request to save subtitles, the system can categorize and store subtitles based on the application's information. This enables automatic categorization and storage of subtitles for different applications without relying on manual user intervention, thus optimizing the user experience of using AI subtitles and improving the effectiveness of the stored subtitle information.

[0070] The subtitle management method provided in this application can be applied to electronic devices. For example, the electronic device can be... Figure 1 The electronic device 100 shown can be a portable computer (such as a mobile phone), tablet computer, laptop computer, personal computer (PC), wearable electronic device (such as a smartwatch), augmented reality (AR) / virtual reality (VR) device, in-vehicle computer, etc. The following embodiments do not impose any special restrictions on the specific form of the electronic device.

[0071] Please refer to Figure 1This diagram illustrates a structural block diagram of an electronic device (such as electronic device 100) provided in an embodiment of this application. Electronic device 100 may include a processor 310, an external memory interface 320, an internal memory 321, a universal serial bus (USB) interface 330, a charging management module 340, a power management module 341, a battery 342, an antenna 1, a communication module 360, an audio module 370, a speaker 370A, a receiver 370B, a microphone 370C, a headphone jack 370D, a sensor module 380, a display screen 394, etc.

[0072] The structures illustrated in this embodiment of the invention do not constitute a limitation on the electronic device 100. It may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of both.

[0073] Processor 310 may include one or more processing units. For example, processor 310 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU), etc. The different processing units may be independent devices or integrated into one or more processors.

[0074] The aforementioned controller can be the decision-maker that directs the various components of the electronic device 100 to work in a coordinated manner according to instructions. It is the nerve center and command center of the electronic device 100. The controller generates operation control signals based on the instruction opcode and timing signals to complete the control of instruction fetching and execution.

[0075] The processor 310 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 310 is a cache memory, which can store instructions or data that the processor 310 has just used or that are used repeatedly. If the processor 310 needs to use the instruction or data again, it can directly retrieve it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 310, and thus improves the efficiency of the system.

[0076] In some embodiments, the processor 310 may include interfaces. Interfaces may include inter-integrated circuit (I2C) interfaces, inter-integrated circuit sound (I2S) interfaces, pulse code modulation (PCM) interfaces, universal asynchronous receiver / transmitter (UART) interfaces, mobile industry processor interfaces (MIPI), general-purpose input / output (GPIO) interfaces, SIM interfaces, and / or USB interfaces, etc.

[0077] The interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the electronic device 100. The electronic device 100 may employ different interface connection methods or combinations of multiple interface connection methods as illustrated in the embodiments of the present invention.

[0078] The charging management module 340 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 340 receives charging input from the wired charger via a USB interface 330. In some wireless charging embodiments, the charging management module 340 receives wireless charging input via the wireless charging coil of the electronic device 100. While charging the battery 342, the charging management module 340 can also supply power to the electronic device 100 via the power management module 341.

[0079] The power management module 341 connects the battery 342, the charging management module 340, and the processor 310. The power management module 341 receives input from the battery 342 and / or the charging management module 340, providing power to the processor 310, internal memory 321, external memory interface 320, display screen 394, camera 393, and communication module 360. The power management module 341 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some embodiments, the power management module 341 may also be located within the processor 310. In some embodiments, the power management module 341 and the charging management module 340 may also be located in the same device.

[0080] The wireless communication function of electronic device 100 can be realized through antenna 1, communication module 360, modem and baseband processor, etc.

[0081] Antenna 1 is used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, a cellular antenna can be reused as a wireless local area network diversity antenna. In some embodiments, the antenna can be used in conjunction with a tuning switch.

[0082] A modem may include a modulator and a demodulator. The modulator modulates a low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates a received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to a baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to an application processor. The application processor outputs sound signals through an audio device (not limited to a speaker 370A, a receiver 370B, etc.) or displays images or videos through a display screen 394. In some embodiments, the modem may be a standalone device. In some embodiments, the modem may be integrated into the same device as other functional modules, independent of the processor 310.

[0083] The communication module 360 ​​provides a communication processing module for wireless communication solutions applied to the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The communication module 360 ​​can be one or more devices integrating at least one communication processing module. The communication module 360 ​​receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 310. The communication module 360 ​​can also receive signals to be transmitted from processor 310, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 1.

[0084] In some embodiments, the antenna 1 of the electronic device 100 is coupled to the communication module 360, enabling the electronic device 100 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time-Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include satellite-based augmentation systems (SBAS), global navigation satellite system (GLONASS), BeiDou navigation satellite system (BDS), Quasi-Zenith satellite system (QZSS), and / or satellite-based augmentation systems (SBAS).

[0085] Electronic device 100 implements display functions through a GPU, display screen 394, and application processor. The GPU is a microprocessor for image processing, connecting the display screen 394 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 310 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0086] The display screen 394 is used to display images, videos, etc. For example, when the AI ​​caption function is enabled on the electronic device, the display screen 394 can display captions corresponding to the audio and video played by the application holding the audio focus. In this embodiment of the application, while displaying captions, the display screen 394 can also display application information of the application holding the audio focus, such as the application's name, icon, etc.

[0087] Optionally, in some embodiments, when the AI ​​subtitle function is not enabled on the electronic device, and audio or video is playing in a preset application, the display screen 394 may display a function recommendation control on the current display interface of that application. The function recommendation control provides the user with an entry point to enable the AI ​​subtitle function. The user can enable the AI ​​subtitle function through the function recommendation control, thereby allowing the display screen 394 to display subtitles corresponding to the audio or video played by the application holding the audio focus. The preset application can be determined based on the application type, usage frequency, etc.

[0088] In some embodiments, if the electronic device supports multiple applications holding audio focus, when the AI ​​subtitle function is enabled, the display screen 394 can display multiple pop-up windows, floating display interfaces, and other interface controls. In the multiple pop-up windows and floating display interfaces, the subtitles corresponding to the audio and video played by the application holding audio focus are displayed respectively.

[0089] The display screen 394 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a minimized LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include one or N displays 394, where N is a positive integer greater than 1.

[0090] The external storage interface 320 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 310 through the external storage interface 320 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0091] Internal memory 321 can be used to store computer executable program code, which includes instructions. Processor 310 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 321. Memory 321 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). Furthermore, memory 321 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, other volatile solid-state storage devices, universal flash storage (UFS), etc.

[0092] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 370, speaker 370A, receiver 370B, microphone 370C, headphone jack 370D, and application processor.

[0093] In some embodiments, the audio module 370 can determine, according to preset rules, the application that can hold the audio focus among one or more applications playing audio and video. For example, if the number of audio focuses is set to one, and the electronic device is currently playing audio and video using only one application, then the audio module 370 determines that the currently playing audio and video application is the application holding the audio focus. If the electronic device is currently playing audio and video using multiple applications, then the audio module 370 can determine the application holding the audio focus from among the multiple currently playing audio and video applications according to preset rules. The preset rules may include, for different applications, determining the application with the highest priority as the application holding the audio focus based on a pre-set application priority; or, the preset rules may also include, determining the application with the highest usage frequency as the application holding the audio focus based on the application's usage frequency, etc.

[0094] For example, the number of audio focus points can be set to N. If the electronic device is currently playing audio / video from only one application, then the audio module 370 determines that the application currently playing audio / video is the one holding the audio focus. If the electronic device is currently playing audio / video from N+M applications, then the audio module 370 can determine N applications holding the audio focus from the N+M applications currently playing audio / video according to preset rules. The preset rules may include determining the N applications with the highest priority as the applications holding the audio focus based on the preset application priority; or, the preset rules may also include determining the N applications with the highest usage frequency as the applications holding the audio focus, etc.

[0095] The audio module 370 can transmit the application information of the application holding the audio focus to the processor 310. The processor 310 can determine the application holding the audio focus based on the application information of the application holding the audio focus, and perform operations such as transmitting instructions for generating and displaying AI subtitles and processing data based on the audio and video played by the application.

[0096] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses a layered architecture. Taking the system as an example, the software structure of electronic device 100 is illustrated.

[0097] Figure 2 This is a software structure block diagram of an electronic device 100 according to an embodiment of the present invention. The layered architecture divides the software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and requires further context]. The system is divided into four layers, from top to bottom: application layer, application framework layer, etc. The runtime (Android runtime), system libraries, and kernel layer.

[0098] The application layer can include a series of application packages.

[0099] like Figure 2 As shown, the application package may include captioning apps, camera, gallery, calendar, phone, map, WLAN, Bluetooth, music, and other applications.

[0100] In this embodiment, the user can enable the AI ​​captioning function of the electronic device by launching a captioning application. Alternatively, in some embodiments, the user can enable the AI ​​captioning function through the system settings interface of the electronic device.

[0101] In some embodiments, if the electronic device has the AI ​​subtitle function enabled, when audio and video are detected, the subtitles corresponding to the audio and video being played can be displayed on the display interface. At the same time, the application information of the application playing the audio and video can also be displayed on the display interface. For example, the application information can be the application's icon, name, and other information.

[0102] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0103] like Figure 2 As shown, the application framework layer may include an audio manager, window manager, content provider, view system, phone manager, resource manager, and notification manager, etc.

[0104] In this embodiment, the audio manager is used to control and access the system audio model and volume of the electronic device. For example, after enabling the AI ​​caption function, the audio manager can detect the audio playback status of the electronic device and, based on the audio playback status, determine whether to perform operations such as switching between audio and video playback. That is, the audio manager can identify the application currently holding the audio focus based on the audio playback status.

[0105] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0106] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0107] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0108] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection, hang-up, etc.).

[0109] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0110] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of completed downloads or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0111] The runtime includes the core libraries and the virtual machine. runtime is responsible for System scheduling and management. The core library consists of two parts: one part contains the functionalities that the Java language needs to call, and the other part... The core library.

[0112] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0113] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0114] The Surface Manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The Media Library supports playback and recording of various common audio and video formats, as well as still image files. The Media Library supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D Graphics Processing Library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D Graphics Engine is the drawing engine for 2D graphics.

[0115] The kernel layer is the layer between hardware and software. The kernel layer contains at least the display driver, camera driver, audio driver, and sensor driver.

[0116] The following describes a subtitle management method provided by an embodiment of this application, using specific scenarios as examples. Figure 3 A flowchart illustrating a subtitle management method is provided, including:

[0117] S201. The electronic device receives the user's command to enable the AI ​​subtitle function and displays the first area on the first interface.

[0118] The AI ​​captioning function can be accessed through the system settings of an electronic device. Users can then enable the AI ​​captioning function by entering the system settings interface. For example, refer to... Figure 4 (a) is a schematic diagram of the system settings interface of an electronic device, which includes multiple function options such as "Applications", "Battery", "Storage", "Security", "Privacy", "Healthy Use of Phone", "Smart Assistant", "Wallet", "Accessibility", "Accounts", and "System Update". The entry point for the AI ​​captioning function can be set in "Smart Assistant", or alternatively, the entry point for the AI ​​captioning function can be set in other function options.

[0119] Taking the AI ​​caption function's entry point as an example, which can be set in the "Smart Assistant," if the electronic device receives a user's selection operation for the "Smart Assistant" function item 101, in response to this operation, the electronic device displays the "Smart Assistant" display interface 102. For example, refer to... Figure 4 (b) provides a schematic diagram of a display interface 102 for a "smart assistant," which may include a "AI caption" function and other functions belonging to the smart assistant, such as a "voice assistant." If the electronic device receives a user's selection of the "AI caption" function, it displays a details interface 103 for "AI captions" in response to the operation.

[0120] For example, refer to Figure 4 (c) provides a schematic of an AI subtitle details interface 103. The AI ​​subtitle details interface 103 may include an introduction to AI subtitles, an AI subtitle function switch 1031, and other functional items. For example, other functional items may include "Writing Mode," "Language," "Subtitle Font Size," and "Panel Transparency." The "Writing Mode" allows selection of subtitle or dialogue modes. The "Language" function may include the language used to generate subtitles, such as Chinese, English, Japanese, and Korean; or the "Language" function may include the translation language in dialogue mode, such as Chinese to English, English to Chinese, Chinese to Japanese, Japanese to Chinese, Chinese to Korean, and Korean to Chinese. The "Subtitle Font Size" function allows selection of the subtitle font size, such as large, medium, small, and tiny. The "Panel Transparency" function allows selection of the transparency of the panel containing the subtitles; the default transparency is 0%. The transparency can be adjusted to any integer value between 0% and 100%.

[0121] If the electronic device receives a user's activation operation via the AI ​​subtitle function switch 1031, it displays a first area in response to the operation. The first area displays the generated subtitles and some function options related to the subtitles. In this case, the first interface is the current "AI Subtitle" details interface 103.

[0122] In some embodiments, the default display size of the first region is smaller than the full-screen size. The default display size can be determined based on the size of the electronic device's display screen (full-screen size), for example, the default display size can be 1 / N times the full-screen size. Alternatively, the default display size can be a preset fixed value.

[0123] In this embodiment, the first area can also be understood as the subtitle display panel. Figure 4 In (c), adjusting the panel transparency means adjusting the transparency of the first area. For example, refer to... Figure 4 (e) provides a schematic diagram of the first region 105. The first region 105 includes subtitle content, a zoom-in control 1051 for the first region, a zoom-out control 1052 for the first region, a close control 1054 for the first region, and a control 1053 for switching between subtitle mode and dialogue mode.

[0124] If the electronic device is not currently playing any audio or video, the first area may not display any subtitles; alternatively, the "Subtitle Content" display area in the first area may display some characters to indicate that subtitles are pending generation. For example, see reference... Figure 4 In the “Subtitle Content” display area of ​​(e), the ellipsis is displayed with a static or flashing dynamic effect.

[0125] In some embodiments, if the electronic device receives a user's selection operation on the zoom-in control 1051 for the first area, the electronic device may respond to the operation by zooming in the first area to full screen. If the electronic device receives a user's selection operation on the zoom-out control 1052 for the first area, the electronic device may respond to the operation by shrinking the first area to the left / right edge of the display screen; or, shrinking the first area to the bottom of the display screen; or, shrinking the first area to a specified position on the display screen. If the electronic device receives a user's selection operation on the close control 1054 for the first area, the electronic device may respond to the operation by not displaying the first area and exiting the AI ​​caption function.

[0126] The electronic device responds to the user's selection of the zoom control 1051 for the first area and zooms in the first area to full screen. (See reference...) Figure 4 (f), where display area 106 is a schematic diagram of the first area 105 after full-screen magnification. The magnified display area 106 may include a storage control 1061, an exit magnification control 1062, a language selection control 1063, and a function selection control 1064. If the electronic device receives a user's selection operation on the storage control 1061, the electronic device can respond to the operation by displaying at least one list item of a preset storage space. Based on the user's selection of the list item of the storage space, the electronic device can store the generated subtitles in the storage space indicated by the list item. If the electronic device receives a user's selection operation on the exit magnification control 1062, the electronic device can respond to the operation by exiting full-screen magnification and restoring the default size display of the first area (see reference). Figure 4 (e) If the electronic device receives a selection operation from the user on the language selection control 1063, the electronic device can respond to the operation by displaying a language selection interface and switching between subtitle language and dialogue language. If the electronic device receives a selection operation from the user on the function selection control 1064, the electronic device can respond to the operation by displaying a function selection details interface and selecting and implementing the corresponding function in the function selection details interface.

[0127] Optionally, in some embodiments, when the electronic device receives a user's activation operation on the AI ​​caption function switch control 1031, the electronic device may also display permission reminder information about the AI ​​caption function before displaying the first area 105. For example, refer to... Figure 4 (d) In response to the user's activation of the AI ​​caption function via the on / off control 1031, the electronic device displays permission reminder information 104 in the "AI Caption" details interface 103. This information reminds the user that enabling the AI ​​caption function may compromise their privacy. For example, the permission reminder information may include: "Do you allow the 'Smart Assistant' to record / project your screen? If allowed, this application will record / project all content displayed on your screen. Please protect your sensitive personal information, such as audio, passwords, payment information, photos, messages, etc." The permission reminder information 104 may also include "Disable" and "Allow" controls. If the user selects the "Allow" control, the electronic device can obtain screen recording / projection permissions to enable the AI ​​caption function.

[0128] Alternatively, in another implementation, the entry point for the AI ​​captioning function can be located in a floating ball. This floating ball can be displayed on the main interface or application interface of the electronic device. The first interface can be the current display interface of the electronic device, such as the main interface or the display interface of the currently running application. Users can select the floating ball to access its function list interface, and then select the "AI Captioning" function item to enable the AI ​​captioning function.

[0129] For example, refer to Figure 5 (a) Taking the first interface as the main interface as an example, the main interface of the electronic device displays a floating ball 201, which provides access to preset functions. The electronic device can respond to the user's selection of the floating ball 201 by displaying a list of preset functions of the floating ball 201 on the main interface. (See reference) Figure 5 (b) The preset functions include AI subtitles, screenshots, etc. The preset function list interface 202 includes function items such as "AI subtitles" and "screenshots". In response to the user's selection of the function item "AI subtitles", the electronic device displays the first area 203 on the main interface, see reference. Figure 5 (c). The first area is used to display the generated subtitles and some functional options related to the subtitles. The controls or content included in the first area 203 can be found in [reference needed]. Figure 4 The relevant information about the first region 105 in (e) will not be elaborated here.

[0130] Optionally, the size of the first area can be fixed or adjustable. For example, users can adjust the height and width of the first area according to their actual usage scenario. When the length of the generated subtitle content exceeds the display area width of the first area, it can automatically wrap to continue generating subtitles. If the length of the subtitle content exceeds the area of ​​the first area, the first area can use a sliding display mechanism to hide the first subtitle generated in the first area and continue to display the newly generated subtitles in the first area.

[0131] It should be noted that when the first area is zoomed in to full screen (105), it displays all the subtitles generated after the AI ​​subtitle function was enabled. Users can browse all the subtitle content by swiping up or down.

[0132] In some embodiments, in response to a user's selection of the "AI captions" function, the electronic device may, in addition to displaying the first area 203, minimize the floating ball 201 to a side area of ​​the display screen or other designated location. (See reference) Figure 5 The display method of the floating ball 201 in (c).

[0133] S202. If the electronic device detects a change in the audio state and the currently playing audio meets preset conditions, it obtains the application information of the target application corresponding to the currently playing audio.

[0134] The target application corresponding to the currently playing audio is the application that holds the audio focus. The audio played by the application holding the audio focus is the currently playing audio, that is, the audio that the electronic device is playing after detecting a change in the audio state.

[0135] In this embodiment, changes in audio status include various scenarios. For example, the audio playback source changes, from application A to application B. The audio playback status changes, from not playing audio to playing audio, or from currently playing audio to paused / stopped playback. The audio volume changes, with the volume of the currently playing audio increasing or decreasing. The audio output device changes, from headphone output to speaker output, and so on.

[0136] When the electronic device detects any of the aforementioned changes in the audio state, it further determines whether the currently playing audio meets preset conditions. If the currently playing audio meets the preset conditions, it obtains the application information of the target application. In this embodiment, the target application refers to the application holding the audio focus. When the electronic device detects any of the aforementioned changes in the audio state, it can obtain the identifier of the application currently holding the audio focus; for example, the identifier can be the application's user identification (UID). The electronic device obtains the application information based on the application's UID. For example, the application information may include the application's icon, name, and other information.

[0137] Optionally, in some embodiments, the electronic device detects that the currently playing audio meets preset conditions to determine the validity of the playing audio, thereby determining the application holding the audio focus and obtaining the application information of the target application (the application holding the audio focus).

[0138] Further detection by the electronic device of whether the currently playing audio meets preset conditions may include:

[0139] In one scenario, the electronic device determines whether the current audio playback channel is a media playback channel. The electronic device can obtain the playback channel information of the currently playing audio; if the audio playback channel is a media playback channel, the electronic device determines that the currently playing audio is valid audio.

[0140] In another scenario, the electronic device determines whether the currently playing audio is a preset noise level. The device can determine this by obtaining the type of audio being played. Preset noise levels include system notification sounds, on-screen keyboard clicks, alarm clock sounds, incoming call ringtones, etc. If the currently playing audio does not belong to the preset noise level, the electronic device determines that the currently playing audio is valid.

[0141] Another scenario is that the electronic device determines whether the media player is currently in a playback state. If the media player is currently in a playback state, the electronic device determines that the currently playing audio is valid audio.

[0142] Other scenarios include the electronic device determining whether the currently playing audio is played by the application holding the audio focus; that is, determining whether the audio played by the application holding the audio focus has changed. If it has changed, the device determines that the currently playing audio meets preset conditions and obtains the application information of the new application holding the audio focus.

[0143] When an electronic device determines that all of the above conditions are met, it determines that the currently playing audio is valid audio and the application playing the valid audio is the application holding the audio focus. In this case, the electronic device obtains the application information of the application holding the audio focus.

[0144] In some embodiments, if the electronic device determines that the currently playing audio does not meet at least one of the above conditions, then it determines that the currently playing audio does not meet the preset conditions. For example, the playback channel of the currently playing audio is not a media playback channel; or, the currently playing audio belongs to a preset noise category; or, the media player's current playback state is not playing; or, the playback channel of the currently playing audio is not a media playback channel, and the currently playing audio belongs to a preset noise category, etc. If the electronic device determines that the currently playing audio is invalid audio and does not meet the preset conditions, it will not perform AI subtitle generation and display operations for the currently playing audio, nor will it perform application information acquisition operations.

[0145] After the electronic device obtains the application information of the target application, it can display the application information in the first area.

[0146] S203. The electronic device displays application information of the target application and subtitles generated by the target application in the first area.

[0147] In this embodiment, the electronic device determines that the played audio is valid, identifies the application holding the audio focus as the target application based on the played audio, and obtains the application information of the target application in the first area. The application information may include at least one of the following: the name and icon of the target application. For example, refer to... Figure 6 As shown, Figure 6 A schematic diagram is provided of an electronic device displaying application information of a target application and subtitles generated by the target application. Figure 6 In this example, the target application is a video application. The video application plays video 1. After determining that the audio output from video 1 is valid, the electronic device obtains the application information of the video application and displays its icon 401 in the first area 400. Simultaneously, the electronic device generates corresponding subtitles based on video 1 and displays them synchronously in the first area 400. Optionally, the first area may also display the time when subtitle generation began.

[0148] Optionally, if the user wants to save the subtitles, refer to Figure 7 The subtitle management method provided in this embodiment may further include, after executing S201-S203:

[0149] S204. When the electronic device receives an operation from the user to save subtitles, the electronic device saves the subtitles of the target application according to the application information of the target application.

[0150] For example, when an electronic device receives a user's request to save subtitles, the electronic device categorizes and saves the subtitles according to the application information of the target application. For example, after obtaining the application information of the target application, the electronic device establishes a storage space based on the application information, establishes a correspondence between the subtitles generated after obtaining the application information and the application information, and stores the subtitle in the storage space corresponding to the application information when a save subtitle request is received. Alternatively, after obtaining the application information of the target application, the electronic device writes the application information into the storage space. When a save subtitle request is received, the generated subtitle is stored after the application information. For example, after obtaining the application information of the target application, the electronic device writes the application information into the electronic device's memo. Optionally, the application information of the target application can be written as a title into the memo. When a save subtitle request is received, the electronic device writes the generated subtitle under the title of the application information of the target application in the memo, thus achieving the categorization of the generated subtitles.

[0151] In some embodiments, if the change caused by playing audio is from application 1 playing audio to application 2 playing audio, then when application 1 plays audio, the first area displays the application information of application 1. When the electronic device determines that the audio played by application 2 is valid audio, it obtains the application information of application 2 and updates the content of the display area of ​​the application information in the first area from the application information of application 1 to the application information of application 2.

[0152] After switching between audio playback for Application 1 and Application 2, the first area is enlarged for display. This first area can simultaneously display application information for Application 1, the start time of generating the subtitles for Application 1, the subtitle content for Application 1, application information for Application 2, the start time of generating the subtitles for Application 2, and the subtitle content for Application 2. (Reference) Figure 8 As shown, Figure 8 The first area includes an application icon 501 for application 1, the start time for generating the subtitles for application 1, and the subtitle content for application 1; and an application icon 502 for application 2, the start time for generating the subtitles for application 2, and the subtitle content for application 2. The first area also includes a storage control 503. When the electronic device receives a user's operation on the storage control 503, the electronic device can classify and store the corresponding subtitles according to the application information in the first area. For example, the subtitle content for application 1 can be saved to the storage space / directory / title of application 1, and the subtitle content for application 2 can be saved to the storage space / directory / title of application 2, and so on.

[0153] It should be noted that the first area in full-screen magnification displays application information and subtitle content for all applications generated after enabling the AI ​​subtitle function. Users can view the subtitle content by swiping down or up. The first area in non-full-screen magnification (default size) displays a portion of the subtitle content generated in real time.

[0154] In some embodiments, if the electronic device detects a change in the audio state, the audio playback state changes to not playing, and there is no audio playback for a preset period of time, the electronic device can directly store the corresponding subtitles according to the application information of the application, even if it does not receive the user's operation to save subtitles, thereby realizing the automatic classification and storage of subtitles.

[0155] The subtitle management method provided in this application does not obtain application information of the foreground running application. Instead, the electronic device obtains application information of the application holding the audio focus when it detects a change in the audio state, provided that the audio state meets preset conditions. Obtaining application information based on the application holding the audio focus is more accurate than directly obtaining application information of the foreground running application, and avoids the problem of not being able to obtain application information of the application holding the audio focus when it is running in the background.

[0156] In some embodiments, the subtitle management method provided in this application is further described in conjunction with various modules in an electronic device. The electronic device includes an AI subtitle module, an audio manager, and an application management module. The audio manager includes an audio management module and an audio playback configuration module. (Reference) Figure 9 As shown, it includes:

[0157] S301, the AI ​​subtitle module responds to the user's operation of enabling the AI ​​subtitle function by sending a registration request to the audio management module.

[0158] The registration request is used to instruct the audio management module to perform operations such as registering audio instance objects of the system audio model.

[0159] S302, The audio management module obtains the audio instance object.

[0160] In some embodiments, the audio management module obtains the audio instance object of the system audio model corresponding to the audio or video currently being played by the electronic device.

[0161] S303, Audio Management Module Registration Example.

[0162] The audio management module calls the AudioPlaybackCallback method to register an instance of an abstract class. This instance of the abstract class is used to receive notifications of audio playback status updates for the currently playing audio or video.

[0163] S304. Register audio playback monitoring in the audio management module.

[0164] The audio management module calls the registerAudioPlaybackCallback method to register a listener for the playback status of the currently playing audio or video.

[0165] S305, The audio management module rewrites the audio state change method.

[0166] When registering and initializing an instance of the abstract class in the audio management module, you can override the audio state change method (onPlaybackConfigChanged). The purpose of overriding the audio state change method (onPlaybackConfigChanged) is to call back the onPlaybackConfigChanged method when a change in the audio state of the currently playing audio or video is detected, returning an instance object from a preset property list. The preset property list can be set to AudioPlaybackConfiguration, and the instance object is AudioAttribute.

[0167] S306. When the audio playback configuration module detects a change in the audio state, it obtains the instance object of the changed audio state.

[0168] After the audio management module overrides the `onPlaybackConfigChanged` method, the audio playback configuration module can call the overridden `onPlaybackConfigChanged` method. For example, when the audio playback configuration module detects a change in the audio state of the currently playing audio / video, it calls back the `onPlaybackConfigChanged` method, returns an instance of the `AudioAttribute` from the `AudioPlaybackConfiguration` property list, and performs a validity check on the `AudioAttribute` instance.

[0169] The `AudioAttribute` instance describes the audio playback session information. For example, this session information may include the audio playback source, audio volume, audio playback status, etc. The audio playback source refers to which application or electronic device's system the audio is originating from. The audio playback status includes states such as play, pause, and stop. The audio volume is the playback volume in decibels.

[0170] In some embodiments, running application 1 to play audio, or switching to application 2 to play audio while playing audio from application 1, will change the audio state of the currently playing audio. In this case, onPlaybackConfigChanged will return the changed audio state of the currently playing audio to the instance object AudioAttribute.

[0171] Therefore, the audio playback configuration module can obtain the instance object AudioAttribute to get relevant information about the audio status of the currently playing audio.

[0172] S307, The audio playback configuration module obtains the playback channel of the currently playing audio.

[0173] After obtaining the instance object AudioAttribute, the audio playback configuration module can determine the validity of the audio based on the audio status of the currently playing audio described by the instance object AudioAttribute.

[0174] In some embodiments, the audio playback configuration module can call the getUsage method to obtain the playback channel of the currently playing audio, and determine whether it is a media playback channel (USAGE_MEDIA) based on the obtained playback channel of the currently playing audio. If it is determined that the playback channel of the currently playing audio is a media playback channel, the audio playback configuration module performs the second condition judgment, that is, executes S308.

[0175] S308, The audio playback configuration module obtains the audio type of the currently playing audio.

[0176] The audio playback configuration module can call the `getContentType` method to obtain the audio type of the currently playing audio. Based on the obtained audio type, it determines whether it is a preset noise (`CONTENT_TYPE_SONIFICATION`). Preset noise can include phone screen keystrokes, system notification sounds, alarm clock sounds, incoming call ringtones, etc. If the audio type of the currently playing audio does not belong to any of the preset noise types, the audio playback configuration module executes the third condition check, that is, executes S309.

[0177] S309, The audio playback configuration module obtains the playback status of the media player.

[0178] The audio playback configuration module uses Java reflection to call the getPlayerState method in the AudioPlaybackConfiguration instance object to obtain the playback status of the media player currently playing the audio. If the media player's playback status is Play (PLAYER_STATE), execute S310.

[0179] The audio playback configuration will only perform the operation of obtaining application information when it is confirmed that the media player is in a playback state, thus avoiding the redundant operation of obtaining application information even when the application is open but audio is not playing.

[0180] S310 The audio playback configuration module obtains the UID of the application playing the audio and transmits the application's UID to the application management module.

[0181] The audio playback configuration module uses reflection to call the predefined `getClientUid` method in the `AudioPlaybackConfiguration` instance object to obtain the UID of the target application currently playing the audio. The application's UID is a unique identifier assigned during application installation. Upon determining that the currently playing audio is valid, the application (source) playing the audio can be identified as the application holding the audio focus (target application).

[0182] S311. The application management module obtains the application information based on the application's uid.

[0183] The audio playback configuration module sends the target application's UID to the application management module. The application management module uses the UID to call a preset method to obtain application information such as the target application's package name, application name, and application icon.

[0184] The default methods include android.content.pm.ApplicationInfo and android.content.pm.PackageManager.

[0185] The application management module can call the getPackageManager().getNameForUid(uid), getApplicationInfo(), and loadIcon() methods in android.content.pm.PackageManager to obtain application information such as the application package name, application name, and application icon.

[0186] This includes:

[0187] S312. The application management module obtains the application package name of the application based on the application's uid.

[0188] The application management module obtains the application package name of the application through getPackageManager().getNameForUid(uid).

[0189] S313. The application management module obtains the application name based on the application's uid.

[0190] The application management module obtains the application name through getPackageManager().getApplicationInfo().

[0191] S314. The application management module obtains the application icon based on the application's uid.

[0192] The application management module obtains the application icon through the getPackageManager().loadIcon() method.

[0193] S315 The application management module returns the application information to the audio playback configuration module.

[0194] S316 The audio playback configuration module returns the application information to the audio management module.

[0195] S317. The audio management module returns the application information to the AI ​​subtitle module.

[0196] The S318 AI subtitle module displays subtitles and application information.

[0197] S319. When the AI ​​subtitle module receives the user's command to save subtitles, it saves the subtitles according to the application information.

[0198] In some embodiments, if the electronic device detects a change in the audio state, that the audio playback state changes to not playing, and there is no audio playback for a period of time, the electronic device can directly store the corresponding subtitles based on the application information of the application, even without receiving a user's subtitle saving operation, thereby achieving automatic classification and storage of subtitles.

[0199] When a user enables the AI ​​subtitle function, the system can identify and retrieve the application information of the application holding the audio focus. Simultaneously, it generates and displays subtitles for the audio and video played by the application with the audio focus, while also displaying the application information of that application. Upon receiving a user's subtitle saving operation, the system can automatically categorize and store subtitles based on the application information of the application with the audio focus, eliminating the need for manual user intervention. This optimizes the user experience of using AI subtitles and improves the effectiveness of the stored subtitle information. Whether it's the initial activation of the AI ​​subtitle function or during switching between multiple audio and video playback applications, the electronic device can promptly retrieve and display the application information of the application currently holding the audio focus when it detects a change in audio status. Furthermore, it can effectively categorize subtitles by application when saving them, further enhancing the effectiveness of the subtitle information.

[0200] In some embodiments, the electronic device can also identify one or more applications whose usage frequency exceeds a preset threshold based on the frequency of the user's use of the AI ​​captioning function. When the user opens these applications, the entry point for the AI ​​captioning function is recommended to the user through a floating window or pop-up window, making it easier for the user to access the AI ​​captioning function when using these applications.

[0201] In this embodiment, the electronic device can display a recommended entry for the AI ​​subtitle function in applications that are frequently used by the user, based on the frequency with which the user turns on the AI ​​subtitle function while running the application. This makes it more convenient for the user to turn on the AI ​​subtitle function and optimizes the user experience.

[0202] Correspondingly, for other preset auxiliary functions of electronic devices, such as screenshot, magnifier, full-screen notes, etc., the electronic device can provide the corresponding entry point of the auxiliary function to the user in the form of a floating window or pop-up window when the user opens the corresponding application, based on the frequency of the user's use of the corresponding auxiliary function in different applications, thereby optimizing the user's experience of using the auxiliary functions of electronic devices.

[0203] For example, if the frequency of enabling preset function 1 when running application 1 is greater than a threshold, then preset function 1 is considered to be a recommended function of application 1. In this case, when the electronic device receives the user's operation to enable application 1, it responds to the operation by displaying the recommended control of preset function 1 on the current display interface of application 1, so as to provide the user with the entry point to enable preset function 1.

[0204] In this embodiment, the electronic device can determine the correspondence between the application and the preset function based on the frequency with which the user uses the corresponding auxiliary function in different applications. Thus, when the user opens the application, a recommended control for opening the preset function is displayed, making it more convenient for the user to open the corresponding preset function while running the application and optimizing the user experience.

[0205] In the above embodiments, when the application holding the audio focus is considered as one application, the AI ​​subtitle function can generate and display subtitle content corresponding to the audio and video played by the application holding the audio focus in the first area.

[0206] Optionally, in some embodiments, the application holding the audio focus can also be a user-specified application; that is, the user can select one of multiple applications playing audio and video as the application holding the audio focus. The AI ​​captioning function generates and displays the caption content corresponding to the audio and video played by the application holding the audio focus, as well as the application information of the application holding the audio focus, in the first area.

[0207] Optionally, in some embodiments, if the electronic device supports multiple audio focuses—that is, if the electronic device detects that multiple applications holding audio focuses are running in the foreground or background—the electronic device can display multiple first areas on the first interface. Each first area corresponds to an application holding an audio focus, and each first area displays subtitles generated by the audio and video played by its corresponding application holding the audio focus. Simultaneously, each first area displays application information for its corresponding application holding the audio focus.

[0208] Optionally, in some embodiments, the application holding the audio focus can also be an application specified by the user. That is, the user can select multiple applications as the applications holding the audio focus from among multiple applications playing audio and video. The AI ​​captioning function generates and displays the caption content corresponding to the audio and video played by the application holding the audio focus in each first area, as well as the application information of the application holding the audio focus.

[0209] Optionally, in some embodiments, the technical solution of displaying the source (application icon) of the subtitle on the first interface can help users categorize and organize the subtitle text of different applications in scenarios such as online multinational conferences and entertainment, which require constant switching between multiple applications, making it convenient and quick.

[0210] In this embodiment, the AI ​​subtitle function can generate and display subtitles based on one or more applications that hold audio focus, which can meet the user's need to browse subtitles simultaneously in multiple applications in complex scenarios and optimize the user experience.

[0211] Figure 10 A possible structural schematic diagram of the electronic device involved in the above embodiments is shown. Figure 10 The electronic device 1000 shown includes a processor 1001, a display screen 1002, and a storage module 1003.

[0212] The processor 1001 may be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The processor may include an application processor and a baseband processor. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc. The storage module 1003 may be a memory.

[0213] For example, processor 1001 can be as follows: Figure 1 The processor 310 shown; the display 1002 can be as follows Figure 1 The display screen 394 shown, and the storage module 1003 can be as follows: Figure 1 The internal memory 321 is shown. The electronic device provided in this application embodiment can be... Figure 1 The electronic device 100 shown.

[0214] This application also provides a chip system (e.g., a system-on-a-chip (SoC)). Figure 11 As shown, the chip system includes at least one processor 701 and at least one interface circuit 702. The processor 701 and the interface circuit 702 are interconnected via lines. For example, the interface circuit 702 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 702 can be used to send signals to other devices (e.g., the processor 701 or the camera of an electronic device). Exemplarily, the interface circuit 702 can read instructions stored in the memory and send those instructions to the processor 701. When the instructions are executed by the processor 701, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete components, which are not specifically limited in this application embodiment.

[0215] This application also provides a computer-readable storage medium including computer instructions that, when executed on the electronic device, cause the electronic device to perform various functions or steps performed by the electronic device 100 in the above method embodiment.

[0216] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the electronic device 100 in the above method embodiments. For example, the computer may be the aforementioned electronic device 100.

[0217] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0218] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0219] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0220] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0221] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0222] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A subtitle management method, characterized in that, The method includes: When the electronic device receives the user's first operation, it initializes the audio instance object of the currently playing audio; the first operation is to enable AI subtitles. During the initialization of the audio instance object of the currently playing audio by the electronic device, the electronic device overrides a preset audio state change method; the preset audio state change method is used to detect changes in the audio playback state of the currently playing audio. The electronic device responds to the user's first operation by displaying a first area on a first interface; When the electronic device detects a change in audio playback status, if the currently playing audio meets preset conditions, the electronic device obtains the application information of the target application corresponding to the currently playing audio; the target application corresponding to the currently playing audio is the application holding the audio focus. The electronic device displays subtitles corresponding to the currently playing audio and application information of the target application in the first area; the application information includes at least one of the following: application name and application icon.

2. The method according to claim 1, characterized in that, The method further includes: In response to the user's operation to save subtitles, the electronic device saves the subtitles corresponding to the currently playing audio to the storage space corresponding to the application information of the target application, based on the application information of the target application.

3. The method according to claim 1, characterized in that, The electronic device is currently playing audio that meets preset conditions, including: The playback channel of the currently playing audio is a media channel, and the type of the currently playing audio does not belong to the preset noise type, and the state of the media player of the currently playing audio is the playback state; The preset noise types include at least one of the following: system background noise, system keyboard typing sound, alarm clock ringtone, and incoming call ringtone.

4. The method according to claim 1, characterized in that, The change in audio playback status includes any of the following: The electronic device changed from not playing audio to playing audio; The source of the currently playing audio on the electronic device is changed from the first application to the second application; The playback volume of the currently playing audio on the electronic device is increased or decreased; The playback status of the currently playing audio on the electronic device is changed from playing to paused, or the playback status of the currently playing audio is changed from paused to playing.

5. The method according to any one of claims 1-3, characterized in that, The audio instance object is used to describe the session information of the currently playing audio, and the session information includes the source, playback volume, and playback status of the currently playing audio.

6. The method according to claim 5, characterized in that, When the electronic device detects a change in audio playback status, if the currently playing audio meets preset conditions, the electronic device obtains application information of the target application of the currently playing audio, including: When the electronic device detects a change in the audio playback state, the electronic device calls the preset audio state change method to obtain the audio instance object of the currently playing audio; The electronic device determines whether the currently playing audio meets the preset conditions based on the session information of the currently playing audio described by the audio instance object; If the audio currently being played by the electronic device meets preset conditions, the electronic device obtains the application information of the target application of the currently playing audio.

7. The method according to claim 6, characterized in that, The electronic device determines whether the currently playing audio meets the preset conditions based on the session information of the currently playing audio described by the audio instance object, including: The electronic device obtains the playback channel, audio type, and media player status of the currently playing audio based on the session information of the currently playing audio. If the playback channel of the currently playing audio is a media channel, the type of the currently playing audio does not belong to the preset noise type, and the media player of the currently playing audio is in the playback state, the electronic device determines that the currently playing audio meets the preset conditions. If the playback channel of the currently playing audio is not the media channel, or if the type of the currently playing audio belongs to any of the preset noise types, or if the media player of the currently playing audio is not in a playback state, the electronic device determines that the currently playing audio does not meet the preset conditions.

8. The method according to claim 6, characterized in that, The electronic device acquires application information of the target application currently playing the audio, including: The electronic device uses a reflection mechanism to call a preset method for obtaining the application identifier of the audio instance object to obtain the identifier of the target application of the currently playing audio; The electronic device obtains the application information of the target application based on the identifier of the target application.

9. The method according to any one of claims 1-3, characterized in that, The method further includes: In response to a user's operation to open a preset application, the electronic device displays recommended controls for accessibility functions corresponding to the preset application on a second interface; the recommended controls are used to provide an entry point for opening the accessibility functions; the accessibility functions include at least one of AI captions, screenshots, and full-screen notes.

10. The method according to any one of claims 1-3, characterized in that, The display size of the first area is smaller than the full-screen size of the electronic device, and the method further includes: The electronic device responds to the user's full-screen zoom operation on the first area by enlarging the display size of the first area to the full-screen size, forming a second area; The second region includes application information for at least one target application and subtitles corresponding to the currently playing audio of each target application.

11. An electronic device, characterized in that, The electronic device includes a memory, a display screen, and one or more processors; the memory, the display screen, and the processors are coupled; the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-10.

12. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-10.

Images