A method, device and computer-readable storage medium for controlling audio latency during screen projection

Abstract

This invention discloses a method, device, and computer-readable storage medium for controlling audio latency during screen mirroring. The method includes: monitoring the audio latency control status of the current program on the mobile device when mirroring from a mobile device to a computer; if the audio latency control status is enabled, determining the timing of modulated audio data being written to the program's audio track service based on a preset trigger condition; then, intercepting the writing of the modulated audio data to the program's audio track service and copying the modulated audio data; finally, the mobile device sends the copied modulated audio data to the audio playback component on the computer via the screen mirroring link. This achieves a user-friendly audio latency control scheme for screen mirroring, resulting in lower transmission latency and higher fidelity of the mirrored audio, significantly improving the user's screen mirroring experience.

Description

Technical Field

[0001] This invention relates to the field of mobile communications, and more particularly to a method, device, and computer-readable storage medium for controlling audio latency during screen projection. Background Technology

[0002] In current technologies, with the continuous development of smart terminal devices, users' demand for screen mirroring between multiple devices is also increasing. In particular, mobile game interfaces can provide a better gaming experience when mirrored to a larger computer screen. However, current screen mirroring solutions suffer from high latency in audio transmission, which undoubtedly causes significant interference and audio-visual desynchronization during gameplay. Specific analysis reveals that audio transmission during screen mirroring requires a series of system audio processing steps; for example, the system's DTS and Dolby audio effects introduce significant audio latency. Therefore, fundamentally solving the audio latency issue in screen mirroring has become a crucial user experience problem that needs to be addressed in screen mirroring applications. Summary of the Invention

[0003] To address the aforementioned technical deficiencies in the prior art, this invention proposes a method for controlling audio latency during screen projection, the method comprising:

[0004] When projecting screen from a mobile device to a computer, monitor the audio latency control status of the current program on the mobile device.

[0005] If the audio delay control state is enabled, the timing when the modulated audio data begins to be written to the program audio track service is determined by a preset trigger condition.

[0006] The modulated audio data is intercepted and written to the program's audio track service, and the modulated audio data is copied.

[0007] The mobile device sends the copied modulated audio data to the audio playback component on the computer via a screen mirroring link.

[0008] Optionally, when projecting screen from a mobile device to a computer, monitoring the audio latency control status of the current program on the mobile device includes:

[0009] Identify a first type of program within the mobile device that can be screen-cast, and parse the audio attribute information of the first type of program.

[0010] In the first type of program, a second type of program is obtained by filtering based on the audio attribute information to perform the modulated audio data transmission based on the program's audio track service.

[0011] Optionally, when projecting the screen from the mobile device to the computer, monitoring the audio latency control status of the current program on the mobile device further includes:

[0012] Configure an audio delay control list that includes programs of the second type.

[0013] The audio delay control list provides audio delay control switches for each program.

[0014] Optionally, when projecting the screen from the mobile device to the computer, monitoring the audio latency control status of the current program on the mobile device further includes:

[0015] Within the preset screen mirroring settings interface, a first setting command is received from the audio delay control switch to adjust the audio delay control status of any program in the audio delay control list.

[0016] Within the program interface of the currently projected program, an audio delay control icon is generated in an empty area, and the audio delay control status of the current program is adjusted according to the second setting instruction of the audio delay control icon.

[0017] Optionally, if the audio delay control state is enabled, the timing for determining when the modulated audio data begins to be written to the program audio track service is determined through a preset trigger condition, including:

[0018] Based on the audio attribute information, a trigger flag is preset in the second type of program to detect when the modulated audio data begins to be written to the program's audio track service.

[0019] If the confirmation prompt option is confirmed, then when the current program is projected from the mobile terminal to the computer terminal, and the trigger flag indicates that the modulated audio data has started to be written to the program's audio track service, the trigger condition is generated.

[0020] Optionally, if the audio delay control state is enabled, the step of determining the timing of the start of writing modulated audio data to the program audio track service through a preset trigger condition further includes:

[0021] When the audio delay control state is adjusted to the on state according to the first setting instruction and / or the second setting instruction, a confirmation prompt option is displayed in the screen casting settings interface or the program interface, indicating that all other audio except the current program cannot be played on the computer.

[0022] If the prompt option is confirmed, the timing of when the modulated audio data begins to be written to the program's audio track service will be determined through the triggering condition during the screen mirroring process.

[0023] Optionally, the step of intercepting the modulated audio data and writing it to the program audio track service, and copying the modulated audio data, includes:

[0024] Identify whether the current program contains multiple different types of modulated audio data transmitted by multiple program audio track services.

[0025] If present, the modulated audio data is copied and synthesized.

[0026] Optionally, the audio playback component that sends the copied modulated audio data from the mobile terminal to the computer terminal via a screen mirroring link includes:

[0027] Monitor the current program's transition from the first running stage with audio output to the second running stage without audio output.

[0028] When entering the second running phase, the system acquires and adjusts the audio events corresponding to the other audio generated during the first running phase, generates audio reminders corresponding to the audio events in the adjusted order, and sends the audio reminders to the audio playback component on the computer.

[0029] The present invention also proposes a screen projection audio delay control device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the screen projection audio delay control method as described in any of the preceding claims.

[0030] The present invention also proposes a computer-readable storage medium storing a screen projection audio delay control program, which, when executed by a processor, implements the steps of the screen projection audio delay control method as described in any of the above claims.

[0031] The screen mirroring audio latency control method, device, and computer-readable storage medium of this invention monitor the audio latency control status of the current program on the mobile device when mirroring from a mobile device to a computer. If the audio latency control status is enabled, the timing of the start of writing modulated audio data to the program's audio track service is determined through a preset trigger condition. Then, the writing of the modulated audio data to the program's audio track service is intercepted, and the modulated audio data is copied. Finally, the mobile device sends the copied modulated audio data to the audio playback component on the computer via the screen mirroring link. This achieves a user-friendly screen mirroring audio latency control scheme, resulting in lower transmission latency and higher fidelity of screen mirrored audio, significantly improving the user's screen mirroring experience. Attached Figure Description

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0033] Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal according to the present invention;

[0034] Figure 2 This is a communication network system architecture diagram provided in an embodiment of the present invention;

[0035] Figure 3 This is a flowchart of the first embodiment of the screen projection audio delay control method of the present invention;

[0036] Figure 4 This is a flowchart of the second embodiment of the screen projection audio delay control method of the present invention;

[0037] Figure 5 This is a flowchart of the third embodiment of the screen projection audio delay control method of the present invention;

[0038] Figure 6 This is a flowchart of the fourth embodiment of the screen projection audio delay control method of the present invention;

[0039] Figure 7 This is a flowchart of the fifth embodiment of the screen projection audio delay control method of the present invention;

[0040] Figure 8 This is a flowchart of the sixth embodiment of the screen projection audio delay control method of the present invention;

[0041] Figure 9 This is a flowchart of the seventh embodiment of the screen projection audio delay control method of the present invention;

[0042] Figure 10 This is a flowchart of the eighth embodiment of the screen projection audio delay control method of the present invention. Detailed Implementation

[0043] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0044] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0045] Terminals can be implemented in various forms. For example, the terminals described in this invention may include mobile terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0046] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to embodiments of the present invention can also be applied to fixed-type terminals.

[0047] Please see Figure 1 This is a schematic diagram of the hardware structure of a mobile terminal implementing various embodiments of the present invention. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0048] The following is combined with Figure 1 A detailed introduction to each component of the mobile terminal:

[0049] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), and TDD-LTE (Time Division Duplexing-Long Term Evolution).

[0050] WiFi is a short-range wireless transmission technology. Mobile terminals, through the WiFi module 102, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 102 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted as needed without changing the nature of the invention.

[0051] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the mobile terminal 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, etc. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the mobile terminal 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0052] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0053] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. The ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity and can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition-related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0054] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0055] User input unit 107 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands from processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being limited here.

[0056] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal. The specific implementation is not limited here.

[0057] Interface unit 108 serves as an interface through which at least one external device can connect to mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more elements within mobile terminal 100, or it may be used to transmit data between mobile terminal 100 and the external device.

[0058] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0059] The processor 110 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. The application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0060] The mobile terminal 100 may also include a power supply 111 (such as a battery) that supplies power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0061] although Figure 1 As not shown, the mobile terminal 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0062] To facilitate understanding of the embodiments of the present invention, the communication network system on which the mobile terminal of the present invention is based is described below.

[0063] Please see Figure 2 , Figure 2 This invention provides a communication network system architecture diagram. The communication network system is an LTE system based on the universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and the operator's IP services 204, which are connected in sequence.

[0064] Specifically, UE201 can be the aforementioned terminal 100, which will not be elaborated here.

[0065] E-UTRAN202 includes eNodeB2021 and other eNodeB2022s. Among them, eNodeB2021 can connect to other eNodeB2022s via backhaul (e.g., X2 interface), and eNodeB2021 connects to EPC203. eNodeB2021 can provide UE201 with access to EPC203.

[0066] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gateway) 2034, PGW (Packet Data Network Gateway) 2035, and PCRF (Policy and Charging Rules Function) 2036, etc. Among them, MME2031 is the control node that handles signaling between UE201 and EPC203, providing bearer and connection management. HSS2032 provides registers to manage functions such as the Home Location Register (not shown in the diagram) and stores user-specific information such as service characteristics and data rates. All user data can be sent through SGW2034. PGW2035 can provide UE 201 IP address allocation and other functions. PCRF2036 is the policy and charging control decision point for service data flow and IP bearer resources. It selects and provides available policy and charging control decisions for the policy and charging enforcement function unit (not shown in the figure).

[0067] IP services 204 may include the Internet, intranet, IMS (IP Multimedia Subsystem), or other IP services.

[0068] Although the above description uses the LTE system as an example, those skilled in the art should understand that the present invention is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems, etc., which are not limited here.

[0069] Based on the aforementioned mobile terminal hardware structure and communication network system, various embodiments of the method of the present invention are proposed.

[0070] Example 1

[0071] Figure 3 This is a flowchart of the first embodiment of the screen projection audio latency control method of the present invention. A screen projection audio latency control method, the method comprising:

[0072] S1. When casting the screen from a mobile device to a computer, monitor the audio latency control status of the current program on the mobile device.

[0073] S2. If the audio delay control state is enabled, the timing when the modulated audio data begins to be written to the program audio track service is determined by the preset trigger condition.

[0074] S3. Intercept the modulation audio data being written to the program audio track service, and copy the modulation audio data.

[0075] S4. The mobile terminal sends the copied modulated audio data to the audio playback component on the computer via a screen mirroring link.

[0076] Optionally, in this embodiment, when projecting screen from a mobile device to a computer, the audio latency control status of the current program on the mobile device is monitored. This audio latency control status is a setting specific to the program being projected; that is, for any program being projected, it is provided with both an on and off state for audio latency control, as described in this embodiment. Furthermore, the switching between these two states can be achieved via a switch, or the timing of the switching between the two states can be adaptively determined by the program's functionality.

[0077] Optionally, in this embodiment, if the audio latency control state is enabled, the timing of the start of writing modulated audio data to the program audio track service is determined through a preset trigger condition. Specifically, to reduce the transmission latency of the projected audio, this embodiment employs the following approach: directly obtain the modulated audio data (e.g., Pulse Code Modulation, PCM) written by the current program at the system framework layer, and then directly transmit this PCM data to the computer being projected for playback. This bypasses the processing of the program audio track service (audioflinger), i.e., it avoids the system's series of default sound effect processing stages, thereby reducing the transmission latency of the projected audio. Furthermore, since the above-mentioned solution in this embodiment also reduces the steps of audiooflinger in synthesizing various audio streams, the transmission latency of the projected audio can be further reduced.

[0078] Optionally, in this embodiment, the writing of the modulated audio data to the program audio track service is intercepted, and the modulated audio data is copied. As described in the example above, the purpose of interception is to prevent the aforementioned PCM data from being sent to a series of default audio effect processing stages of the system.

[0079] Optionally, in this embodiment, the mobile terminal sends the copied modulated audio data to the audio playback component on the computer via a screen mirroring link. As described in the example above, the audio playback component on the computer directly plays the transmitted data, significantly improving the audio fidelity and latency control.

[0080] Optionally, in this embodiment, the audio transmission during game projection is used as an example for explanation. Under current game projection solutions, high audio latency is a common complaint from most users and professional game streamers. The reported audio effects clearly exhibit a low-frequency enhancement; for example, gunshots and footsteps sound muffled. Obviously, distortion and high latency severely impact the gaming experience. Therefore, this embodiment analyzes the above problem. First, it confirms that the game application's own PCM data does not contain the aforementioned muffled gunshots or footsteps. Further analysis of the process reveals that the problem lies in the system's audiooflinger, which typically adds extra audio effect processing. Therefore, even with a low-latency solution that directly transmits data at the audiooflinger layer, the problem persists. Thus, this application directly obtains the PCM data written by the current program at the framework level and then directly transmits this PCM data to the projection computer. Since it does not require audiooflinger processing (i.e., there is no audio effect processing stage), a lower projection latency can be achieved.

[0081] The beneficial effect of this embodiment is that, when casting from a mobile device to a computer, the audio latency control status of the current program on the mobile device is monitored. If the audio latency control status is enabled, the timing of the start of writing modulated audio data to the program's audio track service is determined through preset trigger conditions. Then, the writing of the modulated audio data to the program's audio track service is intercepted, and the modulated audio data is copied. Finally, the mobile device sends the copied modulated audio data to the audio playback component on the computer via the casting link. This achieves a user-friendly audio latency control scheme for casting, resulting in lower transmission latency and higher fidelity of the cast audio, significantly improving the user's casting experience.

[0082] Example 2

[0083] Figure 4This is a flowchart of the second embodiment of the screen mirroring audio latency control method of the present invention. Based on the above embodiment, when mirroring from a mobile terminal to a computer terminal, monitoring the audio latency control status of the current program on the mobile terminal includes:

[0084] S11. Determine the first type of program that can be screen-cast within the mobile terminal, and parse the audio attribute information of the first type of program.

[0085] S12. In the first type of program, a second type of program is obtained by filtering based on the audio attribute information to perform the modulated audio data transmission based on the program audio track service.

[0086] Optionally, in this embodiment, considering that not all applications write PCM data within their own application's audiotrack—for example, some applications directly play MP3 format data, and the data is written by the mediaserver service—this embodiment filters out a second category of programs that transmit modulated audio data based on the program's audiotrack service according to the audio attribute information. That is, it identifies programs belonging to the audio writing modes that this embodiment can handle, forming a set of second-category programs. Within this set, any program can enable the low-latency mode provided by this embodiment.

[0087] The beneficial effect of this embodiment is that by identifying a first type of program within the mobile device that can be screen-casting, the audio attribute information of the first type of program is analyzed; then, within the first type of program, a second type of program is selected based on the audio attribute information to transmit the modulated audio data according to the program's audio track service. This provides a low-latency function enabling condition for realizing a user-friendly screen-casting audio latency control scheme, resulting in lower transmission latency and higher fidelity of screen-casting audio, significantly improving the user's screen-casting experience.

[0088] Example 3

[0089] Figure 5 This is a flowchart of the third embodiment of the screen mirroring audio latency control method of the present invention. Based on the above embodiment, when mirroring from a mobile terminal to a computer terminal, monitoring the audio latency control status of the current program in the mobile terminal further includes:

[0090] S13. Set an audio delay control list that includes the second type of program.

[0091] S14. Provide audio delay control switches for each program in the audio delay control list.

[0092] Optionally, in this embodiment, an audio delay control list containing the second type of program is set in the system's screen mirroring settings interface.

[0093] Optionally, in this embodiment, an audio delay control list containing the second type of program is set in the settings options during the screen casting process.

[0094] Optionally, in this embodiment, in addition to providing audio latency control switches for each program in the audio latency control list, options for enabling switches by time period and by scenario are added. For example, low latency control can be enabled during late-night hours, or in a gaming scenario.

[0095] The beneficial effect of this embodiment is that by setting an audio latency control list that includes the second type of programs, and then providing audio latency control switches for each program in the audio latency control list, a low-latency function management method is provided to achieve a user-friendly screen casting audio latency control scheme. This results in lower transmission latency and higher fidelity of screen casting audio, significantly improving the user's screen casting experience.

[0096] Example 4

[0097] Figure 6 This is a flowchart of the fourth embodiment of the screen mirroring audio latency control method of the present invention. Based on the above embodiment, when mirroring from a mobile terminal to a computer terminal, monitoring the audio latency control status of the current program on the mobile terminal further includes:

[0098] S15. Within the preset screen mirroring settings interface, receive the first setting instruction of the audio delay control switch to adjust the audio delay control status of any program in the audio delay control list.

[0099] S16. In the program interface of the current program being projected, generate an audio delay control icon in an empty area, and adjust the audio delay control state of the current program according to the second setting instruction of the audio delay control icon.

[0100] Optionally, in this embodiment, a first setting instruction for the audio delay control switch is received in the preset system screen casting settings interface or the screen casting settings interface during the screen casting process, so as to adjust the audio delay control status of any program in the audio delay control list.

[0101] Optionally, in this embodiment, an audio delay control icon is generated in an empty area within the program interface of the currently projected program, and the audio delay control state of the current program is adjusted according to a second setting instruction of the audio delay control icon. The second setting instruction can be a single click on the icon to toggle between on and off.

[0102] The beneficial effect of this embodiment is that, by receiving a first setting instruction from the audio latency control switch within a preset screen casting settings interface, the audio latency control status of any program in the audio latency control list can be adjusted; then, an audio latency control icon is generated in an empty area within the program interface of the currently cast program, and the audio latency control status of the current program is adjusted according to a second setting instruction from the audio latency control icon. This provides a way to enable or disable low-latency functionality for a user-friendly screen casting audio latency control scheme, resulting in lower transmission latency and higher fidelity of the cast audio, significantly improving the user's screen casting experience.

[0103] Example 5

[0104] Figure 7 This is a flowchart of the fifth embodiment of the audio delay control method for screen projection of the present invention. Based on the above embodiment, if the audio delay control state is enabled, the timing of the start of writing the modulated audio data to the program audio track service is determined through a preset trigger condition, including:

[0105] S21. Based on the audio attribute information, a trigger flag is preset in the second type of program to detect when the modulated audio data begins to be written to the program's audio track service.

[0106] S22. If the prompt option is confirmed, then when the current program is projected from the mobile terminal to the computer terminal, and the trigger flag indicates that the modulated audio data has started to be written to the program's audio track service, the trigger condition is generated.

[0107] Optionally, in this embodiment, using the PCM data written at the screen mirroring program level, a hook trigger is set at the framework level, and then directly transmitted to the computer for playback. This allows the playback of the screen mirroring program's most original audio data, that is, audio data without any sound effects processing. As described above, the above processing scheme in this embodiment will only be activated when low-latency mode is started.

[0108] The beneficial effect of this embodiment is that, through the audio attribute information, a trigger flag is preset in the second type of program to detect when the modulated audio data begins to be written to the program's audio track service; if the prompt option is confirmed, when the current program is projected from the mobile terminal to the computer terminal, when the trigger flag indicates that the modulated audio data has begun to be written to the program's audio track service, the trigger condition is generated. This provides a low-latency trigger condition for realizing a user-friendly screen projection audio latency control scheme, resulting in lower transmission latency and higher fidelity of the projected audio, significantly improving the user's screen projection experience.

[0109] Example 6

[0110] Figure 8 This is a flowchart of the sixth embodiment of the audio delay control method for screen projection of the present invention. Based on the above embodiment, if the audio delay control state is enabled, the timing of the start of writing the modulated audio data to the program audio track service is determined through a preset trigger condition, further including:

[0111] S23. When the audio delay control state is adjusted to the on state according to the first setting instruction and / or the second setting instruction, a confirmation prompt option is displayed in the screen casting settings interface or the program interface, indicating that all other audio except the current program cannot be played on the computer.

[0112] S24. If the prompt option is confirmed, during the screen mirroring process, the timing of when the modulated audio data begins to be written to the program audio track service is determined through the trigger condition.

[0113] Optionally, in this embodiment, since the steps of audiooflinger synthesizing various program audios are reduced in the above example, only the audio data of the screen casting program can be transmitted during the screen casting process, while other audio data cannot be transmitted, such as system sounds, ringtones, notification sounds, music, and other audio data cannot be transmitted to the computer at the same time.

[0114] Optionally, in this embodiment, to address the aforementioned problem, the solution adopted is to provide the user with explicit usage prompts and suggestions. For example, when the audio delay control state is adjusted to the enabled state according to the first setting instruction and / or the second setting instruction, a confirmation prompt option is displayed in the screen casting settings interface or the program interface, indicating that all audio except the current program cannot be played on the computer. Based on this, if the prompt option is confirmed, the timing of the modulation audio data being written to the program's audio track service is determined through the trigger condition during the screen casting process.

[0115] The beneficial effect of this embodiment is that when the audio latency control state is adjusted to the enabled state through the first setting instruction and / or the second setting instruction, a confirmation prompt option is displayed in the screen casting settings interface or the program interface, indicating that all other audio besides the current program cannot be played on the computer. If the prompt option is confirmed, the timing of the modulation audio data being written to the program's audio track service is determined through the trigger condition during the screen casting process. This provides a low-latency function usage reminder to achieve a user-friendly screen casting audio latency control scheme, resulting in lower transmission latency and higher fidelity of screen casting audio, significantly improving the user's screen casting experience.

[0116] Example 7

[0117] Figure 9 This is a flowchart of the seventh embodiment of the screen projection audio latency control method of the present invention. Based on the above embodiment, the step of intercepting the modulation audio data and writing it to the program audio track service, and copying the modulation audio data, includes:

[0118] S31. Identify whether the current program contains multiple different types of modulated audio data transmitted by multiple program audio track services.

[0119] S32. If present, copy and synthesize multiple modulated audio data.

[0120] Optionally, in this embodiment, considering that many games have built-in teammate voice call functions, when applying the screen projection audio latency control method of this embodiment, if there is teammate voice data and game process data, then for the PCM data of multiple audiotracks that need to be transmitted at the same time, this embodiment first synthesizes the PCM data of multiple audiotracks, and then outputs it to the playback component on the computer, such as a speaker, for playback.

[0121] The beneficial effect of this embodiment is that it identifies whether the current program contains multiple types of modulated audio data transmitted by multiple program audio track services; if so, it copies and synthesizes multiple types of modulated audio data. This provides an extended application of low-latency functionality to achieve a user-friendly screen projection audio latency control scheme, resulting in lower transmission latency and higher fidelity for screen projection audio, significantly improving the user's screen projection experience.

[0122] Example 8

[0123] Figure 10 This is a flowchart of the eighth embodiment of the screen-casting audio latency control method of the present invention. Based on the above embodiment, the audio playback component in which the mobile terminal sends the copied modulated audio data to the computer terminal via the screen-casting link includes:

[0124] S41. Monitor the current program's transition from the first running stage with audio output to the second running stage without audio output.

[0125] S42. When entering the second running stage, acquire and adjust the audio events corresponding to the other audio generated in the first running stage, generate audio reminders corresponding to the audio events in the adjusted order, and send the audio reminders to the audio playback component on the computer.

[0126] Optionally, in this embodiment, since audio from other programs or systems cannot be received when this solution is applied, a phase-differentiated processing scheme is proposed to avoid delays for users. Specifically, firstly, the current program is monitored to switch from a first running phase with audio output to a second running phase without audio output; then, when entering the second running phase, the audio events corresponding to the other audio generated in the first running phase are acquired and adjusted, audio reminders corresponding to the audio events are generated in the adjusted order, and the audio reminders are sent to the audio playback component on the computer.

[0127] Optionally, in this embodiment, various audio events in the first operating phase are modulated in order of importance, and then, when entering the second operating phase, they are played in the adjusted order.

[0128] The beneficial effect of this embodiment is that by monitoring the current program's transition from a first running stage with audio output to a second running stage without audio output; upon entering the second running stage, it acquires and adjusts the audio events corresponding to the other audio generated during the first running stage, generates audio reminders corresponding to the audio events in the adjusted order, and sends the audio reminders to the audio playback component on the computer. This provides a supplementary reminder method for low-latency audio control during the disabled phase, resulting in lower transmission latency and higher fidelity of the projected audio, significantly improving the user's projected audio experience.

[0129] Example 9

[0130] Based on the above embodiments, the present invention also proposes a screen projection audio delay control device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the screen projection audio delay control method as described in any of the above embodiments.

[0131] It should be noted that the above-described device embodiments and method embodiments belong to the same concept. The specific implementation process can be found in the method embodiments, and the technical features in the method embodiments are also applicable to the device embodiments, which will not be repeated here.

[0132] Example 10

[0133] Based on the above embodiments, the present invention also proposes a computer-readable storage medium storing a screen projection audio delay control program, which, when executed by a processor, implements the steps of the screen projection audio delay control method as described in any of the above embodiments.

[0134] It should be noted that the above-described medium embodiments and method embodiments belong to the same concept. The specific implementation process can be found in the method embodiments, and the technical features in the method embodiments are also applicable to the medium embodiments, which will not be repeated here.

[0135] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0136] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0137] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0138] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for controlling audio delay during screen projection, characterized in that, The method includes: When casting screen from a mobile device to a computer, monitor the audio latency control status of the current program on the mobile device; If the audio delay control state is enabled, the timing when the modulated audio data begins to be written to the program audio track service is determined by a preset trigger condition. The modulated audio data is intercepted and written to the program's audio track service, and the modulated audio data is copied. The mobile device sends the copied modulated audio data to the audio playback component on the computer via a screen mirroring link.

2. The screen projection audio delay control method according to claim 1, characterized in that, When projecting screens from a mobile device to a computer, monitoring the audio latency control status of the current program on the mobile device includes: Identify a first type of program within the mobile device that can be screen-cast, and parse the audio attribute information of the first type of program; In the first type of program, a second type of program is obtained by filtering based on the audio attribute information to perform the modulated audio data transmission based on the program's audio track service.

3. The screen projection audio delay control method according to claim 2, characterized in that, The step of monitoring the audio latency control status of the current program on the mobile device when casting the screen from the mobile device to the computer also includes: Configure an audio delay control list that includes programs of the second type; The audio delay control list provides audio delay control switches for each program.

4. The screen projection audio delay control method according to claim 3, characterized in that, The step of monitoring the audio latency control status of the current program on the mobile device when casting the screen from the mobile device to the computer also includes: Within the preset screen mirroring settings interface, a first setting instruction for the audio delay control switch is received to adjust the audio delay control status of any program in the audio delay control list. Within the program interface of the currently projected program, an audio delay control icon is generated in an empty area, and the audio delay control status of the current program is adjusted according to the second setting instruction of the audio delay control icon.

5. The screen projection audio delay control method according to claim 4, characterized in that, If the audio delay control state is enabled, the timing for the modulation audio data to begin being written to the program audio track service is determined through preset trigger conditions, including: Based on the audio attribute information, a trigger flag is preset in the second type of program to detect when the modulated audio data begins to be written to the program's audio track service; If the confirmation prompt option is confirmed, then when the current program is projected from the mobile terminal to the computer terminal, and the trigger flag indicates that the modulated audio data has started to be written to the program's audio track service, the trigger condition is generated.

6. The screen projection audio delay control method according to claim 5, characterized in that, If the audio delay control state is enabled, the timing for determining when modulated audio data begins to be written to the program audio track service is determined through a preset trigger condition, further including: When the audio delay control state is adjusted to the on state according to the first setting instruction and / or the second setting instruction, a confirmation prompt option is displayed in the screen casting settings interface or the program interface, indicating that all other audio except the current program cannot be played on the computer. If the prompt option is confirmed, the timing of when the modulated audio data begins to be written to the program's audio track service will be determined through the triggering condition during the screen mirroring process.

7. The screen projection audio delay control method according to claim 6, characterized in that, The interception of the modulated audio data being written to the program audio track service and the copying of the modulated audio data include: Identify whether the current program contains multiple different types of modulated audio data transmitted by multiple program audio track services; If present, the modulated audio data is copied and synthesized.

8. The screen projection audio delay control method according to claim 7, characterized in that, The audio playback component that sends the copied modulated audio data from the mobile terminal to the computer terminal via a screen mirroring link includes: Monitor the current program's transition from the first running stage with audio output to the second running stage without audio output; When entering the second running phase, the system acquires and adjusts the audio events corresponding to the other audio generated during the first running phase, generates audio reminders corresponding to the audio events in the adjusted order, and sends the audio reminders to the audio playback component on the computer.

9. A screen projection audio delay control device, characterized in that, The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the screen projection audio delay control method as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a screen projection audio delay control program, which, when executed by a processor, implements the steps of the screen projection audio delay control method as described in any one of claims 1 to 8.

Images