Distributed audio playing method and electronic device

Abstract

Embodiments of the present application provide a distributed audio playing method and an electronic device, which can be applied to a first device and a second device, and can adaptively configure and play audio corresponding to an application window in the first device between the first device and the second device according to the position of the application window. The audio use experience of a user when using multiple devices is improved.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a distributed audio playback method and electronic device. Background Technology

[0002] With the development of terminal and display technologies, multi-device distributed display has brought increasing convenience to people's lives. Multi-device distributed display refers to achieving interface display through multiple electronic devices. For example, multiple electronic devices can be used to extend the display of an application interface.

[0003] For example, Figure 1 (a) illustrates a scenario where device B and device A are connected in a splicing manner to display a single interface across devices in a distributed manner. For example... Figure 1 As shown in (a), interface a1 is displayed on the screen of device A, and interface a2 is displayed on the screen of device B. Interface a1 and interface a2 are geometrically the two parts that make up interface a. For example... Figure 1 As shown in (b), interface a displayed on device A is extended to device B. After the extension, interface a is transferred from device A to device B. In some scenarios, such as when an interface on device A is extended to device B by dragging the interface, the display process of the interfaces on device A and device B typically includes the following steps: Figure 1 Scene (a) shown in the middle → Figure 1 The scenario shown in (b) is as follows. In this scenario... Figure 1 (a) or Figure 1 In the scenario shown in (b), device A is usually referred to as the primary device, and device B is usually referred to as the extended device or secondary device.

[0004] When an interface is distributed across multiple devices, users typically need to manually switch audio playback devices to transfer audio to the extended devices. This method is cumbersome and provides a poor user experience. Furthermore, this method cannot adaptively switch audio based on the distributed display of the interface. For example, in... Figure 1 In the extended display scenario shown in (a), it is not possible to switch audio reasonably using conventional methods. Summary of the Invention

[0005] This application provides a distributed audio playback method and electronic device, applicable to a first device and a second device. It adaptively distributes and plays audio corresponding to a window in the first device between the first and second devices based on the window's position. This improves the audio experience for users using multiple devices.

[0006] In a first aspect, the present application provides a distributed audio playing method. A first device displays a first application window, and plays a first audio corresponding to the first application window. In response to a first user operation, the first device displays a first part of the first application window, and a second device receives video data sent by the first device and displays a second part of the first application window according to the video data. At this time, the first device plays the first audio, and the second device plays the first audio. In response to a second user operation, the second device receives video data sent by the first device and displays the first application window according to the video data. At this time, the second device plays the first audio.

[0007] In a possible design, the first device monitors a position of the first application window in a display region corresponding to the first device, and determines whether to send the video data to the second device according to the position of the first application window. When the first application window is in the display region corresponding to the first device, the video data is not sent to the second device. When a first part of the first application window is in the display region corresponding to the first device and a second part of the first application window is in a display region corresponding to the second device, the video data is sent to the second device. When the first application window is in the display region corresponding to the second device, the video data is sent to the second device.

[0008] In a possible design, when an audio state of the first application window is a playing state, the first device sends the first audio to the second device. When the audio state of the first application window is at least one of a pausing playing state, a mute state and a quitting playing state, the first audio is not sent to the second device.

[0009] In a possible design, the first device and the second device play the first audio at a same volume.

[0010] In a possible design, the first device plays the first audio at a first volume, and the second device plays the first audio at a second volume. The first volume and the second volume are obtained by the first device according to the position of the first application window.

[0011] In a possible design, the first device obtains an audio stream associated with the first application window from an audio session list according to an identifier of the first application window, and generates the first audio according to the audio stream associated with the first application window.

[0012] In a possible design, the identifier of the first application window includes at least one of a window identifier, a browser identifier and a path.

[0013] In a possible design, a first application corresponding to the first application window runs on the first device.

[0014] In a possible design, the first user operation and the second user operation are operations of dragging the first application window.

[0015] In a possible design, the first device is a notebook computer, and the second device is a tablet computer.

[0016] In a second aspect, the present application provides an electronic device, comprising: one or more processors, one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program codes, the computer program codes comprising computer instructions, which, when executed by the one or more processors, cause the electronic device to perform a distributed audio playing method.

[0017] In a third aspect, the present application provides a computer storage medium, which stores a computer program, the computer program comprising program instructions, which, when executed by a processor, perform a distributed audio playing method.

[0018] The distributed audio playing method provided by the embodiments of the present application can automatically manage the audio corresponding to the window according to the position of the window when the user uses multiple electronic devices, without the need for the user to manually switch, thereby improving the audio use experience in the case of multiple devices. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 Two extended display scene example diagrams provided by the embodiments of the present application;

[0020] Figure 2 The hardware structure diagram of the electronic device provided by the embodiments of the present application;

[0021] Figure 3 The software structure diagram of the electronic device provided by the embodiments of the present application;

[0022] Figure 4 The distributed audio playing framework example diagram provided by the embodiments of the present application;

[0023] Figure 5A The first device plays a video scene example diagram provided by the embodiments of the present application;

[0024] Figure 5B The first device extends and displays the video playing interface to the second device scene example diagram provided by the embodiments of the present application;

[0025] Figure 5C Another first device extends and displays the video playing interface to the second device scene example diagram provided by the embodiments of the present application;

[0026] Figure 6 The distributed audio playing method flowchart provided by the embodiments of the present application;

[0027] Figure 7 An internal implementation logic block of a first device during distributed audio playback provided in this application embodiment. Figure 1 ;

[0028] Figure 8 A schematic diagram illustrating a method for determining a first part and a second part of an application interface, provided in an embodiment of this application;

[0029] Figure 9 An internal implementation logic block of a first device during distributed audio playback provided in this application embodiment. Figure 2 ;

[0030] Figure 10 An internal logic block for an audio playback module provided in this application embodiment Figure 1 ;

[0031] Figure 11 An internal logic block for an audio playback module provided in this application embodiment Figure 2 . Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0033] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0034] This application provides a distributed audio playback method, which is applied in a multi-device distributed display process. For example, this method can be applied to... Figure 1 (a) or Figure 1 In the scenario shown in (b) above, device A (i.e., the first device) and device B (i.e., the second device) are distributed and extended for display.

[0035] The first device and the second device in the present application establish a communication connection. The first device and the second device can realize information transmission between the first device and the second device by means of the established communication connection. The information transmitted between the first device and the second device includes but is not limited to application interface configuration parameters, video data, audio data, and control instructions.

[0036] For example, the first device and the second device can establish a wireless communication connection through "touch", "scan" (such as scanning a two-dimensional code or a bar code), "automatic discovery by proximity" (such as through Bluetooth (BT) or wireless fidelity (WiFi)), and the like. The first device and the second device can comply with a wireless transmission protocol to transmit information through a wireless connection transceiver. The wireless transmission protocol can include but is not limited to a Bluetooth transmission protocol or a WiFi transmission protocol, and the like. For example, the WiFi transmission protocol can be a WiFi P2P transmission protocol. The wireless connection transceiver includes but is not limited to a Bluetooth, WiFi, and the like. The first device and the second device realize information transmission through the established wireless communication connection.

[0037] For another example, the first device and the second device can establish a wired communication connection. For example, the first device and the second device establish a wired communication connection through a video graphics array (VGA), a digital visual interface (DVI), a high definition multimedia interface (HDMI), or a data transmission line, and the like. The first device and the second device realize information transmission through the established wired communication connection. The present application does not limit the specific connection mode between the first device and the second device.

[0038] The electronic device (e.g., the first device and the second device) in the present application includes one or more display screens. For example, the electronic device can be a smartphone, a netbook, a tablet computer, a smart camera, a palmtop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), an augmented reality (AR) / virtual reality (VR) device, a notebook computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), or the like. Alternatively, the electronic device can also be an electronic device of other types or structures including a display screen, and the present application is not limited.

[0039] Please refer to Figure 2 , Figure 2 For example, a hardware structure diagram of an electronic device provided by an embodiment of the present application is shown by taking a smartphone as an example. As shown in Figure 2 , the electronic device can include a processor 210, a memory (including an external memory interface 220 and an internal memory 221), a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a speaker 270A, a receiver 270B, a microphone 270C, a headset jack 270D, a sensor module 280, a key 290, a motor 291, an indicator 292, a camera 293, a display screen 294, and a subscriber identification module (SIM) card interface 295, and the like. The sensor module 280 can include a touch sensor 280A and a fingerprint sensor 280B. Further, in some embodiments, the sensor module 280 can also include one or more of a gyroscope sensor, an acceleration sensor, a magnetic sensor, a pressure sensor, an air pressure sensor, a distance sensor, a proximity light sensor, a temperature sensor, an ambient light sensor, or a bone conduction sensor, and the like.

[0040] It can be understood that the structure shown in the embodiment of the present application does not constitute a specific limitation on the electronic device. In another embodiment of the present application, the electronic device can include more or fewer components than the diagram, or combine certain components, or split certain components, or different arrangement of components. The components shown in the diagram can be implemented in hardware, software, or a combination of software and hardware.

[0041] The processor 210 can include one or more processing units. For example, the processor 210 can include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units can be independent devices or integrated in one or more processors.

[0042] The processor 210 can be the nerve center and command center of the electronic device. The processor 210 can fetch instructions according to the instructions, generate operation control signals, and then execute the control of the instructions.

[0043] In some embodiments of the present application, the processor 210 (for example, a controller in the processor 210) can be used to control the audio module 270 to collect audio data corresponding to the dragged interface window, make an audio switching decision, and control the output of the audio data corresponding to the dragged interface window to the corresponding device, etc.

[0044] The memory in the processor 210 can also be provided to store instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory can save instructions or data that the processor 210 has just used or repeatedly uses. If the processor 210 needs to use the instructions or data again, it can be directly called from the memory. This avoids repeated access and reduces the waiting time of the processor 210, thereby improving the efficiency of the system.

[0045] In some embodiments, the processor 210 can include one or more interfaces. The interfaces can include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0046] The charging management module 240 is configured to receive charging input from a charger. The charger can be a wireless charger or a wired charger. In some embodiments of wired charging, the charging management module 240 can receive charging input from a wired charger through the USB interface 230. In some embodiments of wireless charging, the charging management module 240 can receive wireless charging input through a wireless charging coil of the electronic device. The charging management module 240 can supply power to the electronic device through the power management module 241 while charging the battery 242.

[0047] The power management module 241 is configured to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to supply power to the processor 210, the internal memory 221, the display 294, the camera 293, and the wireless communication module 260, etc. The power management module 241 can also be configured to monitor parameters such as battery capacity, battery cycle count, battery health status (leakage, impedance), etc. In some embodiments, the power management module 241 can also be disposed in the processor 210. In other embodiments, the power management module 241 and the charging management module 240 can also be disposed in the same device.

[0048] The wireless communication function of the electronic device can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, a modem processor, and a baseband processor, etc.

[0049] Antenna 1 and antenna 2 are used for transmitting and receiving electromagnetic wave signals. Each antenna in the electronic device can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization of the antennas. For example: antenna 1 can be multiplexed as a diversity antenna for wireless local area networks. In some other embodiments, the antennas can be used in combination with a tuning switch.

[0050] The mobile communication module 250 can provide a solution including 2G / 3G / 4G / 5G wireless communication applied to the electronic device. The mobile communication module 250 can include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves by antenna 1, and perform filtering, amplification, etc. on the received electromagnetic waves, and transmit the processed signals to the modem processor for demodulation. The mobile communication module 250 can also amplify the signals modulated by the modem processor and convert them into electromagnetic waves radiated by antenna 1. In some embodiments, at least part of the functional modules of the mobile communication module 250 can be disposed in the processor 210. In some embodiments, at least part of the functional modules of the mobile communication module 250 can be disposed in the same device as at least part of the modules of the processor 210.

[0051] The modem processor can include a modulator and a demodulator. The modulator is used to modulate a low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. The low-frequency baseband signal processed by the baseband processor is transmitted to the application processor. The application processor outputs sound signals through the audio device (not limited to the loudspeaker 270A, the receiver 270B, etc.), or displays images or videos through the display screen 294. In some embodiments, the modem processor can be a separate device. In some other embodiments, the modem processor can be independent of the processor 210, and disposed in the same device as the mobile communication module 250 or other functional modules.

[0052] The wireless communication module 260 can provide a solution for wireless communication, including wireless local area networks (WLAN) (e.g., a WiFi network), Bluetooth (BT), a global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc., which are applied to the electronic device. The wireless communication module 260 can be one or more devices that integrate at least one communication processing module. The wireless communication module 260 receives an electromagnetic wave via the antenna 2, frequency-modulates and filters the electromagnetic wave signal, and transmits the processed signal to the processor 210. The wireless communication module 260 can also receive a signal to be transmitted from the processor 210, frequency-modulate it, amplify it, and radiate it as an electromagnetic wave via the antenna 2.

[0053] In some embodiments, the antennas 1 and the mobile communication module 250 of the electronic device are coupled, and the antennas 2 and the wireless communication module 260 are coupled, so that the electronic device can communicate with a network and other devices through wireless communication technology. The wireless communication technology can include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The GNSS can include global positioning system (GPS), global navigation satellite system (GLONASS), beidou navigation satellite system (BDS), quasi-zenith satellite system (QZSS), and / or satellite based augmentation systems (SBAS). In some embodiments of the present application, the first device and the second device can achieve information transmission between the first device and the second device based on the wireless communication technology by means of the respective antennas and the mobile communication module.

[0054] The electronic device implements a display function through a GPU, a display screen 294, an application processor, and the like. The GPU is a microprocessor for image processing, connected to the display screen 294 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 can include one or more GPUs that execute program instructions to generate or change display information. In embodiments of the present application, the display screen 294 is used to display images, videos, and the like. The display screen 294 includes a display panel. The display panel can adopt a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a Miniled, a MicroLed, a Micro-oLed, a quantum dot light emitting diode (QLED), and the like. In some embodiments, the electronic device can include 1 or N display screens 294, N being a positive integer greater than 1.

[0055] In embodiments of the present application, the GPU can be used to render an application interface, and correspondingly, the display screen 294 can be used to display the application interface rendered by the GPU. For example, in the scenario shown in (a) of FIG. 13, the GPU of the first device can be used to render interface a before the display is extended to the second device, and correspondingly, the display screen of the first device can be used to display the interface a rendered by the GPU. After the display of the first device is extended to the second device, the GPU of the first device can be used to render interface a before the display is extended to the second device, and interface a is split into interface a1 and interface a2, interface a1 is displayed on the display screen of the first device, and interface a2 is displayed on the display screen of the second device. Figure 1

[0056] The electronic device can implement a shooting function through an ISP, a camera 293, a video codec, a GPU, a display screen 294, and an application processor, and the like.

[0057] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to implement the expansion of the storage capacity of the electronic device. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function. For example, music, video, and the like files are saved in the external memory card.

[0058] ​The internal memory 221 can be used to store computer executable program codes including instructions. The internal memory 221 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function (such as a sound playing function, an image playing function, etc.), and the like. The data storage area can store data created during use of the electronic device (such as audio data, a phonebook, etc.), and the like. In addition, the internal memory 221 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, a universal flash storage (UFS), and the like. The processor 210 executes various function applications and data processing of the portable device by running instructions stored in the internal memory 221 and / or instructions stored in a memory disposed in the processor.

[0059] The touch sensor 280A can be referred to as a "touch panel". The touch sensor 280A can be disposed on the display screen 294, and the touch sensor 280A and the display screen 294 can constitute a touch screen, also referred to as a "touch panel". The touch sensor 280A is configured to detect a touch operation applied thereto or in the vicinity thereof. The touch sensor 280A can transmit the detected touch operation to the application processor to determine a touch event type. The electronic device can provide visual output related to the touch operation through the display screen 294, and the like. In other embodiments, the touch sensor 280A can also be disposed on the surface of the electronic device, which is different from the position where the display screen 294 is located. In the embodiments of the present application, the touch sensor of the first device can be used to detect the user's drag operation on the interface window of the display screen of the first device.

[0060] The fingerprint sensor 280B is configured to collect a fingerprint. The electronic device can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locking, fingerprint photographing, fingerprint answering a call, and the like.

[0061] The electronic device can implement audio functions through the audio module 270, the speaker 270A, the receiver 270B, the microphone 270C, the application processor, and the like. For example, music playing, recording, and the like. The specific working principles and roles of the audio module 270, the speaker 270A, the receiver 270B, and the microphone 270C can be referred to the introduction in the conventional technology.

[0062] In some embodiments of the present application, the audio module 270 can be used to collect audio data corresponding to the dragged window. The speaker 270A can be used to output the audio data corresponding to the dragged window. The microphone 270C can be used to collect sound in the environment, for example, collect the user's voice during a telephone video call.

[0063] The keys 290 include a power-on key, a volume key, and the like. The keys 290 can be mechanical keys. The keys 290 can also be touch keys. The electronic device can receive key input, and generate key signal input related to user settings and function control of the electronic device.

[0064] It should be noted that, Figure 2 The hardware modules included in the electronic device are described only by way of example, and do not limit the specific structure of the electronic device. For example, the electronic device can further include other functional modules.

[0065] In the present application, the operating system of the electronic device can include, but is not limited to Harmony, and the like. The present application is not limited.

[0066] Please refer to Figure 3 , Figure 3 Taking the Android operating system as an example, the software structure diagram of the electronic device in the embodiments of the present application is specifically introduced.

[0067] As Figure 3 shown, the Android operating system can include an application layer, an application framework layer (FWK), a system library, an Android runtime, and a kernel layer.

[0068] The application layer can provide some core applications. For convenience of description, the application is referred to as an application below. The applications in the application layer can include native applications (such as applications installed in the electronic device when the operating system is installed before the electronic device is shipped), for example Figure 3 shown, cameras, maps, music, short messages, galleries, emails, address books, Bluetooth, and the like. The applications in the application layer can also include third-party applications (such as applications downloaded and installed by the user through an application store), for example Figure 3 shown, and video applications, and the like.

[0069] The application framework layer provides application programming interfaces (APIs) and programming frameworks for the applications in the application layer. The application framework layer includes some pre-defined functions. As Figure 3 shown, the application framework layer can include a window manager service (WMS), an activity manager service (AMS), an input manager service (IMS), a resource manager, a notification manager, and a view system, and the like.

[0070] The WMS is mainly used for managing window programs. The window management server can acquire a display screen size, determine whether there is a status bar, lock a screen, and intercept a screen, etc.

[0071] The AMS is mainly used for managing activities, and is responsible for starting, switching, scheduling components in the system, and managing and scheduling applications.

[0072] The IMS is mainly used for translating and packaging original input events, obtaining input events containing more information, and sending the input events to the WMS. The WMS stores position information of clickable areas (such as controls) of each application program and a focus window. Therefore, the WMS can correctly distribute input events to specified controls or a focus window. For example, in the embodiment of the present application, the WMS can be used to distribute a received window dragging event to a specified control or a focus window.

[0073] The view system includes visual controls, such as a control for displaying text and a control for displaying pictures. The view system is mainly used for constructing an application program. A display interface can be composed of one or more views. For example, a display interface including a short message notification icon can include a view for displaying text and a view for displaying pictures. For example, in the embodiment of the present application, the view system can be used to construct a text control, a picture control, and the like on an application interface displayed on an electronic device.

[0074] The notification manager enables an application program to display notification information in a status bar, and can be used to convey a message of a notification type, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to notify a download completion, a message reminder, and the like. The notification manager can also be a notification in a form of a chart or a scroll bar text appearing in a top status bar of a system, such as a notification of a background running application, and can also be a notification in a form of a dialogue window appearing on a screen. For example, the notification manager can be used to prompt text information in a status bar, issue a prompt sound, vibrate an electronic device, and flick an indicator light, and the like. For example, in the embodiment of the present application, the notification manager can be used to notify a user that an expansion display is being expanded when the first device detects a user operation of dragging an interface window to the second device. Further, the notification manager can also be used to notify a user that an expansion display has been completed when the dragged interface window is completely expanded to the second device by the first device.

[0075] The resource manager is mainly used for providing various resources for an application program, such as localized strings, icons, pictures, layout files, video files, and the like. For example, in the embodiment of the present application, the resource manager can be used to provide localized strings, icons, pictures, layout files, or video files, and the like for an application interface displayed on an electronic device.

[0076] The system libraries and Android runtime contain the functionalities that FWK needs to call, the core Android libraries, and the Android virtual machine. System libraries can include multiple functional modules, such as the surface manager, 3D graphics processing libraries, 2D graphics engine, and media libraries.

[0077] The kernel layer is the foundation of the Android operating system; all the final functions of the Android operating system are implemented through the kernel layer. The kernel layer can contain display drivers, input / output device drivers (e.g., keyboard, touchscreen, headphones, speakers, microphones, etc.), device nodes, Bluetooth drivers, camera drivers, audio drivers, and sensor drivers. When users input through input devices, the kernel layer generates corresponding raw input events based on the input operations and stores them in the device node.

[0078] In this embodiment, the sensor driver can detect a user's swiping operation on the electronic device's display screen. When the swiping operation is a dragging operation of a certain interface window, the sensor driver reports the window dragging event to the IMS in the application framework layer. Upon receiving the dragging event reported by the IMS, the WMS distributes the window dragging event to the corresponding application to trigger the application interface window to move on the electronic device interface as the user drags it.

[0079] It should be noted that, Figure 3 This application uses the layered architecture of the Android system as an example to introduce a software structure for an electronic device. This application does not limit the specific architecture of the software system of the electronic device (including the first device and the second device). For details on other software systems, please refer to conventional technologies.

[0080] As described above, the distributed audio playback method provided in this application embodiment is applied to the process of multi-device distributed display. As one implementation, the multi-device distributed display in this application embodiment can be achieved based on the user dragging the application window. For example, based on the user dragging the first device (e.g., ...) Figure 1 Drag the interface displayed on device A) shown in (b) of 1 (as shown in interface a) to the extended device (i.e., the second device, such as...) Figure 1 In the operation of device B shown in (b), the interface dragged on the first device is extended to the second device. Specifically, during the process of the user dragging the interface (i.e., interface A) window displayed on the first device (i.e., device A) to the second device (i.e., device B), the first part of interface A (such as...) is extended to the second device. Figure 1 Interface a1) shown in (a) is displayed on the screen of the first device (i.e., device A), and the second part of interface a (as shown in (a)) is displayed on the screen of the first device (i.e., device A). Figure 1 Interface a2) shown in (a) is displayed on the screen of the second device (i.e., device B).

[0081] In the scenario that the user drags the interface window displayed on the first device to the second device as described above, the distributed audio playing method provided by the embodiments of the present application can adaptively switch the audio according to the distributed display of the interface. Specifically, when the user has not dragged the interface window or has not dragged the interface window out of the display screen of the first device, only the first device plays the audio corresponding to the interface. When the user drags the interface window to the state of distributed display across the devices as shown in (a) of FIG. 1B, the audio corresponding to the interface is switched to the second device, and the second device and the first device synchronously play the audio corresponding to the interface in a distributed manner. When the user drags the interface window to the state of display transfer as shown in (b) of FIG. 1B, the audio corresponding to the interface is switched from the first device to the second device, and the first device no longer plays the audio corresponding to the interface. Figure 1 Figure 1

[0082] Further, in some embodiments, during the process that the interface is dragged by the user from the first device to the second device, the first device can adaptively adjust the volume of the audio played by the first device and the second device in a distributed and synchronous manner according to the proportion of the first part of the interface (for example, the interface a1 shown in (a) of FIG. 1C) and the second part of the interface (for example, the interface a2 shown in (a) of FIG. 1C) on the display screen of the first device and the second device, respectively. Figure 1 Figure 1

[0083] The distributed audio playing method provided by the embodiments of the present application can be implemented based on the framework as shown in FIG. 1D. As shown in FIG. 1D, the distributed audio playing framework 400 provided by the embodiments of the present application can include a monitoring module 410, an audio output control module 420, a virtual audio module 430 and an audio playing module 440. Figure 4 Figure 4 The monitoring module 410 is mainly used for monitoring the positions of all interface windows displayed on the first device, so as to timely detect the window dragging event that the application window is dragged by the user to the second device. When the window dragging event that the application window is dragged by the user to the second device is detected, the monitoring module 410 sends the window dragging event to the audio output control module 420, so that the audio output control module 420 intercepts (for example, hooks) the window dragging event, and triggers the virtual audio module 430 to collect the audio data corresponding to the dragged interface window according to the window dragging event. In some embodiments, the monitoring module 410 can only detect the position of the focus window, that is, the position of the window last operated by the user, or the position of the foreground window.

[0084]

[0085] ​​​​​​Interception refers to capturing and monitoring the transmission of an event before it reaches its destination. For example, using hook technology, an event can be hooked like a hook before it reaches its destination, allowing for timely processing and response. In this embodiment, the audio output control module 420 can respond to a window dragging event immediately upon interception.

[0086] Furthermore, the monitoring module 410 is also used to monitor the audio status corresponding to all interfaces displayed on the first device that are accompanied by audio information, such as playing, paused, muted, or exited playback, so as to determine the specific strategy for distributed playback of audio on the first and second devices based on the specific audio status. For example, Figure 1 In the scenario shown in (b), when the audio is playing, the first device switches the audio corresponding to the interface from the first device to the second device for playback; when the audio is muted, paused, or exited, the switching of the audio corresponding to the interface from the first device to the second device is abandoned.

[0087] In one possible configuration, a portion of the monitoring module 410 may be located within the sensor module of the electronic device (e.g., Figure 2 In the sensor module 280 shown, a portion is located in the audio module of the electronic device (such as...). Figure 2 The audio module 270 shown is included. The monitoring module 410's functionality can be partly derived from the electronic device's sensor module (such as...). Figure 2 The sensor module 280 shown is implemented, and part of it is implemented by the audio module of the electronic device (such as...). Figure 2 The audio module 270 shown is used for implementation. For example, the sensor module of the electronic device is used to monitor the position of all interface windows displayed on the first device and, when a window dragging event is detected by the user on the second device, to send the window dragging event to the audio output control module 420. The audio module of the electronic device is used to monitor the audio status corresponding to all interfaces with audio information displayed on the first device, such as playing, paused, muted, or exited playback.

[0088] In another possible configuration, the monitoring module 410 may be located in the processor of the electronic device (e.g., Figure 2 In the processor 210 shown. For example, the monitoring module 410 of the first device can receive data from the sensor module of the first device (such as...). Figure 2 The sensor module 280 (shown) detects a window drag event where the application window is dragged by the user to the second device, and sends this window drag event to the audio output control module 420. The monitoring module 410 of the first device can also receive audio signals from the first device's audio module (e.g., ...). Figure 2The audio module 270) detects the audio state corresponding to all the interfaces displayed on the first device that are accompanied by audio information. The present application does not limit the specific arrangement of the monitoring module 410 in the electronic device (such as the first device).

[0089] The audio output control module 420 is mainly used to intercept (for example, hook) the window dragging event from the monitoring module 410 when the monitoring module 410 detects that the user drags the window of the application interface on the first device to the second device, and trigger the virtual audio module 430 to collect the audio data corresponding to the dragged interface window according to the window dragging event.

[0090] The window dragging event also carries information such as the position to which the window is dragged. Further, the audio output control module 420 is also used to make an audio switching decision according to the position to which the window is dragged, such as whether to switch the audio from the first device to the second device, how to switch the audio (such as distributed synchronous playback or complete switching, etc.). In addition, the audio output control module 420 is also used to control the switching of the audio data corresponding to the dragged interface window between the first device and the second device after determining how to switch the audio, such as controlling the output of the audio data corresponding to the dragged interface window to the corresponding device.

[0091] Further, the audio output control module 420 is also used to control the volume of the audio corresponding to the dragged interface window played on the first device and / or the second device. For example, by sending a control instruction to the first device and / or the second device to control the volume of the audio corresponding to the dragged interface window played on the first device and / or the second device.

[0092] In one possible structure, the audio output control module 420 can be located in the processor (such as the processor 210) of the electronic device. For example, the audio output control module 420 can be located in the controller in the processor 210. For another example, the audio output control module 420 can be located in the processor 210 independently of the controller in the processor 210. The present application does not limit the specific arrangement of the audio output control module 420 in the electronic device (such as the first device). Figure 2 Figure 2 In one possible structure, the audio output control module 420 can be located in the processor (such as the processor 210) of the electronic device. For example, the audio output control module 420 can be located in the controller in the processor 210. For another example, the audio output control module 420 can be located in the processor 210 independently of the controller in the processor 210. The present application does not limit the specific arrangement of the audio output control module 420 in the electronic device (such as the first device).

[0093] The virtual audio module 430 is mainly used to collect the audio data corresponding to the interface during the process in which the monitoring module 410 detects that the interface window on the first device is dragged by the user to the second device, so that the audio output control module 420 controls the output of the audio data corresponding to the dragged interface window collected by the virtual audio module 430 to the corresponding device after making an audio switching decision according to the position to which the window is dragged.

[0094] ​In one possible structure, the virtual audio module can be located within the audio module of the electronic device (e.g., ...). Figure 2 In the audio module 270 shown. Specifically, Figure 2 The audio module 270 shown may include a first audio module and a second audio module. The first audio module is a conventional audio module, and the second audio module is a virtual audio module 430.

[0095] In another possible structure, the virtual audio module can also be independent of... Figure 2 The audio module 270 is shown. In this case, it is necessary to... Figure 2 A virtual audio module 430 has been added to the hardware structure diagram shown. In this case... Figure 2 The audio module 270 shown is a conventional audio module. This application does not limit the specific configuration of the virtual audio module in an electronic device (such as the first device).

[0096] The conventional audio module is primarily used to input the audio data corresponding to the dragged interface window to the audio playback module 440 of the first device for playback, and to adjust the volume of the audio playback module 440 when playing the audio data corresponding to the dragged interface window according to the instructions of the audio output control module 420. Furthermore, the conventional audio module can also be used to input the audio data corresponding to an un-dragged interface window displayed on the first device to the audio playback module 440 of the first device for playback.

[0097] The audio playback module 440 is mainly used to receive audio data from the conventional audio module and play the audio. The audio data from the conventional audio module can be the audio data corresponding to a dragged interface window, or the audio data corresponding to an undragged interface window displayed on the first device. If the audio data from the conventional audio module corresponds to the audio data of a dragged interface window, the audio playback module 440 is also used to control the audio playback volume according to the volume control instructions from the conventional audio module. In one possible structure, the audio playback module 440 can be… Figure 4 The speaker shown is 270A.

[0098] It should be noted that, Figure 4 This is merely an example of a distributed audio playback framework, where the monitoring module 410, audio output control module 420, virtual audio module 430, and audio playback module 440 are only one possible module division method. The distributed audio playback framework in this embodiment may have more than Figure 4 The more or fewer modules shown can be combined into two or more modules, or they can have different module divisions. Additionally, Figure 5AThe various modules shown in the figures can be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing or application specific integrated circuits.

[0099] The following will take the first device as a notebook computer, the second device as a tablet computer, the dragged interface window as a video application window, and take the state change of Figure 5B → Figure 5C → Figure 5A The following will take the first device as a notebook computer, the second device as a tablet computer, the dragged interface window as a video application window, and take the state change of

[0100] The following will take the first device as a notebook computer, the second device as a tablet computer, the dragged interface window as a video application window, and take the state change of Figure 5A The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5B The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5B The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5B The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6.

[0101] The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5C The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5C The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5A The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6.

[0102] The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5B The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5A The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6.

[0103] The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5B The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 5C The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6. Figure 6 The video application window 501 shown in FIG. 5, wherein the video application window 501 includes a video playing interface 502. The first device, in response to the user dragging the video application window 501 from the state shown in FIG. 5 to the state shown in FIG. 6, drags the video application window 501 to the state shown in FIG. 6.

[0104] As shown in FIG. 5, the first device displays a video application window 501, wherein the video application window 501 includes a video playing interface 502. Figure 5AAs shown, the distributed audio playing method provided in the embodiments of the present application can include the following steps S601-S606:

[0105] S601, the first device displays a video playing interface 502 and synchronously plays audio corresponding to the video playing interface 502. The video playing interface 502 is located in a video application window 501.

[0106] The first device displays the video playing interface 502 and synchronously plays the audio corresponding to the video playing interface 502 specifically means that the first device plays the video in the video playing interface 502 and plays the audio corresponding to the video at the same time.

[0107] For example, the first device can obtain video data corresponding to the video playing interface 502 from the video application through a video driver of the first device, display the video data to a display screen to play the video through the display screen, and synchronously obtain audio data corresponding to the video playing interface 502 from the video application through a conventional audio driver (such as an audio driver corresponding to a conventional audio module described above) of the first device, and play the audio through a loudspeaker. For a specific implementation process of the first device playing the video and the audio synchronously, refer to the introduction in the conventional technology, which will not be described here.

[0108] S602, when detecting that a user drags the video application window 501, the first device obtains real-time position information of the video application window 501.

[0109] For example, in the embodiments of the present application, the first device can detect a sliding operation of a user on the display screen of the first device through a sensor driver. When the sliding operation is a dragging operation on a certain interface window (such as the video application window 501 shown in the figure), the first device obtains the real-time position information of the video application window 501. Figure 5A

[0110] The real-time position information of the video application window 501 is real-time coordinate information of the video application window 501 in a preset coordinate system.

[0111] For example, the preset coordinate system can be a preset coordinate system of the first device, a world coordinate system, or a ground coordinate system, which is not limited in the present application. For example, the preset coordinate system of the first device can be a two-dimensional coordinate system corresponding to the display screen of the first device. For example, the first device is a notebook computer as shown in the figure. Figure 5A For example, the two-dimensional coordinate system can be a coordinate system composed of the lower left corner of the notebook computer as the coordinate origin O, the lower side as the x-axis, and the left side as the y-axis in the state shown in the figure. Figure 7 For example, the two-dimensional coordinate system can be a coordinate system composed of the lower left corner of the notebook computer as the coordinate origin O, the lower side as the x-axis, and the left side as the y-axis in the state shown in the figure. ​

[0112] It should be noted that in the embodiments of the present application, the first device constantly acquires real-time position information of the video application window 501 in the process of being dragged by the user. By constantly acquiring the real-time position information of the video application window 501, the first device can accurately acquire the timing of extending the display of the video playing interface 502 to the second device by the first device, so as to respond in time.

[0113] In the embodiments of the present application, when the first device detects the operation of the user dragging the video application window 501, the first device can monitor the position of the video application window 501 in real time through the monitoring module 410, so as to acquire the position information of the video application window 501 in real time through the sensor driver.

[0114] For example, the monitoring module 410 of the first device constantly monitors the sensor driver in real time, and timely intercepts (for example, hooks) the window dragging event. As shown in Figure 7 the monitoring module 410 intercepts the window dragging event after monitoring the window dragging event. And, for example, in the process of the window being dragged, the monitoring module 410 acquires the window process information of the dragged window (such as the video application window 501) in real time. Wherein, the window process information includes the position information of the window.

[0115] For example, the monitoring module 410 can acquire the following position information of the dragged window (such as the video application window 501) in real time through the sensor driver: the coordinates of the upper left corner of the dragged window in the preset coordinate system (such as left_up_x, left_up_y), the coordinates of the lower right corner of the dragged window in the preset coordinate system (such as right_down_x, right_down_y). For another example, the monitoring module 410 can acquire the following position information of the dragged window (such as the video application window 501) in real time through the sensor driver: the coordinates of the lower left corner of the dragged window in the preset coordinate system (such as left_down_x, left_down_y), the coordinates of the upper right corner of the dragged window in the preset coordinate system (such as right_up_x, right_up_y).

[0116] Further, as shown in Figure 7 the window process information of the dragged window can also include information such as window identification (such as pid), browser identification (such as parentId) and path (such as path). As Figure 5AAs shown, during the process of dragging a window, the monitoring module 410 can, for example, also obtain window process information such as the window identifier (e.g., pid), browser identifier (e.g., parentId), and path (e.g., path) of the dragged window. The browser identifier can be used to filter out the tab that is playing audio from multiple tabs in the browser.

[0117] S603. When the first device detects that the video application window 501 has crossed the edge of the first device's display screen, it obtains the audio status corresponding to the video playback interface 502, and determines the first part and the second part of the video playback interface 502 based on the real-time position of the video application window 501.

[0118] In some embodiments, when the first device detects that the video application window 501 extends beyond the edge of the first device's display screen, it means that a portion of the video application window 501 extends beyond the edge of the display area corresponding to the first device's display screen. A portion of the video application window 501 is displayed in the display area corresponding to the second device's display screen. The display area corresponding to the second device's display screen is also used for drawing and rendering operations on the first device. It can be a virtual display area set by the first device, i.e., it is not displayed on the first device, but can be used for application drawing and rendering.

[0119] In this embodiment of the application, the audio state corresponding to the video playback interface 502 may include, but is not limited to, any of the following: playing, paused, muted, or exited playback.

[0120] In scenarios where video and audio are played simultaneously, such as for Figure 7 In the scenarios shown, if the video is playing but the audio is muted, the audio status is muted; if both the video and audio are playing, the audio status is playing; if both the video and audio are paused, the audio status is paused; and if both the video and audio have exited playback, the audio status is exited. For purely audio playback scenarios, such as playing music through a music application, "mute" means the audio is muted during playback, "playing" means the music is currently playing, "paused" means the music is paused, and "exited" means the music has exited playback.

[0121] In this embodiment of the application, the first device can monitor the audio status corresponding to the video playback interface 502 in real time through the monitoring module 410, so as to update the status corresponding to the audio session in real time when the audio status changes.

[0122] Please refer to Figure 7 ,like Figure 7 As shown, the monitoring module 410 of the first device obtains the audio status corresponding to the interface in the dragged window by real-time monitoring the session notification unit, such as... Figure 7The start, pause / stop, mute, and exit playing are shown.

[0123] In an example, in the embodiment of the present application, an audio session list is maintained in the monitoring module 410. The audio session list includes session information of all audio sessions currently running in the first device.

[0124] As shown in the figure, Figure 7 The monitoring module 410 of the first device maintains an audio session list. The audio session list includes session information such as session window identifier (e.g., pid), session browser identifier (e.g., parentId), session state (e.g., state), and session path (e.g., path). The session window identifier refers to the identifier of the window in which the video playing interface 502 corresponding to the audio session is located. The session browser identifier refers to the identifier of the browser in which the video playing interface 502 corresponding to the audio session is located. The session state includes an active state and an inactive state. For example, if the audio state is start or mute, the state of the audio session is the active state; if the audio state is pause / stop or exit, the state of the audio session is the inactive state. The session path refers to the file path of the audio session, such as the application path. The session browser identifier can be used to identify the audio corresponding to the tab page dragged by the user when multiple tabs in the browser play audio.

[0125] In the embodiment of the present application, if multiple audio sessions are running in the first device, the audio session list includes session information of multiple audio sessions. In this case, the monitoring module 410 of the first device can associate the session information in the audio session list with the obtained window process information to determine the correspondence between the dragged window and the audio session. That is, to determine which audio session the dragged window corresponds to.

[0126] For example, as shown in the figure, Figure 5B If the session window identifier of an audio session in the audio session list = the window identifier of the dragged window in the window process information, or the browser identifier of an audio session in the audio session list = the window identifier of the dragged window in the window process information, or the browser identifier of an audio session in the audio session list = the browser identifier of the dragged window in the window process information, or the session path of an audio session in the audio session list = the path of the dragged window in the window process information, it can be determined that the audio session corresponds to the dragged window.

[0127] In this embodiment, after determining the first and second parts of the video playback interface 502, the monitoring module 410 of the first device can send this information to the audio output control module 420. The audio output control module 420 then further determines the expansion mode and expansion ratio for the first device to extend the display to the second device. This will be described in detail below.

[0128] The first part of the video playback interface 502 is used to display on the screen of the first device, and the second part of the video playback interface 502 is used to display on the screen of the second device. For example, for Figure 5B The extended display state shown, the first part of the video playback interface 502 can be Figure 5B The video playback interface 502-1 in the video playback interface 502, the second part of the video playback interface 502 can be Figure 8 The video playback interface in the video is 502-2.

[0129] For example, the first device can calculate the specific size of the video playback interface 502 displayed across devices based on the specific real-time position of the video application window 501 on the first device's display screen, and determine the first part and the second part of the video playback interface 502 based on the calculated size and the configuration parameters of the video playback interface 502. The configuration parameters of the video playback interface 502 may include, but are not limited to, the controls displayed on the video playback interface 502 (icons, text, etc., and the specific display position and / or size of each icon, text, etc.). For specific configuration parameters of the application interface, conventional techniques can be referenced, and this application is not limited thereto.

[0130] like Figure 4 As shown, assuming the width of the video application window 501 is L and the height is H, and the distance between the left edge of the video application window 501 and the right edge of the first device's display screen is x1, then the distance between the right edge of the video application window 501 and the left edge of the second device's display screen can be calculated as x2, where x2 = L - x1. After obtaining the specific dimensions x1 and x2 of the video playback interface 502 displayed across devices, the first device can, in conjunction with the display position and size of each icon, text, etc., on the video playback interface 502, divide the video playback interface 502 into a first part and a second part. The first part includes the interface configuration parameters within the range of x1 × H on the video playback interface 502; the second part includes the interface configuration parameters within the range of x2 × H on the video playback interface 502.

[0131] S604, The first device collects the audio data corresponding to the video playback interface 502.

[0132] In this embodiment of the application, the first device can be via, as follows: Figure 5AThe virtual audio module 430 shown collects audio data corresponding to the video playing interface 502.

[0133] For example, the audio output control module 420 can send a data collection instruction to the virtual audio module 430, where the data collection instruction carries a window identifier corresponding to the video application window 501, an interface identifier corresponding to the video playing interface 502, or an application identifier corresponding to the video application, etc., for the virtual audio module 430 to collect corresponding audio data according to the identifier.

[0134] It should be noted that the examples shown in the present application Figure 5B , Figure 5C and Figure 9 are examples of an audio session on the first device running an application interface (i.e., an audio session corresponding to the video playing interface 502), so in step S604, the first device can only collect audio data corresponding to the video playing interface 502 through the virtual audio module 430. For the case where multiple audio sessions run on the first device, in step S604, the first device needs to collect audio data corresponding to all application interfaces played on the first device through the virtual audio module 430. Assuming that the first device runs audio sessions corresponding to application 1, application 2, …, application N (where N is a positive integer, N≥3), in this case, as shown in Figure 9 , the audio data collected by the virtual audio module 430 of the first device is a mixed stream of audio stream 1 (such as session1), audio stream 2 (such as session2), …, audio stream N (such as sessionN). Among them, audio stream 1 is audio data corresponding to application 1, audio stream 2 is audio data corresponding to application 2, …, and audio stream N is audio data corresponding to application N. As shown in Figure 9 , the virtual audio module 430 can collect the above audio data from the endpoint buffer.

[0135] For example, as shown in Figure 9 , in an embodiment of the present application, the first device can control the virtual audio module 430 to filter out audio data (such as audio stream I) corresponding to the dragged window from the mixed stream and transmit it to the audio output control module 420. For example, the audio output control module 420 can send a filtering instruction carrying session information of the audio session corresponding to the dragged window to the virtual audio module 430 to instruct the virtual audio module 430 to filter out audio data corresponding to the session information from the mixed stream. As shown in Figure 8As shown, the virtual audio module 430 transmits audio stream I to the audio output control module 420 through one signal, and transmits audio streams 1 to (I-1) and audio streams (I+1) to N to the audio output control module 420 through another signal.

[0136] S605, the first device sends first extended data to the second device based on the real-time location of the video application window 501. The first extended data includes first video data and first audio data.

[0137] The first video data is the video data corresponding to the second part of the video playback interface 502, and the first audio data includes the audio stream corresponding to the video playback interface 502.

[0138] Furthermore, in this embodiment of the application, the first audio data in the first extended data may also include the volume (e.g., volume 2) when the second device plays the audio stream corresponding to the video playback interface 502, which is used to instruct the second device to play the second part of the video playback interface 502 at volume 2.

[0139] In some embodiments of this application, the audio output control module 420 of the first device can determine the expansion mode of the first device extending its display to the second device based on the first and second parts of the video playback interface 502 sent by the monitoring module 410. The expansion mode can include horizontal expansion and vertical expansion. Horizontal expansion refers to expansion in a lateral movement manner, i.e., as shown... Figure 9 The expansion method is shown. Vertical expansion refers to expansion by moving vertically.

[0140] It should be noted that if the expansion mode of the first device extending the display to the second device is vertical expansion, then the monitoring module 410 needs to obtain the distance y1 between the top edge of the video application window 501 and the bottom edge of the first device's display screen, and the distance y2 between the bottom edge of the video application window 501 and the top edge of the second device's display screen, where y1 + y2 equals the height H of the video application window 501. This embodiment uses horizontal expansion as an example; for the specific implementation process of vertical expansion, please refer to the specific implementation process of vertical expansion.

[0141] For example, when part of the video playback interface 502 is located on the first device and another part is located on the second device, for example... Figure 9 When x1 > 0 and x2 > 0, the first video data in the first extended data sent by the first device to the second device may specifically include the interface configuration parameters corresponding to the second part of the video playback interface 502; the first audio data in the first extended data may specifically include the audio stream corresponding to the video playback interface 502 (such as...). Figure 9 The audio stream (I) and volume (as shown) are shown. Figure 8The volume 2) is shown. Among them, the first extension data is used for the second device and the first device to display the video playing interface 502 in a cross-device distributed manner by splicing, and synchronously play the audio corresponding to the video playing interface 502.

[0142] Among them, in the embodiment of the application, the first device can send the first extension data to the audio channel of the second device to instruct the second device and the first device to display the video playing interface 502 in a cross-device distributed manner by splicing, and synchronously play the audio corresponding to the video playing interface 502.

[0143] In some embodiments of the application, the above-mentioned volume 2 can be determined by the first device according to the volume (such as volume 1) when the first device currently plays the audio corresponding to the video playing interface 502. For example, volume 2 = volume 1.

[0144] In another embodiment of the application, the first device can also control the volume 1 and the volume 2 when the first device and the second device respectively play the audio corresponding to the video playing interface 502 based on the extension ratio of the first device to the second device for extension display.

[0145] Among them, the audio output control module 420 of the first device can determine the extension ratio of the first device to the second device for extension display according to the first part and the second part of the video playing interface 502 sent by the monitoring module 410. In the embodiment of the application, for the extension mode such as Figure 9 As shown in the left and right extension mode, the extension ratio is x1 / x2. For the up and down extension mode, the extension ratio is y1 / y2.

[0146] As shown in the left and right extension mode, the extension ratio is x1 / x2. For the up and down extension mode, the extension ratio is y1 / y2. Figure 5A As shown in the left and right extension mode, the extension ratio is x1 / x2. For the up and down extension mode, the extension ratio is y1 / y2. Figure 5B As shown in the left and right extension mode, the extension ratio is x1 / x2. For the up and down extension mode, the extension ratio is y1 / y2.

[0147] S606, the first device displays the first part of the video playing interface 502, and the second device displays the second part of the video playing interface 502; the first device and the second device distribute and synchronously play the audio corresponding to the video playing interface 502.

[0148] Among them, when the first device and the second device distribute and synchronously play the audio corresponding to the video playing interface 502, as Figure 9As shown, the first device plays the audio corresponding to the video playback interface 502 at volume 1, and the second device plays the audio corresponding to the video playback interface 502 at volume 2.

[0149] In some embodiments, the audio playback module 440 of the first device and the audio playback module of the second device can synchronously play the audio corresponding to the video playback interface 502 at the same volume. That is, volume 1 = volume 2.

[0150] In other embodiments, such as Figure 5A As shown, when x1 > 0 and x2 > 0, the audio playback module 440 of the first device and the audio playback module of the second device can play the audio corresponding to the video playback interface 502 at volume 1 and volume 2 respectively. Where volume 1 / volume 2 = x1 / x2. For example, volume 1 = volume 0 × x1 / L, volume 2 = volume 0 × x2 / L, where L is the width of the video application window 501, and volume 0 is the width of the video application window 502. Figure 9 The state shown is the initial volume (e.g., volume 0) of the first device playing video on the playback interface 502.

[0151] It should be noted that for situations where multiple audio sessions are running on the first device, such as... Figure 9 The first device shown has audio sessions corresponding to applications 1, 2, ..., N (where N is a positive integer, N≥3) running. The audio playback module 440 of the first device needs to play the audio corresponding to the video playback interface 502 at volume 1, while simultaneously playing the audio corresponding to other audio sessions (i.e., ...). Figure 10 The audio streams shown are audio streams 1 to (I-1) and (I+1) to N. In this case, as... Figure 11 As shown, the audio channel of the second device sets the volume to 2 when playing audio stream I through a player (such as a player), and then plays the audio through a speaker. The audio playback module 440 of the first device sets the volume to 1 when playing audio stream I through a player (such as a player), and then mixes the audio stream I with audio streams 1 to (I-1) and audio streams (I+1) to N with the volume set, and then plays the mixed audio stream through a speaker.

[0152] For the case where an audio session (such as the audio session corresponding to the video playback interface 502) is running on the first device, the audio output control module 420 of the first device determines the audio stream corresponding to the audio session filtered by the virtual audio module 430 (such as...). Figure 11 Following the audio stream I shown, the audio playback module 440 of the first device and the audio playback module of the second device are instructed to play the audio corresponding to the video playback interface 502 at volumes 1 and 2, respectively. In this case, as... Figure 5BAs shown, the audio channel of the second device sets the volume 2 when playing the audio stream I through the player (such as player), and then plays the audio through the speaker. The audio playing module 440 of the first device sets the volume 1 when playing the audio stream I through the player (such as player), and then directly plays the audio through the speaker.

[0153] Further, as the user continuously drags the video application window 501, when the video application window 501 changes from the state shown in Figure 5C to the state shown in Figure 6 , as shown in Figure 5B , the distributed audio playing method provided by the embodiments of the present application can further include the following steps S607-S608.

[0154] S607, the first device sends second extension data to the second device when detecting that the video application window 501 is dragged out of the edge of the display screen of the first device. The second extension data includes second video data and second audio data.

[0155] When the video application window 501 changes from the state shown in Figure 5C to the state shown in Figure 5C , the first device completely transfers the audio corresponding to the video application window 501 and the video playing interface 502 to the second device. The second video data is the video data corresponding to the video playing interface 502, and the second audio data includes the audio stream corresponding to the video playing interface 502.

[0156] Further, in the embodiments of the present application, the second audio data in the second extension data can further include the volume (such as volume 3) when the second device plays the audio stream corresponding to the video playing interface 502, for instructing the second device to play the video playing interface 502 with the volume 2.

[0157] In the embodiments of the present application, the first device can send the second extension data to the audio channel of the second device, to instruct the second device to display the video playing interface 502 and synchronously play the audio corresponding to the video playing interface 502.

[0158] S608, the second device displays the video application window 501, and plays the audio corresponding to the video playing interface 502 in the video application window 501.

[0159] In some embodiments, the second device can play the audio corresponding to the video playing interface 502 with a default volume. For example, the default volume is the current setting volume of the audio playing module of the second device.

[0160] In other embodiments, as shown in Figure 6 and Figure 5AAs shown, the second device can play the audio corresponding to the video playing interface 502 at volume 3. Among them, volume 3 = volume 0; volume 0 is the initial volume of the video playing interface 502 played by the first device at the state shown. ​ As shown, the first device plays the initial volume of the video playing interface 502.

[0161] For example, the audio channel of the second device can set the volume 3 when playing the audio stream I through the player (such as player), and then play the audio through the loudspeaker.

[0162] Through the distributed audio playing method provided by the embodiments of the present application, the first device can realize the effect of automatically adapting the distributed display of the audio with the interface in the process of expanding the display of the interface including the audio to the second device. For example, when the user has not dragged the interface window or has not dragged the interface window out of the range of the display screen of the first device, only the first device plays the audio corresponding to the interface. When the user drags the interface window to a state where a part is displayed in the range of the display screen of the first device and another part is displayed in the range of the display screen of the second device, the audio corresponding to the interface is distributed and synchronously played. When the user drags the interface window to the range of the display screen of the second device, only the second device plays the audio corresponding to the interface. Through the above method, the user's audio setting operation during the expansion of the display can be reduced, and the user experience can be improved.

[0163] Further, in some embodiments, the first device can also control the size of the volume of the audio corresponding to the interface played by the first device and the second device synchronously when the user drags the interface window to a state where a first part is displayed in the range of the display screen of the first device and a second part is displayed in the range of the display screen of the second device. For example, according to the specific proportion of the first part and the second part, the volume of the audio corresponding to the interface played by the first device and the second device synchronously is gradually changed with the movement of the interface window. For example, as the interface window is expanded and displayed from the first device to the second device, the volume of the first device becomes smaller and smaller, and the volume of the second device becomes larger and larger. In this way, the change of the volume can be automatically adapted to the change of the expansion of the interface, and better user experience can be provided.

[0164] In some embodiments of the present application, the channel between the first device and the second device can be one channel for transmitting audio, transmitting video, transmitting control signaling, transmitting parameters, etc., that is, all interactive data are interacted through this channel, and all interactive data need to meet the data format of this channel. The channel between the first device and the second device can also be multiple channels, respectively for transmitting one or more of audio, video, control signaling, and parameters.

[0165] The above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for distributed audio playback, the method comprising: The method comprises: The first device displays a first application window, and the first device plays first audio corresponding to the first application window; In response to a first user operation, the first device displays a first part of the first application window, the second device receives video data sent by the first device, and a second part of the first application window is displayed according to the video data; at this time, the first device plays the first audio at a first volume, and the second device plays the first audio at a second volume, the first volume / the second volume=an expansion ratio; for an expansion mode in which the first device is expanded to the left and right of the second device, the expansion ratio is x1 / x2, the x1 is a distance between a left edge of the first part and a right edge of a display screen of the first device, the x2 is a distance between a right edge of the second part and a left edge of a display screen of the second device, and x1+x2 is equal to a width of the first application window; for an expansion mode in which the first device is expanded to the top and bottom of the second device, the expansion ratio is y1 / y2, the y1 is a distance between an upper edge of the first part and a lower edge of the display screen of the first device, the y2 is a distance between a lower edge of the second part and an upper edge of the display screen of the second device, and y1+y2 is equal to a height of the first application window; In response to a second user operation, the second device receives the video data sent by the first device, and displays the first application window according to the video data; at this time, the second device plays the first audio.

2. The method of claim 1, wherein, The method further comprises: The first device monitors a position of the first application window in a display area corresponding to the first device, and determines whether to send the video data to the second device according to the position of the first application window; When the first application window is in the display area corresponding to the first device, the video data is not sent to the second device; When a first part of the first application window is in the display area corresponding to the first device and a second part of the first application window is in a display area corresponding to the second device, the video data is sent to the second device; When the first application window is in the display area corresponding to the second device, the video data is sent to the second device.

3. The method according to claim 1 or 2, characterized in that, The method further comprises: When an audio state of the first application window is a playing state, the first device sends the first audio to the second device; When the audio state of the first application window is at least one of a pause playing state, a mute state, and a quit playing state, the first audio is not sent to the second device.

4. The method according to claim 1 or 2, characterized in that, The method further comprises: The first device and the second device play the first audio at the same volume.

5. The method according to claim 1 or 2, characterized in that, The method further comprises: The first device acquires an audio stream associated with the first application window from an audio session list according to an identifier of the first application window, and generates the first audio according to the audio stream associated with the first application window.

6. The method of claim 5, wherein, The identifier of the first application window comprises at least one of a window identifier, a browser identifier, and a path.

7. The method according to claim 1 or 2, characterized in that, The first application corresponding to the first application window runs on the first device.

8. The method of claim 1 or 2, wherein, The first user operation and the second user operation are operations of dragging the first application window.

9. The method of claim 1 or 2, wherein, The first device is a notebook computer, and the second device is a tablet computer.

10. An electronic device, comprising: Comprise: One or more processors, one or more memories; wherein the one or more memories are coupled to the one or more processors, and the one or more memories are configured to store computer program codes, and the computer program codes comprise computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the method in any one of claims 1 to 9.

11. A computer storage medium, characterized in that The computer storage medium stores a computer program, and the computer program comprises program instructions, and the program instructions are executed by a processor to execute the method in any one of claims 1 to 9.

Images