An audio data processing method, an audio system and an electronic device

Abstract

The application provides an audio data processing method, an audio system and an electronic device, relates to the technical field of data processing, is applied to a first electronic device, and the first electronic device is also in communication connection with a second electronic device and a third electronic device; the method comprises the following steps: the first electronic device sends first audio data to the second electronic device, so that the second electronic device plays the first audio data, and the first electronic device sends second audio data to the third electronic device, so that the third electronic device plays the second audio data; wherein the characteristic information corresponding to the first audio data is different from the characteristic information corresponding to the second audio data; the characteristic information at least comprises at least one of a transmission path, a source application and a flow type in the first electronic device. The scheme can realize the simultaneous playing of multiple audio in different devices, and improves the audio experience of the user.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to an audio data processing method, an audio system, and an electronic device. Background Technology

[0002] With the development of electronic technology and mobile internet, a user can simultaneously own multiple electronic devices such as mobile phones, tablets, personal computers (PCs), and smart home devices (such as smart TVs).

[0003] Multiple electronic devices can perform various cross-device services. For example, a user may own a PC, tablet, and mobile phone. The phone and PC can perform screen mirroring, while the phone and tablet can perform a super call service. However, if a user is watching a video on their PC by mirroring it from their phone, and simultaneously answers a call on their phone while on their tablet, the video and audio on the PC will stop, leaving only the call audio on the tablet, negatively impacting the user's audio experience. Summary of the Invention

[0004] This application provides an audio data processing method, an audio system, and an electronic device, which can enable multiple audio streams to be played simultaneously on different devices, thereby improving the user's audio experience.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] In a first aspect, an audio data processing method is provided. This method can be applied to a first electronic device, which is also communicatively connected to a second electronic device and a third electronic device. The method includes: simultaneously sending first audio data from the first electronic device to the second electronic device to cause the second electronic device to play the first audio data, and simultaneously sending second audio data from the first electronic device to the third electronic device to cause the third electronic device to play the second audio data. The feature information corresponding to the first audio data is different from the feature information corresponding to the second audio data; the feature information includes at least one of the following: source application, stream type, and transmission path within the first electronic device.

[0007] By adopting this technical solution, this solution can enable the simultaneous playback of audio data with different feature information from the first electronic device on the second electronic device and the third electronic device, thereby improving the user's audio experience.

[0008] When the feature information is the source application, this solution can enable the playback of audio data from different source applications on different electronic devices. When the feature information is the stream type, this solution can enable the playback of audio data of different stream types on different electronic devices. When the feature information is the transmission path within the first electronic device, this solution can enable the playback of audio data with different transmission paths within the first electronic device on different electronic devices. Different transmission paths within the first electronic device can mean that the audio data passes through different modules within the first electronic device; for example, the audio data transmission path may pass through an audio frame (audio source is the audio frame) or the audio data transmission path may pass through a call audio module (audio source is the call audio module).

[0009] In one possible implementation of the first aspect, before the first electronic device sends the first audio data to the second electronic device, the method further includes: the first electronic device establishing a first mapping relationship between the first audio data and the second electronic device based on the feature information corresponding to the first audio data. Specifically, the feature information corresponding to the first audio data can be set as first feature information, and the first electronic device establishes the first mapping relationship between the first audio data and the second electronic device based on the first feature information. That is to say, the first electronic device sets an audio strategy at this time, specifying the relationship between audio data with different feature information and different electronic devices. Or, the first electronic device sets a priority device, and can specify one or more priority devices as the devices to which the acquired audio data is to be sent.

[0010] In one possible implementation of the first aspect, the first electronic device sending first audio data to the second electronic device includes: the first electronic device determining the first audio data based on feature information of the acquired audio data; and the first electronic device sending the first audio data to the second electronic device based on a first mapping relationship. Specifically, the first electronic device may determine the audio data corresponding to the first feature information as the first audio data based on the feature information of the acquired audio data. Then, the first electronic device sends the first audio data corresponding to the first feature information to the second electronic device based on the first mapping relationship.

[0011] In one possible implementation of the first aspect, before the first electronic device sends the second audio data to the third electronic device, the method further includes: the first electronic device establishing a second mapping relationship between the second audio data and the third electronic device based on the feature information corresponding to the second audio data. Specifically, the feature information corresponding to the second audio data can be set as second feature information, and the first electronic device establishes the second mapping relationship between the second audio data and the third electronic device based on the second feature information. That is to say, the first electronic device sets an audio strategy at this time, specifying the relationship between audio data with different feature information and different electronic devices. Or, the first electronic device sets a priority device, and can specify one or more priority devices as the devices to which the acquired audio data is to be sent.

[0012] In one possible implementation of the first aspect, the first electronic device sends second audio data to the third electronic device, including: the first electronic device determining the second audio data based on feature information of the acquired audio data; and the first electronic device sending the second audio data to the third electronic device based on a second mapping relationship. Specifically, the first electronic device may determine the audio data corresponding to the second feature information as the second audio data based on the feature information of the acquired audio data. Then, the first electronic device sends the second audio data corresponding to the second feature information to the third electronic device based on the second mapping relationship.

[0013] In one possible implementation of the first aspect, before the first electronic device sends the first audio data to the second electronic device, the method further includes: the first electronic device creating a first data channel, the first data channel corresponding to a first channel identifier; and the first electronic device establishing a first association between the first channel identifier and a first device identifier of the second electronic device. In this solution, different data channels can be distinguished by the device identifier, where the data channel is also known as a data session.

[0014] In one possible implementation of the first aspect, the first electronic device sending first audio data to the second electronic device includes: the first electronic device sending the first audio data to the second electronic device through a first data channel based on a first mapping relationship and a first association relationship. In this solution, the first electronic device is also configured to send audio data with different feature information to different electronic devices through different data channels.

[0015] In one possible implementation of the first aspect, before the first electronic device sends the second audio data to the third electronic device, the method further includes: the first electronic device creating a second data channel, the second data channel corresponding to a second channel identifier; and the first electronic device establishing a second association between the second channel identifier and the second device identifier of the third electronic device.

[0016] In one possible implementation of the first aspect, the first electronic device sends second audio data to the third electronic device, including: the first electronic device sending the second audio data to the third electronic device through a second data channel based on a second mapping relationship and a second association relationship. In this solution, the first electronic device is configured to send audio data with different feature information to different electronic devices through different data channels.

[0017] In one possible implementation of the first aspect, the first electronic device may further set a mapping relationship between audio data and data channels. Through the mapping relationship between audio data and data channels, as well as the association relationship between data channels and electronic devices, audio data with different feature information can be sent to different electronic devices through different data channels.

[0018] In one possible implementation of the first aspect, the feature information includes at least one of a source application and a stream type. Before the first electronic device sends first audio data to a second electronic device to cause the second electronic device to play the first audio data, and simultaneously sends second audio data to a third electronic device to cause the third electronic device to play the second audio data, the method further includes: the first electronic device performing stream splitting processing on the audio data acquired by the first electronic device based on the feature information to determine the first audio data and the second audio data. Specifically, the first electronic device may perform stream splitting processing on the acquired audio data based on the source application to determine the first audio data corresponding to a first source application and the second audio data corresponding to a second source application. Alternatively, the first electronic device may also perform stream splitting processing on the acquired audio data based on the stream type to determine the first audio data corresponding to a first stream type and the second audio data corresponding to a second stream type.

[0019] In one possible implementation of the first aspect, the first electronic device includes a hardware abstraction layer and an audio framework; the first electronic device performs split processing on the audio data acquired by the first electronic device based on feature information to determine the first audio data and the second audio data, including: the first electronic device can perform split processing on the audio data acquired by the first electronic device within the audio framework based on feature information to determine the first audio data and the second audio data. Alternatively, the first electronic device does not perform split processing on the acquired audio data within the audio framework, but can instead perform split processing on the audio data acquired by the first electronic device within the hardware abstraction layer based on feature information to determine the first audio data and the second audio data.

[0020] In one possible implementation of the first aspect, while the first electronic device sends first audio data to the second electronic device to cause the second electronic device to play the first audio data, the first electronic device also sends second audio data to the third electronic device to cause the third electronic device to play the second audio data. This includes: while the first electronic device sends the first audio data to the second electronic device to cause the second electronic device to play the first audio data, in response to a user's call answering operation on the third electronic device, the first electronic device acquires the call audio as the second audio data; the first electronic device sends the second audio data to the third electronic device to cause the third electronic device to play the second audio data; wherein the transmission path of the first audio data within the first electronic device is different from the transmission path of the second audio data within the first electronic device.

[0021] In one possible implementation of the first aspect, the first electronic device includes an audio call module and an audio frame; the transmission path of the first audio data passes through the audio frame, and the transmission path of the second audio data passes through the audio call module.

[0022] Secondly, embodiments of this application provide an audio system, which includes a first electronic device, a second electronic device, and a third electronic device, wherein the first electronic device is communicatively connected to both the second and third electronic devices; the system includes: the first electronic device sending first audio data to the second electronic device; the second electronic device receiving the first audio data sent by the first electronic device and playing the first audio data; while the second electronic device is playing the first audio data, the first electronic device sending second audio data to the third electronic device; and the third electronic device receiving the second audio data sent by the first electronic device and playing the second audio data; wherein the feature information corresponding to the first audio data is different from the feature information corresponding to the second audio data; the feature information includes at least one of the following: source application, stream type, and transmission path within the first electronic device.

[0023] By adopting the above technical solution, in the above audio system, the first electronic device can simultaneously play audio data from the first electronic device on the second electronic device and the second electronic device, thereby improving the audio viewing experience.

[0024] Thirdly, this application provides an electronic device comprising: an audio module, a communication module, a memory, and one or more processors; the audio module, the communication module, the memory, and the processors are coupled; wherein the memory stores computer program code, the computer program code including computer instructions, which, when executed by the processor, cause the electronic device to perform the audio data processing method described in any one of the first aspects.

[0025] Fourthly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, enable the computer to perform the audio data processing method described in any one of the first aspects.

[0026] Fifthly, this application provides a computer program product containing instructions that, when run on a computer, enables the computer to perform the audio data processing method described in any one of the first aspects.

[0027] It is understood that the electronic device described in the third aspect, the computer-readable storage medium described in the fourth aspect, and the computer program product described in the fifth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here. Attached Figure Description

[0028] Figure 1 This application provides a schematic diagram illustrating a trust ring service.

[0029] Figure 2 A software architecture diagram for interaction between a mobile terminal and a device terminal provided in an embodiment of this application;

[0030] Figure 3 This is a business process diagram for a scenario where mobile phones and tablets can play videos collaboratively, and PCs can answer super calls, as provided in the current solution.

[0031] Figure 4 This is a sequence diagram of the business modules for a scenario where mobile phones and tablets can play videos collaboratively and PCs can answer super calls, as provided in the current solution.

[0032] Figure 5 This application provides a business process diagram for a scenario where a mobile phone and a tablet computer collaboratively play videos, and a PC answers a super call.

[0033] Figure 6 This application provides a timing diagram of the service modules for a scenario where a mobile phone and a tablet computer collaboratively play videos, and a PC answers a super call.

[0034] Figure 7 This is the original module diagram for a multi-type media audio concurrent scenario provided in the current solution;

[0035] Figure 8 A diagram of a streaming module for a multi-type media audio concurrent scenario provided in this application embodiment;

[0036] Figure 9A diagram of a splitting module for another type of concurrent media audio scenario provided in this application embodiment;

[0037] Figure 10 A business process diagram for concurrent processing of multiple types of media audio provided in this application embodiment;

[0038] Figure 11 A timing diagram of a service module for concurrent processing of multiple types of media audio provided in an embodiment of this application;

[0039] Figure 12 A diagram of a streaming module for a multi-application audio data concurrency scenario is provided in an embodiment of this application;

[0040] Figure 13 A business process diagram for multi-application concurrent audio processing is provided in an embodiment of this application;

[0041] Figure 14 A timing diagram of a business module for multi-application audio concurrent processing provided in an embodiment of this application;

[0042] Figure 15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0043] Figure 16 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort should fall within the scope of protection of this application.

[0045] It should be noted that the terms "first," "second," etc., used below are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0046] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0047] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0048] To better illustrate the technical solution of this application, the terminology involved in this application is briefly summarized below.

[0049] (1) Trust ring: The same user account can log in on different electronic devices. When two or more electronic devices under the same user account trust each other, then the two or more electronic devices constitute a trust ring. For an electronic device in the trust ring (such as electronic device 1), the other electronic devices in the trust ring (such as electronic device 2) can all be understood as trusted devices of electronic device 1.

[0050] (2) Collaborative Services: Collaborative services are cross-device services conducted within the same local area network. Collaborative service scenarios can be mainly applied to office, entertainment, and other similar scenarios. For example, users can use collaborative services to project the screen and audio of their mobile phones onto a tablet / PC to watch videos on the tablet / PC.

[0051] (3) Super Call Service: The main scenario for the Super Call Service is that when there is an incoming call on the mobile phone, the Super Call Service will synchronize with trusted devices within the same trust ring. The trusted devices (such as tablets / PCs) will receive an incoming call banner notification (such as a screen pop-up). Users can answer the call directly on the tablet / PC, transferring the call from the mobile phone to the tablet / PC for answering.

[0052] (4) Media applications: Specific audio or video applications that users can use.

[0053] (5) Distributed Mobile Sensing Development Platform (DMSDP): This virtualization module primarily provides hardware virtualization capabilities for services, allowing the source device to utilize the hardware capabilities of the sink device across devices. For example, a mobile phone can utilize the hardware capabilities of a device (tablet / PC) across devices.

[0054] In this embodiment, the DMSDP module can create a control channel and negotiate audio parameters based on the control channel. After the DMSDP module obtains the audio parameters, the DMSDP module successfully enables the service. The audio parameters may include sampling rate, bit width, number of channels, etc.

[0055] (6) Hardware Abstraction Layer (HAL): The HAL defines abstract device objects corresponding to specific microphone / speaker hardware devices, used to shield the application from the underlying device differences. The HAL primarily handles audio chip driver control. The virtual audio processing object is a Pulse Code Modulation (PCM) audio stream.

[0056] The HAL layer mainly includes two modules: the Audio policy module (which adds working policies for the virtual Mic / Speaker) and the Audio device layer (which registers the virtual Mic / Speaker device and completes the proxy for the near-end audio device).

[0057] The ultimate goal of DMSDP-Audio virtualization is to enable interconnection between audio (Mic / Speaker) devices and mobile phones, allowing mobile applications to use virtual audio (Mic / Speaker) devices just like using the phone's local Mic / Speaker.

[0058] (7) Hardware Abstraction Layer Interface Definition Language (HAL, Hidl) is used to define the interface between the application framework layer and the hardware abstraction layer implementation. Hidl is used for inter-process communication and specifies the interface between the HAL and its users.

[0059] (8) Universally Unique Identifier (UUID): This is a unique identifier for Android devices, used to identify the uniqueness of different devices or applications. In addition to representing the unique identifier of hardware devices, UUIDs can also be used to identify software.

[0060] In this embodiment of the application, when creating a data session, the data session can be associated with the device's UUID so that different data sessions can be distinguished by the device's UUID.

[0061] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating a trust ring service provided in an embodiment of this application. A single trust ring can include multiple electronic devices, such as mobile phones, tablets, and PCs. Figure 1 As shown, mobile phones, tablets, and PCs can perform various cross-device services. Mobile phones and tablets can perform collaborative services, cross-platform screen mirroring, and super call services. Mobile phones and PCs can also perform collaborative services, cross-platform screen mirroring, and super call services. PCs and tablets can follow each other, meaning that tablets can be used as extended screens for PCs.

[0062] In this embodiment, while the mobile phone and tablet are engaged in collaborative services, the mobile phone and PC can also perform a "Super Call" service. Specifically, when the mobile phone and tablet are playing a video together, the audio from the mobile phone can be switched to the tablet. When a call comes in on the mobile phone, both the PC and tablet will display a call banner notification. If the user chooses to answer the call on the PC, the video playback on the tablet will stop playing audio, and only the PC will have audio from the call. Therefore, the mobile phone, tablet, and PC cannot meet the user's requirement of simultaneously watching videos and answering calls, impacting the user's cross-device service experience.

[0063] Therefore, resolving audio conflicts between mobile phones and tablets, and between mobile phones and PCs, is an urgent problem to be solved.

[0064] It should be noted that if the user answers a call on the phone, it will not affect the video playback on the tablet, and the audio from the tablet will still play normally.

[0065] This application provides an audio data processing method that sends audio data with different feature information to different devices, enabling different devices to play audio data with different feature information. This solution can achieve dual-channel audio stream playback, optimizing the audio streaming experience. Here, the audio data with different feature information can refer to audio data from different audio sources, or audio data from the same audio source but different audio stream types, or audio data from the same audio source and audio stream type but with different source applications.

[0066] In this embodiment, a mobile phone is used as an example where it is joined to the same trust ring as at least two other electronic devices. These at least two other electronic devices may include tablets, PCs, smart screens, in-vehicle systems, etc.

[0067] In this application, the examples of multiple services between mobile phones, PCs, and tablets are used for illustration.

[0068] Please see Figure 2 , Figure 2 This application provides a software architecture diagram for interaction between a mobile phone and a device. The software system can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses a layered Android system as an example to exemplify the software structure of the mobile phone and device.

[0069] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, such as... Figure 2 As shown, both mobile and device-side applications can be divided into the application layer, application framework layer, and hardware abstraction layer (HAL) from top to bottom. Device-side applications can include tablets and PCs, as mentioned above.

[0070] The mobile application layer can include a series of application packages. These application packages can include various media applications, calling applications, such as camera, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video, and SMS applications. Figure 2 The applications are distinguished as Application 1, Application 2, Application 3, Application 4, and Application 5. The applications also include settings.

[0071] The mobile application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0072] like Figure 2As shown, the application framework layer may include an audio subsystem and a DMSDP service module.

[0073] The audio submodule may include a media player, a media recorder, an audio track, an audio recorder, an audio policy service, and an audio manager.

[0074] The DMSDP service module is used to start services on demand and may include a device management module, an audio processing module, and a transmission protocol module.

[0075] The Hardware Abstraction Layer (HAL) runs in user space, encapsulates kernel-level drivers, and provides calling interfaces to higher layers. For example... Figure 2 As shown, the hardware abstraction layer can include an audio hidl service and a transport module. The audio hidl service includes a local audio device and a virtual audio device (registered at boot). The virtual audio device is used to load the shared object file (SO) of the virtual audio and to start the virtual audio hidl service. The SO file is a binary file used in Linux and Unix systems. SO files are also known as shared libraries or dynamic link libraries.

[0076] The mobile phone's transmission module is used to establish a connection with the device's transmission module.

[0077] The device-side application framework layer includes the DMSDP-sdk module, the audio subsystem module, and the audio module.

[0078] The hardware abstraction layer on the device side includes the transmission module and the audio device driver / integrated circuit chip (IC).

[0079] In this embodiment, the mobile phone responds to the user's operation by selecting a virtual audio device (1.1). The user's operation may include cross-device operations initiated by the user, such as screen mirroring or collaborative work. After determining the virtual audio device to be used, the mobile phone enables the virtual audio device. Specifically, the application layer sends an enable command to the device management module in the DMSDP service module of the application framework layer to enable the virtual audio device (1.2). Then, the device management module sends a message to the transmission protocol module to initialize the transmission protocol (1.3.1). After successfully enabling the virtual audio device, the transmission protocol module sends a message to the virtual audio device in the hardware abstraction layer to register the transmission interface (1.3.2).

[0080] Then, the device management module synchronizes the virtual audio activation status to the audio policy module (1.4).

[0081] At this point, the mobile device responds to the user's operation and triggers the audio service (2.1). The selected application in the application layer sends a message to the audio subsystem of the application framework layer to start microphone pickup and speaker playback (2.2).

[0082] The audio policy service module switches to the virtual audio device based on the configured audio policy and conditions (2.3). Then, AudioTrack plays (2.4), and AudioRecord picks up sound (2.4).

[0083] Then, the audio subsystem sends audio data packets to the virtual audio device in the audio Hidl service in the hardware abstraction layer. The virtual audio device selects which virtual audio device to operate based on the audio policy (2.5).

[0084] In some embodiments, if it is determined that the device is switching to a local audio device, then after the phone injects the audio data into the hardware abstraction layer, the audio is played by the local speaker.

[0085] After the mobile phone determines that it has switched to the virtual audio device, the virtual audio device starts the virtual audio Hidl service and sends a message (2.6) to the transport protocol module, sending the audio data packet to the audio processing module in the DMSDP service module. The audio processing module can resample and encode / decode the audio data packet.

[0086] The audio processing module encodes the audio data packets and then sends the encoded audio data packets to the transmission module via the transmission protocol module (2.7). The transmission module on the mobile phone sends the encoded audio data packets to the transmission module on the device side via P2P transmission (2.8).

[0087] The device's transmission module sends encoded audio data packets (2.9) to the audio processing and audio transmission modules of the DMSDP-sdk module. The audio processing and audio transmission modules then decode the encoded audio data packets to obtain the audio data packets themselves.

[0088] Then, the Java layer's Audio sends audio data packets to the audio track of the audio subsystem via the Android standard interface (2.10). The C layer's Audio sends audio data packets to the audio module via the C interface (2.10). Finally, the audio device driver / IC plays the audio after receiving the audio data packets from the audio module or audio subsystem.

[0089] In this embodiment, audio data can be classified based on its characteristic information. This characteristic information includes at least one of the following: stream type (audio stream type), source application, and transmission path within the phone. Different transmission paths within the phone indicate different audio sources.

[0090] Specifically, audio data can be categorized based on its audio source. For example, the audio source for call tones is an audio digital signal processor (ADSP) (also known as a call audio module), while the audio source for media audio is an audio frame. Call tones and media audio have different audio sources. Therefore, audio data can be divided into media audio and call tones. When the audio source is the same, audio data can be categorized based on its audio type. For example, audio data can be divided into music-type media audio and other types of media audio. When the audio source and audio stream type are the same, audio data can be categorized based on the application from which it originates. For example, audio data can be divided into audio from social application 1 and audio from social application 2.

[0091] In this embodiment, for audio data from different sources, the mobile phone can use a preset audio strategy to send audio data from different sources to different devices via different data channels. For example, the mobile phone sends media audio to device 1 via data channel 1, and sends call audio to device 2 via data channel 2. Therefore, this embodiment allows media audio and call audio to be played on different devices.

[0092] For audio data from the same source but with different audio stream types, the mobile phone can use a preset audio strategy to split the audio data according to the audio stream type and send the audio data of different audio stream types to different devices through different data channels. For example, the mobile phone splits the audio data into music audio and ringtone audio, sends the music audio to device 1 through data channel 3, and sends the ringtone audio to device 2 through data channel 4. Thus, the embodiments of this application can realize the playback of different types of audio on different devices.

[0093] For audio data with the same source and audio stream type but different source applications, the mobile phone can split the audio data according to the application package name, sending the audio data of different applications to different devices through different data channels. For example, the mobile phone splits the audio data by application package name to obtain audio for application 1 and audio for application 2. The audio for application 1 is sent to device 1 through data channel 5, and the audio for application 2 is sent to device 2 through data channel 6. Thus, the embodiments of this application can realize the playback of audio from different applications on different devices.

[0094] Therefore, this application embodiment can be used to specifically illustrate the comparison between the current solution and the audio data processing method provided in this application embodiment from three scenarios. The first scenario can be a concurrent media audio and call audio scenario, the second scenario can be a concurrent multi-type media audio scenario, and the third scenario can be a concurrent multi-application audio scenario.

[0095] First, this application's embodiments introduce the first scenario: concurrent media audio and call audio. Here, "call" refers to a carrier call. Specifically, the first scenario involves a mobile phone and a tablet computer collaboratively playing a video, while the mobile phone and PC are using the Super Call service; during the video playback, the PC answers an incoming call from the mobile phone. Furthermore, after the mobile phone and tablet computer are connected, the mobile phone can project its content onto the tablet computer.

[0096] In the above scenarios, please refer to Figure 3 , Figure 3 This is a business process diagram for a scenario where mobile phones and tablets collaboratively play videos, while a PC answers incoming calls, as provided in the current solution. (Example:) Figure 3 As shown, steps S301-S310 are as follows:

[0097] S301: Collaborative Business Enablement Device A.

[0098] In this embodiment, a tablet computer is used as device A, and a PC is used as device B. After the mobile phone and device A are connected in a collaborative manner, the collaborative service (android.uid.system:1000) enables the audio capabilities of device A (tablet computer).

[0099] Specifically, the collaborative business call enable interface enables virtual services (enableVirtualService), carrying parameters including (dmsdpDevice, capability, Map).<String,Object> info).

[0100] Here, `dmsdpDevice` refers to the enabled device, and the device identifier (DeviceId) corresponding to the enabled device can be passed in the parameters. `capability` refers to the service capabilities, which can include audio, modem, and camera. Map<String,Object> `info` represents information about a collection of key-value pairs. (Map)<String,Object> The keys are of type String, and the values ​​can be of any object type.

[0101] In this scenario, before enabling the collaborative service on device A, the user can open a media application on the Source side, which means the user starts playing music through the speaker.

[0102] S302: The mobile phone and device A negotiate audio parameters.

[0103] In this embodiment, the mobile phone can negotiate audio parameters with device A through control channel 1 created by the DMSDP module. Once the mobile phone successfully negotiates the audio parameters with device A, the collaborative service is successfully enabled on device A.

[0104] S303: Register virtual device A.

[0105] In this embodiment, the mobile phone registers a virtual device A at the hardware abstraction layer. The mobile phone is the Source side, and device A is the Sink side.

[0106] In this process, the mobile phone registers a virtual device at the hardware abstraction layer, and after successful registration, the device status can be refreshed. At this point, the phone's audio framework can detect that the virtual device has been successfully registered.

[0107] S304: Switch audio to virtual device A.

[0108] In this embodiment, the mobile phone's audio framework can switch the audio to virtual device A according to the default audio policy.

[0109] The default audio strategy is as follows: virtual devices are prioritized, and only one device is processed at a time; devices registered later are prioritized. "Virtual devices prioritized" means that after audio is enabled, the audio framework will switch to prioritizing virtual devices. "Only one device processed" means that the audio framework can only process audio data from one device at a time. "Device registered later prioritized" means that after audio is enabled, the audio framework will prioritize the device registered later on the phone.

[0110] S305: Sound is emitted from device A.

[0111] In this system, the phone's audio framework switches the audio to virtual device A, meaning the sound will be emitted from device A.

[0112] S306: Super Call Service Enabler B.

[0113] In this embodiment, if a user is watching a video on a tablet while simultaneously processing tasks on device B (PC), a call banner notification will be displayed on device B (PC) when the user's phone receives a call. A call banner notification will also be displayed on device A (tablet).

[0114] When a user's mobile phone receives a call, the Super Call service enables the modem capabilities of device B (PC).

[0115] S307: The mobile phone and device B negotiate audio parameters.

[0116] In this embodiment, the mobile phone can negotiate audio parameters with device B through control channel 2 created by the DMSDP module. Once the mobile phone successfully negotiates the audio parameters with device B, the Super Call service is successfully enabled on device B.

[0117] S308: Register virtual device B.

[0118] In this embodiment of the application, the mobile phone registers virtual device B at the hardware abstraction layer.

[0119] S309: Switch audio to virtual device B.

[0120] In this embodiment of the application, if the PC receives the user's click to answer the call, that is, the user switches device B to the device playing the call audio.

[0121] At this point, since virtual device B is a virtual device registered later in the phone, the phone's audio framework will switch the audio to virtual device B according to the default audio policy.

[0122] S310: Sound is emitted from device B.

[0123] Since the phone's audio framework switches the audio to virtual device B, virtual device B will have priority in outputting sound.

[0124] Because the collaborative service does not enable the audio IO capability of device B, device B lacks audio IO capability, meaning virtual device B cannot process HAL layer audio data packets. Therefore, device B (PC) only has one audio call. Furthermore, since the audio framework switches the audio from virtual device A to virtual device B, the tablet computer no longer emits sound. It should be noted that the video playback on the tablet computer continues, but there is no sound from the tablet computer.

[0125] Media applications can include music software, video software, social software, etc.

[0126] Therefore, in the above-mentioned scenario of concurrent media audio and call audio, if the user chooses to answer the call on the PC side, it cannot meet the user's demand to simultaneously watch video (video with sound) and answer the call on the PC side, thus affecting the user experience.

[0127] After introducing the business process diagrams corresponding to the concurrent media audio and call audio scenarios, we will continue to introduce the business module sequence diagrams corresponding to the concurrent media audio and call audio scenarios.

[0128] Please see Figure 4 , Figure 4 This is a sequence diagram of the business modules for a scenario where mobile phones and tablets can collaboratively play videos, while a PC can answer incoming calls, as provided in the current solution. Figure 4 As shown, the mobile phone acts as the source side and can include collaborative / call services, media applications, DMSDP device service, DMSDP virtual audio, audio framework AudioTrack, Audio HAL, and ADSP. The mobile phone connects collaboratively with device A and device B. Device A can include service management (DvService), virtual audio, and audio framework AudioTrack. Device B can also include service management (DvService), virtual audio, and audio framework AudioTrack. DeviceService, often abbreviated as DvService, is a device management module used to manage virtual device services and record which devices are using which services.

[0129] AudioTrack is a class that manages and plays audio resources.

[0130] First, after the user opens media application 1 (S401), the media application sends a message to the audio framework AudioTrack to start the speaker to play sound. Then, service A enables device A. Specifically, service A enables (S402) device A, which means enabling device A's virtual service (audio capabilities). The phone's collaboration / call service sends an enable message to DMSDP DvService. Specifically, the collaboration service calls the enable interface to enable the virtual service (enableVirtualService), carrying parameters including (dmsdpDevice, capability, Map).<String,Object> (info). Among them, the mobile phone responds to the user's initiation of collaborative services and selection of device A, and the mobile phone's collaborative services enable device A.

[0131] After receiving the enable message, the DMSDP module can create a control channel and negotiate audio parameters based on the control channel (S403). The mobile phone can designate this control channel as session 1. The DMSDP module negotiates audio parameters with the service management DvService of device A based on session 1. Specifically, the DMSDP module can call the opensession method, passing in the session identifier (sessionid) and device identifier (deviceid). Here, deviceid can refer to the device's UUID.

[0132] Then, the service management DvService of device A returns device information to the DMSDP DvService of the mobile phone (S404). At this time, the DMSDP module is successfully enabled (S405). The DMSDP DvService sends a registration message to the audio HAL of the mobile phone to register virtual device A (S406). Here, virtual device A is equivalent to virtual audio device A.

[0133] Virtual device A, registered in the phone's Audio HAL, sends a synchronization message to the phone's audio framework AudioTrack to synchronize the state of virtual device A (S407).

[0134] At this point, once the virtual device state is synchronized, the virtual device registration is successful. The audio framework AudioTrack switches the audio to virtual device A (S408) according to the default audio policy (AUDIO_MODE_NORMAL). This default audio policy can be the phone's factory settings, such as prioritizing virtual devices, processing only one device at a time, or prioritizing later-registered devices.

[0135] The audio framework AudioTrack sends audio data packets to virtual device A in the Audio HAL via thread 1 (S409). Virtual device A then sends audio data packets to the DMSDP virtual audio module. The DMSDP virtual audio module then encodes the audio data packets (S410) and sends the encoded audio data packets back to device A (S411). Specifically, the DMSDP virtual audio module sends the encoded audio data packets to the virtual audio module of device A.

[0136] Device A's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device A's audio framework, AudioTrack. Upon receiving the audio data packets, AudioTrack plays the audio.

[0137] At this time, if a call comes in on the mobile phone, Service B (Super Call Service) enables Device B (S412). Specifically, the mobile phone's Super Call Service (Service B) sends an enable message to the DMSDP DvService. Upon receiving the enable message, the DMSDP module can create a control channel (set as session2) and negotiate audio parameters based on this control channel (S413). Specifically, the DMSDP module negotiates audio parameters with Device B's service management DvService based on session1. Specifically, the DMSDP module can call the `opensession` method, passing in the session identifier (sessionid) and device identifier (deviceid).

[0138] Then, the service management DvService of device B returns device information to the DMSDP DvService of the mobile phone (S414). At this time, the DMSDP module is successfully enabled (S415). The DMSDP DvService sends a registration message to the audio HAL of the mobile phone to register virtual device B (S416).

[0139] Virtual device B, registered in the phone's Audio HAL, sends a synchronization message to the phone's AudioTrack audio framework to synchronize the virtual device's state (S417).

[0140] At this point, when the mobile phone receives the user's device switching operation—that is, when the mobile phone receives the user's operation to answer the call on device B—the Super Call service sends a switching message (switch ToRomoteDevice) to the DMSDP DvService. The DMSDP DvService then sends a message to the audio framework AudioTrack to switch to device B. The audio framework AudioTrack switches the audio to virtual device B according to the default audio policy (S418).

[0141] The audio framework AudioTrack sends audio data packets to virtual device B in the Audio HAL via thread 1 (S419). However, since device B lacks audio capabilities, the HAL layer does not process the audio data packets. Therefore, virtual device B cannot forward the received audio data packets to device B for playback.

[0142] At this point, the audio framework AudioTrack can read downlink data packets from the ADSP module (S420). The DMSDP virtual audio can obtain downlink data packets from the audio framework AudioTrack, encode the downlink data packets, and then send the encoded audio data packets (downlink data packets) to the virtual audio module of device B (S421). Downlink data refers to the audio data transmitted from the network to the device's speaker or earpiece.

[0143] Device B's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device B's audio framework, AudioTrack. Upon receiving the audio data packets, AudioTrack plays the audio.

[0144] At this point, because the phone's audio framework, AudioTrack, switches the audio from virtual device A to virtual device B according to the default audio policy, device A corresponding to virtual device A no longer emits sound. Furthermore, since device B lacks audio capabilities and the Hall layer does not process audio data packets, virtual device B cannot forward the received audio data packets to device B for playback. Therefore, the user can only answer calls on device B and cannot hear the sound of video playback on device A.

[0145] Therefore, as Figure 4 As shown in the timing diagram, if the above method is used, it cannot meet the user's demand to watch videos and answer phone calls simultaneously in the scenario of concurrent media audio and call audio, thus affecting the user experience.

[0146] To address the aforementioned issues, embodiments of this application provide an audio data processing method. By sending audio data with different characteristic information to different devices, media audio and call audio can be played on different devices, satisfying users' needs to simultaneously watch videos and answer phone calls. Here, the different characteristic information refers to different audio sources, that is, different transmission paths within the mobile phone.

[0147] In the embodiments of this application, please refer to Figure 5 , Figure 5 This application provides a business process diagram for a scenario where a mobile phone and a tablet computer collaboratively play videos, while a PC answers incoming calls. (See attached diagram.) Figure 5 As shown, steps S501-S514 are as follows:

[0148] S501: Collaborative Business Enablement Device A.

[0149] S502: The mobile phone and device A negotiate audio parameters.

[0150] S503: Register virtual device A.

[0151] Steps S501-S503 can be referred to as steps S301-S303 above, and will not be repeated here.

[0152] S504: Set a specific audio strategy.

[0153] In this embodiment, the DMSDP module of the mobile phone sets a specific audio strategy in the audio framework. This specific audio strategy may include specifying which devices a certain type of audio stream should be played on first. For example, specifying that media audio should be played on a tablet computer first, and specifying that call audio should be played on a PC first. In subsequent embodiments, the specific audio strategy will be illustrated by taking StreamType-Music (media audio) as the preferred playback device (tablet computer) and StreamType-Phone (call audio) as the preferred playback device (PC).

[0154] S505: Switch audio to virtual device A.

[0155] In this embodiment, the mobile phone's audio framework can determine whether the received audio data is media audio or call audio based on the set audio strategy (StreamType-Music is prioritized on device A). Assuming the received audio data belongs to StreamType-Music, the audio framework will switch the audio to virtual device A.

[0156] S506: Device Identifier 1 is associated with Session Identifier 1.

[0157] In this embodiment, the DMSDP module of the mobile phone associates the device identifier (Devices id) of device A with the session identifier (session id) of the created data session 1, that is, Devices1 is associated with session1. The association between the device identifier of device A and the session identifier of data session 1 indicates that the mobile phone sends audio data packets to device A through data session 1.

[0158] In some optional embodiments, when setting a specific audio strategy, the mobile phone may include: specifying that a certain type of audio stream is preferentially transmitted through a created data channel. That is, the mobile phone sets a mapping relationship between audio data and data channels. For example, it may specify that media audio is preferentially transmitted through created data session 1. Then, the mobile phone can establish an association relationship between the data channel and the device side. Specifically, the mobile phone can establish an association relationship between the data session and each electronic device based on the session identifier of the data session. For example, the device identifier of device A is associated with the session identifier of data session 1. Thus, the mobile phone can also send audio data packets to device A through data session 1.

[0159] S507: A specific audio sound is emitted from device A.

[0160] The phone's DMSDP module receives the Music audio data packet from the hardware abstraction layer and sends the audio data packet from the phone (Source) side to the device A side based on the created data session 1. Then, the media sound (specific audio sound) is played on device A (tablet computer).

[0161] S508: Super Call Service Enabler B.

[0162] S509: The mobile phone and device B negotiate audio parameters.

[0163] S510: Register virtual device B.

[0164] Steps S508-S510 can be referred to as steps S306-S308 above, and will not be repeated here.

[0165] S511: Set specific audio strategies.

[0166] Step S511 can be referred to the aforementioned step S504, and will not be repeated here.

[0167] After the DMSDP module of the mobile phone sets a specific audio strategy in the audio framework, it executes step S512.

[0168] S512: Switch audio to virtual device B.

[0169] In this embodiment of the application, if device B receives an operation from the user to answer the call, that is, the user switches device B to play the call audio.

[0170] In this embodiment, the phone's audio framework determines whether the received audio data is media audio or call audio based on a specific audio strategy (StreamType-Phone prioritizes device B). If the received audio data belongs to StreamType-Phone, the audio framework switches the audio to virtual device B.

[0171] S513: Device ID 2 is associated with Session ID 2.

[0172] In this embodiment, the DMSDP module of the mobile phone associates the device identifier of device B with the session identifier of the created data session 2, that is, Devices2 is associated with session 2. The association between the device identifier of device B and the session identifier of data session 2 indicates that the mobile phone sends audio data packets to device B through data session 2.

[0173] S514: A specific audio sound is emitted from device B.

[0174] The phone's DMSDP module receives the Phone audio data packet from the hardware abstraction layer and sends the audio data packet from the phone (Source) side to the device B side based on the created DataSession2. Then, the call sound is played on the device B (PC).

[0175] As a result, the phone's audio is routed to multiple devices, allowing media audio and call audio to be played on different devices respectively.

[0176] After introducing the business process diagrams corresponding to the concurrent media audio and call audio scenarios, we will continue to introduce the business module sequence diagrams corresponding to the concurrent media audio and call audio scenarios.

[0177] Please see Figure 6 , Figure 6 This document presents a timing diagram of a service module for a scenario where a mobile phone and a tablet computer collaboratively play videos, while a PC answers incoming calls. (See the provided text for an example.) Figure 6As shown, the mobile phone acts as the source side and can include collaborative / call services, media applications, DMSDP Dv service, DMSDP virtual audio, audio framework AudioTrack, Audio HAL, and ADSP. The mobile phone connects collaboratively with devices A and B. Device A can include service management DvService, a virtual audio module, and the audio framework AudioTrack. Device B can also include service management DvService, a virtual audio module, and the audio framework AudioTrack.

[0178] in, Figure 6 Steps S601-S607 can be referred to Figure 4 Steps S401-S407 will not be elaborated here.

[0179] After the virtual device is successfully created, i.e., after its state is successfully synchronized to the audio framework, the DMSDP module creates a listener in the audio framework (S608). Specifically, the DMSDP DvService sends a creation message to the audio framework AudioTrack to create the listener. The listener is created to monitor changes to the audio device under a specific audio policy. The code for creating the listener can be as follows:

[0180] aud ioManager.addOnPreferredDevicesForStrategyChangedLi stener(newMyPreferredDevic esLi stener()).

[0181] Then, after setting the audio policy for a specific audio stream in the AudioTrack audio framework, DMSDP DvService can route a specific audio stream to multiple devices (S609). Example code is as follows:

[0182] int strategyId = AuditioManager.STRATEGY_MUSIC; / / Example strategy

[0183] ID int[]deviceIds = new int[]{0,1}; / / Example device list

[0184] aud ioManager.setPreferredDevicesForStrategy(strategyId,deviceIds)

[0185] It should be noted that there is no sequential relationship between the user opening Media Application 1 and the enabling of Service A. It is possible that the user opens Media Application 1 and then Service A enables the device, or the user opens Media Application 1 after Service A successfully enables the device.

[0186] After the audio policy is set in the phone's DMSDP module, the phone's audio framework, AudioTrack, switches the audio to virtual device A (S610) according to the set audio policy. Specifically, the AudioTrack framework sets the output device, carrying the device identifier of the output device. A code example is: SetOutputDevice(outputDeviceid).

[0187] Then, the audio framework AudioTrack sends audio data packets to virtual device A in Audio HAL via Thread 1 (S611). Virtual device A then sends audio data packets to the DMSDP virtual audio module. The DMSDP virtual audio module encodes the audio data packets (S612). Then, the DMSDP virtual audio module can distinguish device sessions by device identifier, determine the data session 1 associated with device A, and send the encoded audio data packets to device A based on data session 1 (S613). Specifically, the DMSDP virtual audio module sends the encoded audio data packets to the virtual audio module of device A.

[0188] Device A's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device A's audio framework, AudioTrack. Device B's audio framework, AudioTrack, plays the audio after receiving the audio data packets.

[0189] in, Figure 6 Steps S614-S619 can be referred to Figure 4 Steps S412-S417 will not be elaborated here.

[0190] After virtual device B is successfully created, DMSDP DvService, by setting the audio policy for a specific audio stream in the AudioTrack audio framework, can route a specific audio stream to multiple devices (S620). Example code is as follows:

[0191] int strategyId = AuditioManager.STRATEGY_PHONE; / / Example strategy

[0192] ID int[]deviceIds = new int[]{0,1}; / / Example device list

[0193] aud ioManager.setPreferredDevicesForStrategy(strategyId,deviceIds)

[0194] At this point, the mobile phone receives the user's device switching operation, for example, the mobile phone receives the user's operation to answer a call on device B. The Super Call service sends a switching message (switch ToRomoteDevice) to the DMSDP DvService. The DMSDP DvService sends the message to switch to device B to the audio framework AudioTrack. The audio framework AudioTrack switches the call audio to virtual device B according to the set audio policy (S621).

[0195] At this point, the audio framework AudioTrack can read downlink data packets from the ADSP module (S622). The DMSDP virtual audio can obtain downlink data packets from the audio framework AudioTrack and encode them. Then, the DMSDP virtual audio can distinguish device sessions by device identifier, determine data session 2 associated with device B, and send the encoded audio data packets (downlink data packets) based on data session 2. Specifically, the DMSDP virtual audio sends the encoded audio data packets to the virtual audio module of device B.

[0196] Device B's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device B's audio framework, AudioTrack. Upon receiving the audio data packets, AudioTrack plays the audio.

[0197] thus, Figure 6 The process shown demonstrates how audio strategies can be set to send audio data from different sources to different devices via different data channels for playback, enabling users to simultaneously watch videos and answer phone calls.

[0198] Next, this application embodiment introduces a second scenario: a multi-type media audio concurrent scenario, specifically: a mobile phone, a tablet computer, and a PC working collaboratively. During the collaborative playback of a video from application 1 on the mobile phone and a tablet computer, in response to the user's playback operation on application 2 on the mobile phone, the video from application 2 is played on the PC. The audio data in application 1 and the audio data in application 2 have the same audio source but different audio stream types.

[0199] In the current embodiment, see Figure 7 , Figure 7This is a diagram of the original modules for a multi-type media audio concurrency scenario provided in the current solution. For example... Figure 7 As shown, after the mobile phone (Source) side injects audio data packets of various audio stream types generated by different applications into the audio framework, the audio framework sends the audio data packets of various audio stream types to the audio Hidden 1 service in the HAL layer through a mixing single thread (thread 1). The audio Hidden 1 service forwards the audio data packets to the DMSDP service module through session 1. The DMSDP service module sends them to the device side (device A) through a data session.

[0200] Depend on Figure 7 It can be seen that the mobile phone sends the entire audio data packet to device A, and device A receives the mixed audio data.

[0201] Therefore, in order to enable playback of different audio stream types on different devices, this application embodiment needs to distinguish audio data of different audio stream types and perform stream splitting processing on the audio data packets. In this application embodiment, the mobile phone can perform stream splitting processing on the received audio data in the audio framework, or the audio framework can not perform processing, and the mobile phone can perform stream splitting processing on the received audio data in the hardware abstraction layer after sending the audio data from the audio framework to the hardware abstraction layer.

[0202] First, this application embodiment describes the process of the mobile phone's audio framework for splitting and processing received audio data. Please refer to... Figure 8 , Figure 8 This is a diagram of a streaming module for a multi-type media audio concurrent scenario provided in an embodiment of this application. For example... Figure 8 As shown, after the mobile phone (Source) injects various types of audio data packets (PCM audio data packets) generated by various applications into the audio framework, the audio framework splits the audio data packets according to their stream type (StreamType). That is, by distinguishing the StreamType of the audio data packets, different threads are used to send audio packets of different stream types to the HAL layer.

[0203] The Android system can categorize system sounds into multiple stream types, including at least the following: TREAM_ALARM: warning sound; STREAM_MUSIC: music sound, such as music; STREAM_RING: ringtone; STREAM_SYSTEM: system sound, such as low battery alert tone, lock screen tone, etc.; STREAM_VOCI_CALL: call sound.

[0204] It's important to note that the above classifications are unrelated to the audio data itself. For example, both "MUSI C" and "RING" can refer to a single MP3 song. Furthermore, there's no fixed standard for choosing the audio stream type; for instance, a ringtone in a ringtone preview can be set to the "MUSIC" type. The classification of audio stream types is related to the audio management strategy of the audio system.

[0205] like Figure 8 As shown, the audio framework can be categorized into streaming voice calls, streaming music, and streaming system based on the type of audio data packets. The audio framework sends voice call type audio data to the HAL layer via thread 1, music type audio data via thread 2, and system type audio data via thread 3. Here, "voice call" refers to Voice over Internet Protocol (VoIP) calls, not SIM card calls.

[0206] Audio data packets are forwarded in parallel through multiple paths at the HAL layer to the DMSDP module. Each audio stream has its own independent audio buffer (WriteStreamBuffer). Specifically, in the HAL layer, audio data for a voice call is sent to the audio buffer via route 1, and then the audio buffer sends the voice call audio data to the DMSDP module (DMSDP service module) via session 1. Similarly, audio data for music is sent to the audio buffer via route 2, and then the audio buffer sends the music audio data to the DMSDP module via session 2. Finally, audio data for the system is sent to the audio buffer via route 3, and then the audio buffer sends the system audio data to the DMSDP module via session 3.

[0207] The DMSDP module sends different types of audio data to different devices through different data sessions. Specifically, the DMSDP module sends the audio data of a voice call to device A through DataSession 1, the audio data of music to device B through DataSession 2, and the audio data of the system to device C through DataSession 3.

[0208] It is understandable that the audio data packet stream types correspond one-to-one with threads, routes, sessions, and data sessions.

[0209] Therefore, mobile phones can split the received audio data packets into different streams within the audio framework to obtain audio data of different stream types, and then send the audio data of different stream types to the corresponding devices through different data channels, so as to enable the playback of audio of different stream types on different devices.

[0210] Secondly, this application embodiment describes the process of a mobile phone performing splitting and processing of received audio data at the hardware abstraction layer. Please refer to... Figure 9 , Figure 9 This is a diagram of a splitting module for another type of concurrent media audio scenario provided in an embodiment of this application. For example... Figure 9 As shown, after the mobile phone (Source) injects various types of audio data packets generated by different applications into the audio framework, the audio framework does not process the audio data. The audio framework sends the various types of audio data packets to the HAL layer through a single mixing thread (thread 1). Then, the mobile phone performs packet splitting and processing on the audio data packets at the HAL layer. Specifically, the mobile phone can process packets based on the parameter (streamtype) carried in the packet header.

[0211] like Figure 9 As shown, the mobile phone splits audio data packets into three types: other, music, and voice call, and then forwards them in parallel to the DMSDP module. Each audio stream has its own independent audio buffer (WriteStreamBuffer). Specifically, in the HAL layer, other audio data is sent to the audio buffer via route 1, and then the audio buffer sends the other audio data to the DMSDP module (DMSDP service module) via session 1. In the HAL layer, music audio data is sent to the audio buffer via route 2, and then the audio buffer sends the music audio data to the DMSDP module via session 2. In the HAL layer, voice call audio data is sent to the audio buffer via route 3, and then the audio buffer sends the voice call audio data to the DMSDP module via session 3.

[0212] The DMSDP module sends different types of audio data to different devices through different data sessions. Specifically, the DMSDP module sends other types of audio data to device A through DataSession 1, sends music audio data to device B through DataSession 2, and sends voice call audio data to device C through DataSession 3.

[0213] Therefore, the mobile phone can obtain audio data of different stream types by unpacking and processing audio data packets at the hardware abstraction layer, and then send different types of audio data to the corresponding devices through different data channels, thus enabling the playback of audio of different stream types on different devices. Specifically, this application enables audio data of the same stream type to flow simultaneously on device A and device B, and also enables audio data of different stream types to flow separately on device A and device B.

[0214] It's important to note that device-to-device communication refers to the seamless transfer or sharing of data, information, and content between different devices. This means that users can start a task or activity on one device and continue it on another without losing any data or progress. With the widespread use of smartphones, tablets, laptops, and other devices, users frequently switch between them, such as checking emails on a phone and continuing to edit them on a computer.

[0215] Next, this application describes the specific process for concurrent processing of multiple types of media audio. Please refer to the embodiments below. Figure 10 , Figure 10 This application provides a business process diagram for concurrent processing of multiple types of media audio, as illustrated in its embodiments. Figure 10 As shown, steps S701-S714 are detailed below:

[0216] S701: Service A enables device A.

[0217] S702: The mobile phone and device A negotiate audio parameters.

[0218] S703: Register virtual device A.

[0219] Steps S701-S703 can be referred to as steps S301-S303 above, and will not be repeated here. Service A in step S701 can also be a collaborative service.

[0220] S704: Set a specific audio strategy.

[0221] In this embodiment, the DMSDP module of the mobile phone sets a specific audio strategy in the audio framework. This specific audio strategy may include specifying which devices will preferentially play audio streams of a certain type. For example, specifying that Music type audio data will preferentially play on tablets, and specifying that other types of audio data will preferentially play on PCs. The aforementioned audio data all originate from the same source, for example, from the media audio of the audio framework.

[0222] S705: Switch audio to virtual device A.

[0223] In this embodiment, the audio framework can determine the audio stream type of the audio data generated by the application opened by the user based on the audio policy set in step S704. Assume the audio stream type of the audio data generated by the application is Music type. The audio framework switches the audio to virtual device A according to the set audio policy.

[0224] S706: Device Identifier 1 is associated with Session Identifier 1.

[0225] S707: A specific audio sound is emitted from device A.

[0226] S708: Service B enables device B.

[0227] S709: The mobile phone and device B negotiate audio parameters.

[0228] S710: Register virtual device B.

[0229] Steps S706-S707 can be referred to as steps S506-S507 above, and steps S708-S710 can be referred to as steps S306-S308 above, and will not be repeated here. The specific audio sound in step S707 refers to audio data of type Music. Service B in step S708 can also be a collaborative service.

[0230] S711: Set specific audio strategies.

[0231] Step S711 can be referred to the aforementioned step S704, and will not be repeated here.

[0232] After the DMSDP module of the mobile phone sets a specific audio strategy in the audio framework, it executes step S712.

[0233] S712: Switch audio to virtual device B.

[0234] In this embodiment, the mobile phone's audio framework determines the audio stream type of the received audio data according to the specific audio strategy set in step S711. Assuming the received audio data's audio stream type is other than the Music type, the audio framework switches the audio to virtual device B.

[0235] S713: Device Identifier 2 is associated with Session Identifier 2.

[0236] S714: A specific audio sound is emitted from device B.

[0237] Steps S713-S714 can refer to the aforementioned steps S513-S514. The specific audio sound in step S714 refers to audio data of an audio stream type other than Music type.

[0238] Thus, the phone's audio is routed to multiple devices. When the audio source is the same, the audio is split according to the audio stream type, so that audio of different stream types can be played on different devices.

[0239] Next, this application's embodiments describe the specific processes between various modules in a multi-type media audio concurrent processing scenario. Please refer to... Figure 11 , Figure 11 This is a timing diagram of a service module for concurrent processing of multiple types of media audio provided in an embodiment of this application. For example... Figure 11 As shown, the mobile phone acts as the source side and can include collaborative / call services, media applications, DMSDP Dv service, DMSDP virtual audio, the audio framework AudioTrack, and Audio HAL. The mobile phone connects collaboratively with devices A and B. Device A can include a service management module (DvService), a virtual audio module, and the audio framework AudioTrack. Device B can also include a service management module (DvService), a virtual audio module, and the audio framework AudioTrack.

[0240] in, Figure 11 Steps S801-S807 can be referred to Figure 4 Steps S401-S407 will not be elaborated here.

[0241] Figure 11 Steps S808-S809 can be referred to Figure 6 Steps S608-S609. Among them, Figure 11 The audio strategy set under the specific audio in step S809 and Figure 6 The audio strategy set in step S609 is different. Figure 11 The audio policy set in the configuration is: specify that audio data of type Music should be sent to device A.

[0242] Example code is as follows:

[0243] int strategyId = AuditioManager.STRATEGY_MUSIC; / / Example strategy

[0244] ID int[]deviceIds = new int[]{0,1}; / / Example device list

[0245] aud ioManager.setPreferredDevicesForStrategy(strategyId,deviceIds)

[0246] After setting the audio policy in the phone's DMSDP module, the phone's audio framework can determine the audio stream type based on the set audio policy and then switch the audio to the corresponding device. For example... Figure 11 As shown, the phone's audio framework determines that the audio data type is Music. Based on the above audio strategy, it can be determined to switch the audio to virtual device A (S810).

[0247] After the phone's audio frame switches the audio to virtual device A, Figure 11 Steps S811-S813 can be referred to Figure 6 Steps S611-S613 will not be elaborated here.

[0248] Then, after the user opens media application 2 and media application 2 starts speaker playback, service B can enable device B. Among these, Figure 11 Steps S814-S820 can be referred to Figure 6 Steps S614-S620 will not be described in detail here.

[0249] in, Figure 11 In step S814, business B can be a collaborative business.

[0250] in, Figure 11 The audio strategy set for a specific audio level in step S820 and Figure 6 The audio strategy set in step S620 is different. Figure 11 The audio strategy set in the configuration is: specify that RING type audio data should be sent to device B. Example code is as follows:

[0251] int strategyId = AuditionManager.STRATEGY_RING; / / Example strategy

[0252] ID int[]deviceIds = new int[]{0,1}; / / Example device list

[0253] aud ioManager.setPreferredDevicesForStrategy(strategyId,deviceIds)

[0254] It should be noted that there is no sequential relationship between the user opening media application 2 and the enabling of business B. It is possible that the user opens media application 2 and then business B enables device B, or the user opens media application 2 after business B successfully enables device B.

[0255] Then, the phone's audio framework AudioTrack determines that the audio data type is RING. Based on the above audio strategy, it can be determined to switch the audio to virtual device B (S821).

[0256] At this point, the phone's audio framework, AudioTrack, sends audio data packets to virtual device B in the Audio HAL via Thread2 (S822). Virtual device B then sends audio data packets to the DMSDP virtual audio module. The DMSDP virtual audio module encodes the audio data packets. Then, the DMSDP virtual audio module can distinguish the device session by the device identifier, determine the data session 2 associated with device B, and send the encoded audio data packets to device B based on data session 2. Specifically, the DMSDP virtual audio module sends the encoded audio data packets to the virtual audio module of device B.

[0257] Device B's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device B's audio framework, AudioTrack. Upon receiving the audio data packets, Device B's AudioTrack plays the audio.

[0258] thus, Figure 11 The illustrated process, by setting an audio strategy, sends audio data from the same audio source to different devices for playback through different data channels based on the stream type, enabling audio of different stream types to be played on different devices. Furthermore, if the phone sets device priorities in the audio strategy, for example, setting two devices (device A and device B) to have the same priority, audio of the same stream type can be streamed simultaneously on device A and device B.

[0259] Finally, this application embodiment introduces a third scenario: a multi-application audio concurrency scenario, specifically: a mobile phone, a tablet computer, and a PC working collaboratively. During the collaborative playback of a video from application 1 on the mobile phone and tablet computer, in response to the user's playback operation on application 2 on the mobile phone, the video from application 2 is played on the PC. The audio data in application 1 and the audio data in application 2 have the same audio source and the same audio stream type. This application embodiment can send audio from different applications with the same audio stream type to different devices, achieving single-application audio splitting.

[0260] In the current embodiment, after the mobile phone (Source) injects the audio data packets generated by various applications into the audio framework, the audio framework sends the audio data packets to the audio Hid 1 service in the HAL layer through a mixing single thread (thread 1). The audio Hid 1 service forwards the audio data packets to the DMSDP service module through session 1. The DMSDP service module then sends the data packets to the device side (device A) through a data session.

[0261] In this process, the mobile phone sends the audio data packets from each application to device A, and device A receives the mixed audio data.

[0262] Therefore, in order to play audio from different applications on different devices, this embodiment of the application needs to distinguish the package name (pkgName) of different applications and process the audio data packets according to the application package name. In this embodiment, the mobile phone can classify the received audio data according to the application's package name or process name within the audio framework.

[0263] First, this application embodiment describes the process of the mobile phone's audio framework for splitting and processing received audio data. Please refer to... Figure 12 , Figure 12 This is a diagram of a streaming module for a multi-application audio data concurrency scenario provided in an embodiment of this application. For example... Figure 12 As shown, after the mobile phone (Source) injects the audio data packets generated by various applications into the audio framework, the audio framework classifies them according to the application's package name or process name. That is, by distinguishing the package name pkgName or process name carried in the audio data packets, different threads are used to send audio packets of different applications to the HAL layer. Figure 12 As shown, taking the classification of audio data packets based on the packet name carried in the audio data packets as an example, the audio framework can classify the audio data packets into Application 1, Application 2, and Application 3 according to the packet name. The audio framework sends the audio data of Application 1 to the HAL layer through thread 1, the audio framework sends the audio data of Application 2 to the HAL layer through thread 2, and the audio framework sends the audio data of Application 3 to the HAL layer through thread 3.

[0264] Audio data packets are forwarded in parallel through multiple paths at the HAL layer to the DMSDP module. Each audio stream has its own independent audio buffer (WriteStreamBuffer). Specifically, in the HAL layer, audio data for application 1 is sent to the audio buffer via route 1, and then the audio buffer sends the audio data for application 1 to the DMSDP module (DMSDP service module) via session 1. Similarly, audio data for application 2 is sent to the audio buffer via route 2, and then the audio buffer sends the audio data for application 2 to the DMSDP module via session 2. Finally, audio data for application 3 is sent to the audio buffer via route 3, and then the audio buffer sends the audio data for application 3 to the DMSDP module via session 3.

[0265] The DMSDP module sends audio data generated by different applications to different devices through different data sessions. Specifically, the DMSDP module sends the audio data of application 1 to device A through DataSession 1, the audio data of application 2 to device B through DataSession 2, and the audio data of application 3 to device C through DataSession 3.

[0266] Therefore, the mobile phone can process the received audio data packets by splitting them according to the packet name carried in the audio data packet in the audio framework, obtain the audio data of different applications, and then send the audio data of different applications to the corresponding devices through different data channels, so as to enable the audio of different applications to be played on different devices.

[0267] In some embodiments, the mobile phone can also classify and process the received audio data according to the application's package name or process name at the hardware abstraction layer. Specifically, the phone's audio framework does not process the received audio data packets. When sending the audio data packets to the hardware abstraction layer, the data packets are encapsulated once, carrying information such as the application's package name and / or process name. Then, the phone processes the received audio data in a stream according to the application's package name or process name at the hardware abstraction layer.

[0268] Next, this application describes the specific process of multi-application audio concurrent processing. Please refer to the embodiments below. Figure 13 , Figure 13 This document provides a business process diagram for concurrent audio processing across multiple applications, as illustrated in an embodiment of this application. Figure 13 As shown, steps S901-S914 are detailed below:

[0269] S901: Service A enables device A.

[0270] S902: The mobile phone and device A negotiate audio parameters.

[0271] S903: Register virtual device A.

[0272] Steps S901-S903 can be referred to as steps S301-S303 above, and will not be repeated here. Service A in step S701 can also be a collaborative service.

[0273] S904: Register listener.

[0274] After virtual device A is successfully created, that is, after the state of virtual device A is successfully synchronized to the audio framework, the phone's DMSDP module registers a listener in the audio framework. This listener registration is to obtain the media application's package name when the audio framework receives audio data from the media application.

[0275] S905: Get the package name of application 1.

[0276] In this embodiment, the phone's audio framework obtains the package name of the application currently opened by the user. If the application currently opened by the user is application 1, and the corresponding package name is pkgName1, then the phone's audio framework can obtain the package name pkgName1 from the received audio data packet.

[0277] S906: Set the audio output device to virtual device A.

[0278] In this embodiment, upon obtaining the packet name, the mobile phone sets up an audio routing device (audio output device). Assuming the audio routing device is device A, meaning the mobile phone sets its audio output device to the virtual device A corresponding to device A, the mobile phone associates the packet name pkgName1 with the device identifier DeviceId1 of device A.

[0279] Then, the phone's DMSDP module can generate the corresponding session name or session identifier based on the packet name. Assuming the phone's DMSDP module generates the corresponding session name session1 based on the obtained packet name pkgName1, the phone can associate pkgName1 with session1 and then send audio data packets to device A corresponding to DeviceId1 through session1.

[0280] It should be noted that the session 'session1' created by the DMSDP module of the mobile phone is also known as the data session 'DataSession1'.

[0281] There is a one-to-one correspondence between pkgName, DeviceId, and sessionname. The mobile phone can store the relationship between these three in a table format, as shown in Table 1 below:

[0282]

[0283]

[0284] Table 1

[0285] S907: Application 1: Sound is emitted from device A.

[0286] The DMSDP module of the mobile phone will take the audio data packet obtained from the hardware abstraction layer and send the audio data packet from the mobile phone (Source) side to the device A side through data session 1 created based on packet name 1. Then, the sound of application 1 will be played on device A (tablet computer).

[0287] S908: Service B enables device B.

[0288] When the mobile phone detects that the user has opened application 2, service B enables device B. In step S908, "service B enables device B" can refer to step S306, and service B in step S908 can also be a collaborative service.

[0289] S909: The mobile phone and device B negotiate audio parameters.

[0290] S910: Register virtual device B.

[0291] S911: Register listener.

[0292] S912: Get the package name of application 2.

[0293] S913: Set the audio output device to virtual device B.

[0294] S914: Application 2 sound is emitted from device B.

[0295] Steps S908-S910 can refer to the aforementioned steps S306-S308, and steps S911-S914 can refer to steps S904-S907, and will not be repeated here.

[0296] In this scenario, assume the packet name for application 2 is pkgName2, the audio routing device is device B, and device B's device identifier is DeviceId2. If the DMSDP module on the mobile phone generates a session named session2 based on pkgName2, then the DMSDP module can send audio data packets to device B (corresponding to DeviceId2) through session2. Therefore, sound from application 2 will be emitted from device B (PC).

[0297] Thus, the phone's audio is routed to multiple devices, enabling audio data from different applications to be played on different devices when the audio source and data type are the same.

[0298] Next, this application's embodiments describe the specific processes between various modules in a multi-application audio concurrent processing scenario. Please refer to... Figure 14 , Figure 14 This is a timing diagram of a business module for multi-application audio concurrent processing provided in an embodiment of this application. (See diagram below.) Figure 14 As shown, the mobile phone acts as the source side and can include collaborative / call services, media applications, DMSDP Dv service, DMSDP virtual audio, the audio framework AudioTrack, and Audio HAL. The mobile phone connects collaboratively with devices A and B. Device A can include the service management DvService, the virtual audio module, and the audio framework AudioTrack. Device B can also include the service management DvService, the virtual audio module, and the audio framework AudioTrack.

[0299] in, Figure 14 Steps S1101-S1107 can be referred to Figure 4 Steps S401-S407 will not be elaborated here.

[0300] After the virtual device is successfully created, that is, after the virtual device's state is successfully synchronized to the audio framework, the mobile phone registers (creates) a listener in the audio framework. Specifically, DMSDP DvService sends a registration message to the audio framework AudioTrack to register a listener (S1108). Registering a listener is for the purpose of obtaining the media application package name (pkgName) when the audio framework detects that application 1 sends a message to the audio framework to start the speaker to play sound, i.e., when the audio framework receives the audio data injected by media application 1.

[0301] The DMSDP module also sets up an audio strategy within the audio framework: audio distribution based on the application's package name. Specifically, the audio framework sends audio data from different applications to different devices based on the application's package name. For example, it specifies that audio data from the source application with package name "package name 1" should be sent to device A.

[0302] After successful registration and listening, the audio framework AudioTrack sends a request to the DMSDP virtual audio to obtain the package name of media application 1 (S1109). Let the package name of media application 1 be pkgName1.

[0303] After obtaining the packet name of the media application, the DMSDP virtual audio device sets the audio routing device (S1110) using `setOutputDevice(outputDeviceId)`. The audio routing device is also the audio output device. Assume the audio routing device at this time is device A, and device A's DeviceId is Device1.

[0304] After configuring the audio routing device in the DMSDP virtual audio, a notification message is sent to the audio framework AudioTrack. The audio framework AudioTrack then sends audio data packets to virtual device A via thread 1 (S1111).

[0305] Virtual device A then sends the received audio data packet to the DMSDP virtual audio server. The DMSDP virtual audio server encodes the audio data packet (S1112) to obtain the encoded audio data packet. Then, the DMSDP virtual audio server associates pkgName1 with DeviceId1 of device A.

[0306] Then, the DMSDP virtual audio can generate a session name (set as session1) based on pkgName1, associate pkgName1 with session1, and send the encoded audio data packet to the virtual audio module of device A through data session 1 corresponding to session1 (S1113).

[0307] Device A's virtual audio module decodes the received encoded audio data packets to obtain audio data packets. Then, the virtual audio module sends the audio data packets to Device A's audio framework, AudioTrack. Upon receiving the audio data packets, AudioTrack plays the audio.

[0308] At this point, the user can open media application 2. The media application sends a message to the audio framework AudioTrack to start the speaker to play sound. Then, service B enables device B (S1114). Among these... Figure 14 Steps S1114-S1119 can be referred to Figure 11 Steps S814-S819 will not be elaborated here.

[0309] After virtual device B is successfully created, DMSDP DvService sends a registration message to the audio framework AudioTrack to register for listening (S1120). Registering for listening is to obtain the media application package name (pkgName) when the audio framework receives audio data injected by media application 2 and detects that application 2 sends a message to the audio framework to start the speaker to play sound.

[0310] The DMSDP module also sets up an audio strategy within the audio framework: audio distribution based on the application's package name. Specifically, the audio framework sends audio from different applications to different devices based on the application's package name.

[0311] It should be noted that there is no sequential relationship between the user opening Media Application 2 and the enabling of Business B. It is possible that the user opens Media Application 2 and then Business B enables the device, or the user opens Media Application 2 after Business B successfully enables the device.

[0312] After successful registration and listening, the audio framework AudioTrack sends a request to the DMSDP virtual audio to obtain the package name of media application 2 (S1121). Let the package name of media application 2 be pkgName2.

[0313] After obtaining the packet name of the media application, the DMSDP virtual audio device sets the audio routing device (S1122) using `setOutputDevice(outputDeviceId)`. Assume the audio routing device at this time is device B, and device B's `DeviceId` is `Device2`. Specifically, the DMSDP virtual audio device associates `pkgName2` with device B's `DeviceId2`.

[0314] After configuring the audio routing device in the DMSDP virtual audio, a notification message is sent to the audio framework AudioTrack. The audio framework AudioTrack then sends audio data packets to virtual device A via thread 2 (S1123).

[0315] Virtual device B then sends the received audio data packets to DMSDP virtual audio, which encodes the audio data packets to obtain encoded audio data packets.

[0316] Then, the DMSDP virtual audio can generate a session name (set as session2) based on pkgName2, associate pkgName2 with session2, and send the encoded audio data packet to the virtual audio module of device B through the data session 2 corresponding to session2.

[0317] Device B's virtual audio module decodes the received encoded audio data packets to obtain the audio data packets. Then, the virtual audio module sends the audio data packets to Device B's audio framework, AudioTrack. Upon receiving the audio data packets, AudioTrack plays the audio.

[0318] thus, Figure 14 The method shown classifies audio data according to the application package name carried in the audio data, selects audio data generated by different applications, and sends it to different devices for playback through different data channels, enabling users to play the sound of different applications on multiple devices at the same time.

[0319] In summary, the audio data processing method provided in this application can achieve the playback of audio data with different feature information on different devices by sending audio data with different feature information to different devices, thereby optimizing the audio streaming experience.

[0320] For example, the electronic device in this application embodiment has an audio module, which may specifically include at least one of the following: mobile phone, foldable electronic device, tablet computer, desktop computer, laptop computer, handheld computer, laptop, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) device, virtual reality (VR) device, artificial intelligence (AI) device, wearable device, in-vehicle device, smart home device, or smart city device. This application embodiment does not impose any special limitation on the specific type of the electronic device.

[0321] The embodiments of this application will now be described in detail with reference to the accompanying drawings. The electronic device described above can be a mobile phone or a device. Taking a mobile phone as an example, the hardware structure of the electronic device 100 will be described. The hardware structure of the electronic device can be found in the detailed description of the electronic device 100 in the embodiments of this application, and will not be repeated here. Please refer to... Figure 15 , Figure 15 A schematic diagram of the electronic device is shown, such as... Figure 15 As shown, the electronic device 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a display screen 194, a subscriber identification module (SIM) card interface 195, a camera, a universal serial bus (USB) connector, a charging management module, a power management module, a battery, an antenna 1, an antenna 2, etc.

[0322] The aforementioned sensor module 180 may include pressure sensors, gyroscope sensors, barometric pressure sensors, magnetic sensors, accelerometers, distance sensors, proximity sensors, fingerprint sensors, temperature sensors, touch sensors, ambient light sensors, bone conduction sensors, and so on.

[0323] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0324] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0325] The processor 110 can generate operation control signals based on the instruction opcode and timing signals to control the instruction fetching and execution.

[0326] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 may be a cache memory. This memory can store instructions or data that the processor 110 has used or that are used frequently. If the processor 110 needs to use the instruction or data, it can directly retrieve it from this memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0327] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc. The processor 110 can connect to modules such as touch sensors, audio modules, wireless communication modules, displays, and cameras through at least one of these interfaces.

[0328] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0329] A USB connector is an interface conforming to the USB standard specification, used to connect an electronic device 100 to peripheral devices. Specifically, it can be a Mini USB connector, a Micro USB connector, a USB Type-C connector, etc. A USB connector can be used to connect a charger, allowing the charger to charge the electronic device 100. It can also be used to connect other electronic devices, enabling data transfer between them. It can also be used to connect headphones, allowing audio stored in the electronic device to be output. The connector can also be used to connect other electronic devices, such as VR devices. In some embodiments, the Universal Serial Bus standard specification can be USB 1.x, USB 2.0, USB 3.x, and USB 4.

[0330] The wireless communication function of electronic device 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0331] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0332] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0333] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), Bluetooth Low Energy (BLE), ultra-wideband (UWB), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies.

[0334] The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0335] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 100 to communicate with networks and other electronic devices via wireless communication technology. This wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0336] Electronic device 100 can implement display functions through a GPU, display screen 194, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0337] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or more display screens 194.

[0338] Electronic device 100 can realize camera function through camera module 193, ISP, video codec, GPU, display screen 194, application processor AP, neural network processor NPU, etc.

[0339] In some embodiments, the electronic device 100 may include one or more camera modules 193. Specifically, the electronic device 100 may include one front-facing camera module 193 and one rear-facing camera module 193. The front-facing camera module 193 is typically used to capture color image data and depth data of the photographer facing the display screen 194, while the rear-facing camera module is used to capture color image data and depth data of the subject (such as a person, landscape, etc.) in front of the photographer.

[0340] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0341] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card, or music, video, and other files can be transferred from the electronic device to the external memory card.

[0342] Internal memory 121 can be used to store computer executable program code, including instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of electronic device 100 (such as audio data, phonebook, etc.). In addition, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional methods or data processing of electronic device 100 by running instructions stored in internal memory 121 and / or instructions stored in memory disposed in the processor.

[0343] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0344] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0345] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The electronic device 100 can listen to music through the speaker 170A or output audio signals for hands-free calling.

[0346] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the electronic device 100 answers a telephone call or voice message, the receiver 170B can be brought close to the ear to listen to the voice.

[0347] Microphone 170C, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. When making a phone call or sending a voice message, the user can speak by bringing their mouth close to microphone 170C, inputting the sound signal into microphone 170C. Electronic device 100 may have at least one microphone 170C. In some embodiments, electronic device 100 may have two microphones 170C, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device 100 may also have three, four, or more microphones 170C, which can collect sound signals, reduce noise, identify the sound source, and perform directional recording, etc.

[0348] The 170D headphone jack is used to connect wired headphones. The 170D headphone jack can be a USB 130 interface or a 3.5mm Open Mobile Terminal Platform (OMTP) standard interface, a CTIA (Cellular Telecommunications Industry Association of the USA) standard interface.

[0349] Button 190 may include a power button, volume buttons, etc. Button 190 may be a mechanical button or a touch button. Electronic device 100 may receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0350] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can correspond to touch operations performed on different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations performed on different areas of the display screen 194. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0351] The indicator 192 can be an indicator light, used to indicate charging status, battery level changes, or messages, missed calls, notifications, etc. In this embodiment, the indicator can be used to indicate incoming call reminders.

[0352] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the electronic device 100. The electronic device 100 can support one or more SIM card interfaces. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The electronic device 100 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the electronic device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the electronic device 100 and cannot be separated from the electronic device 100.

[0353] The methods described in the following embodiments can all be implemented in the electronic device 100 having the above-described hardware structure.

[0354] This application also provides a chip system, such as... Figure 16 As shown, the chip system 1600 includes at least one processor 1601 and at least one interface circuit 1602. The processor 1601 and the interface circuit 1602 are interconnected via lines. For example, the interface circuit 1602 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1602 can be used to send signals to other devices (e.g., the processor 1601). Exemplarily, the interface circuit 1602 can read instructions stored in memory and send those instructions to the processor 1601. When the instructions are executed by the processor 1601, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, and this application embodiment does not specifically limit this.

[0355] This application also provides a computer storage medium that includes computer instructions. When the computer instructions are executed on the electronic device, the electronic device causes the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiment.

[0356] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments.

[0357] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0358] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0359] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0360] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0361] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0362] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An audio data processing method, characterized in that, The method is applied to a first electronic device, which is also communicatively connected to a second electronic device and a third electronic device; the method includes: In response to the first service, the first electronic device creates a first control channel and negotiates audio parameters with the second electronic device through the first control channel to enable the second electronic device; The first electronic device registers the first virtual audio device corresponding to the second electronic device; The first electronic device associates the audio data of the first service with the first virtual audio device based on an audio policy; the first device identifier corresponding to the first virtual audio device is associated with the first data channel; The first electronic device sends the first audio data of the first service to the second electronic device through the first data channel; In response to the second service, the first electronic device creates a second control channel and negotiates audio parameters with the third electronic device through the second control channel to enable the third electronic device; The first electronic device registers the second virtual audio device corresponding to the third electronic device; The first electronic device associates the audio data of the second service with the second virtual audio device based on an audio policy; the second device identifier corresponding to the second virtual audio device is associated with the second data channel; The first electronic device sends the second audio data of the second service to the third electronic device through the second data channel; Wherein, while the first electronic device sends first audio data to the second electronic device to make the second electronic device play the first audio data, the first electronic device sends second audio data to the third electronic device to make the third electronic device play the second audio data; The audio strategy includes: determining the virtual audio device associated with the audio data based on the feature information of the audio data, wherein the feature information includes at least one of the following: source application, stream type, and transmission path within the first electronic device.

2. The method according to claim 1, characterized in that, Before the first electronic device sends the first audio data to the second electronic device, the method further includes: The first electronic device establishes a first mapping relationship between the first audio data and the second electronic device based on the feature information corresponding to the first audio data; Wherein, the first electronic device sends first audio data to the second electronic device, including: The first electronic device determines the first audio data based on the feature information of the acquired audio data; The first electronic device sends the first audio data to the second electronic device based on the first mapping relationship.

3. The method according to claim 2, characterized in that, Before the first electronic device sends the first audio data to the second electronic device, the method further includes: The first electronic device creates a first data channel, and the first data channel has a corresponding first channel identifier; The first electronic device establishes a first association relationship between the first channel identifier and the first device identifier of the second electronic device; Wherein, the first electronic device sends first audio data to the second electronic device, including: The first electronic device sends the first audio data to the second electronic device through the first data channel based on the first mapping relationship and the first association relationship.

4. The method according to any one of claims 1-3, characterized in that, Before the first electronic device sends the second audio data to the third electronic device, the method further includes: The first electronic device establishes a second mapping relationship between the second audio data and the third electronic device based on the feature information corresponding to the second audio data; Wherein, the first electronic device sends second audio data to the third electronic device, including: The first electronic device determines the second audio data based on the feature information of the acquired audio data; The first electronic device sends the second audio data to the third electronic device based on the second mapping relationship.

5. The method according to claim 4, characterized in that, Before the first electronic device sends the second audio data to the third electronic device, the method further includes: The first electronic device creates a second data channel, and the second data channel has a corresponding second channel identifier; The first electronic device establishes a second association between the second channel identifier and the second device identifier of the third electronic device; Wherein, the first electronic device sends second audio data to the third electronic device, including: The first electronic device sends the second audio data to the third electronic device through the second data channel based on the second mapping relationship and the second association relationship.

6. The method according to any one of claims 1-5, characterized in that, The feature information includes at least one of the source application and the stream type; before the first electronic device sends first audio data to the second electronic device to cause the second electronic device to play the first audio data, and before the first electronic device sends second audio data to the third electronic device to cause the third electronic device to play the second audio data, the method further includes: The first electronic device performs split processing on the audio data acquired by the first electronic device based on the feature information, and determines the first audio data and the second audio data.

7. The method according to claim 6, characterized in that, The first electronic device includes a hardware abstraction layer and an audio framework; the first electronic device performs split processing on the audio data acquired by the first electronic device based on the feature information, and determines the first audio data and the second audio data, including: Based on the feature information, the first electronic device performs splitting processing on the audio data acquired by the first electronic device at the hardware abstraction layer to determine the first audio data and the second audio data; or, Based on the feature information, the first electronic device performs split processing on the audio data acquired by the first electronic device within the audio framework to determine the first audio data and the second audio data.

8. The method according to any one of claims 1-5, characterized in that, While the first electronic device sends first audio data to the second electronic device to cause the second electronic device to play the first audio data, the first electronic device also sends second audio data to the third electronic device to cause the third electronic device to play the second audio data, including: While the first electronic device sends first audio data to the second electronic device to cause the second electronic device to play the first audio data, in response to the user's call answering operation on the third electronic device, the first electronic device acquires the call audio as the second audio data. The first electronic device sends second audio data to the third electronic device, so that the third electronic device plays the second audio data; The transmission path of the first audio data within the first electronic device is different from the transmission path of the second audio data within the first electronic device.

9. The method according to claim 8, characterized in that, The first electronic device includes an audio call module and an audio frame; the transmission path of the first audio data passes through the audio frame, and the transmission path of the second audio data passes through the audio call module.

10. An audio system, characterized in that, The system includes a first electronic device, a second electronic device, and a third electronic device, wherein the first electronic device is communicatively connected to both the second electronic device and the third electronic device; the system includes: In response to the first service, the first electronic device creates a first control channel and negotiates audio parameters with the second electronic device through the first control channel to enable the second electronic device; The first electronic device registers the first virtual audio device corresponding to the second electronic device; The first electronic device associates the audio data of the first service with the first virtual audio device based on an audio policy; the first device identifier corresponding to the first virtual audio device is associated with the first data channel; The first electronic device sends the first audio data of the first service to the second electronic device through the first data channel; In response to the second service, the first electronic device creates a second control channel and negotiates audio parameters with the third electronic device through the second control channel to enable the third electronic device; The first electronic device registers the second virtual audio device corresponding to the third electronic device; The first electronic device associates the audio data of the second service with the second virtual audio device based on an audio policy; the second device identifier corresponding to the second virtual audio device is associated with the second data channel; The first electronic device sends the second audio data of the second service to the third electronic device through the second data channel; The second electronic device receives the first audio data sent by the first electronic device and plays the first audio data; The third electronic device receives the second audio data sent by the first electronic device and plays the second audio data; The audio strategy includes: determining the virtual audio device associated with the audio data based on the feature information of the audio data, wherein the feature information includes at least one of the following: source application, stream type, and transmission path within the first electronic device.

11. An electronic device, characterized in that, The electronic device includes: an audio module, a communication module, a memory, and one or more processors; the audio module, the communication module, the memory, and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions, which, when executed by the electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed in an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes instructions that, when executed in an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 9.

Images