Audio processing method, electronic device, storage medium and chip system

Abstract

Embodiments of the present application provide an audio processing method, an electronic device, a storage medium and a chip system, and relate to the technical field of terminals. The method comprises: a first device and a second device establish a communication connection; when a plurality of applications of the first device have first audio data to be played, the first device can perform audio splitting on the first audio data to be played of the plurality of applications to obtain a plurality of first stream data; after obtaining the plurality of first stream data, the first device can start a first thread, and merge the plurality of first stream data into a first data packet; the first device can split the first data packet into a plurality of sub-packet data to be transmitted, and send the plurality of sub-packet data to the second device. In this way, the number of interactions between internal modules or processes of the first device can be reduced, thereby the transmission speed and stability of the first device for audio data can be improved, and the second device can avoid playing audio data with a lag phenomenon, and the user's auditory experience can be improved.

Description

Technical Field

[0001] This application relates to the field of terminal technology, and in particular to an audio processing method, electronic device, storage medium and chip system. Background Technology

[0002] With the rapid development of automotive intelligence, vehicle-to-everything (V2X) connectivity has become widely used. V2X connectivity refers to connecting vehicle-mounted systems and electronic devices via wireless communication technologies (such as Bluetooth) to enable data transmission and function sharing between them. In one scenario, when an electronic device is transmitting multiple audio streams to the vehicle-mounted system while simultaneously performing other tasks or processes, the limited resources of the device can affect the transmission speed and stability of the multi-channel audio data, causing stuttering during audio playback on the vehicle-mounted system. This not only impacts the user's auditory experience but also jeopardizes driving safety. Therefore, there is an urgent need to adjust the way electronic devices process audio. Summary of the Invention

[0003] This application provides an audio processing method, electronic device, storage medium, and chip system, which are applied in the field of terminal technology. They can avoid the stuttering phenomenon of multiple audio data, thereby improving the user's listening experience and driving safety.

[0004] In a first aspect, embodiments of this application propose an audio processing method. The subject executing this method can be a first electronic device (hereinafter referred to as the first device) or a chip in the first electronic device. The following description uses the first device as an example. The method includes: when the first device and the second device establish a communication connection, when multiple applications of the first device have first audio data to be played, the first device can split the first audio data to be played from the multiple applications to obtain multiple first stream data. After obtaining the multiple first stream data, the first device can start a first thread. The first thread is used to merge the multiple first stream data into a first data packet. The first device can split the first data packet into multiple sub-data packets to be transmitted and send the multiple sub-data packets to the second device.

[0005] Here, the first device and the second device are two different types of devices. For example, in a vehicle-to-everything (V2X) scenario, the first device could be a mobile phone, and the second device could be a vehicle-mounted infotainment system. The first thread can correspond to... Figure 6 The sub-thread of the audio HAL can correspond to multiple first-stream data streams. Figure 6 The first data packet of the multi-stream data in the S602 can correspond to... Figure 6 Total data packets of S607 in China.

[0006] In this embodiment, the first device can combine multiple first stream data into a single packet using a first thread. Before splitting the first data packet into multiple sub-data packets to be transmitted, the first device can obtain the combined data (i.e., the first data packet) from the first thread at once. Compared to obtaining multiple stream data separately from the first thread, this greatly reduces the number of interactions between modules or processes within the first device, thereby improving the transmission speed and stability of audio data. It can effectively prevent stuttering when the peer device (i.e., the second device) plays audio data, thus improving the user's auditory experience.

[0007] In one possible implementation, the first device may create the first thread before initiating it, and upon receiving a device connection event. Creating the first thread may include configuring a timer for it, the duration of which indicates the interval for sending data packets. The device connection event may be initiated by a second device or act on a first application (such as a settings application) on the first device.

[0008] It should be understood that the data packet can be the first data packet or other data packets besides the first data packet (such as the second data packet described below).

[0009] In this implementation, the first device creates a first thread, which enables the first device to combine multiple first-stream data streams into a single packet.

[0010] In one possible implementation, during the creation of the first thread by the first device, the first device may create a first buffer, which is used to store multistream data. Since the main thread of the first device is contained within the first device, the creation of the buffer by the main thread of the first device can be referred to as the creation of the first buffer by the first device.

[0011] In this implementation, the first device creates a first buffer, which can store multi-stream data.

[0012] In one possible implementation, after the first thread of the first device merges multiple first stream data into a first data packet, the first device can delete the multiple first stream data in the first buffer.

[0013] In this implementation, since the space of the first buffer is limited, after the first thread of the first device merges multiple first stream data into the first data packet, the first device can delete the multiple first stream data in the first buffer to release the space occupied by the multiple first stream data in the first buffer, which can improve the audio data processing efficiency of the first device.

[0014] In one possible implementation, the first data packet includes, in sequence, a first field, a second field, and a third field. The first field can be used to indicate general information about the multiple first stream data in the first data packet, the second field can be used to indicate the transmission format of the first data packet, and the third field can be used to indicate the multiple first stream data in the first data packet, as well as the attribute information of each first stream data.

[0015] This implementation demonstrates a new method of package assembly, which can be found in the following reference. Figure 7B The first thread of the first device can combine multiple first-stream data streams into a single packet using this new packet assembly method (including writing the multiple first-stream data streams into corresponding fields) to obtain the first data packet. Furthermore, the device virtualization service of the first device can also split the first data packet using this new packet assembly method to obtain multiple first-stream data streams.

[0016] In one possible implementation, the general information for multiple first-stream data streams may include: the identifier of the second device, the service identifier, the session identifier, a first flag, and the length of the first data packet. The first flag indicates that the first data packet is a combined data stream. The attribute information for each first-stream data stream includes the stream type, length, and audio type.

[0017] For example, refer to Figure 7B The general information of multiple first-stream data corresponds to "info", the identifier of the second device can correspond to "deviceId", the service identifier can correspond to "serviceId", the session identifier can correspond to "sessionId", the first flag can correspond to "mergeFlag", and the length of the first data packet can correspond to "totalLen".

[0018] In one possible implementation, the first device may include an audio management module. The audio management module of the first device can receive first audio data to be played from multiple applications, and the audio management module can split the first audio data to be played from multiple applications into multiple first stream data according to the audio data type corresponding to the preset stream type.

[0019] Each of the multiple first-stream data streams includes first audio data to be played from at least one application.

[0020] In this implementation, since the audio management module of the first device can receive first audio data to be played from multiple applications, it can achieve unified management of the first audio data to be played from multiple applications. Furthermore, since the audio management module splits the first audio data to be played from multiple applications based on the preset audio data type corresponding to the stream type, each stream type contains audio data of the corresponding type.

[0021] In one possible implementation, the first device also includes an audio hardware abstraction layer (HAL). The audio management module can send multiple first-stream data to the audio HAL. After receiving the multiple first-stream data, the audio HAL can start a first thread.

[0022] The audio HAL contains a main thread. After receiving multiple first-stream data streams, the audio HAL can start its first thread.

[0023] In this implementation, the audio HAL starts a first thread, which can combine multiple first-stream data streams to obtain a first data packet.

[0024] In one possible implementation, the first device may include a device virtualization service. The first thread can send a first data packet to the device virtualization service via a Binder call. Upon receiving the first data packet, the device virtualization service can split and encode the first data packet to obtain multiple sub-data packets to be transmitted. The device virtualization service of the first device can then send the multiple sub-data packets to the second device.

[0025] In this implementation, the first thread sends the first data packet to the device virtualization service via a Binder call. This single Binder call reduces the number of Binder calls and conserves resources on the first device, thus preventing lag. The device virtualization service splits and encodes the first data packet to ensure its secure transmission. The first device's virtualization service then sends multiple sub-data packets to the second device, enabling playback of these sub-data packets on the second device.

[0026] In one possible implementation, after the first thread sends a first data packet to the device virtualization service via a Binder call, it can start a timer. When the timer expires, the first thread can send a second data packet to the device virtualization service via a Binder call. This second data packet is a merged data packet containing multiple second-stream data. Figure 6 The total data packet of the S620. The multiple second-stream data consists of multiple streams of audio data to be played from multiple applications on the first device, corresponding to the multiple streams of second-stream data. Figure 6 Multi-stream data of S616.

[0027] In this implementation, since the second data packet is a data packet that merges multiple second-stream data, the first thread only needs to make one Binder call to send the second data packet to the device virtualization service. This reduces the number of Binder calls, thereby saving resources for the first device and avoiding stuttering. Furthermore, because the first thread starts a timer, it can continue sending the second data packet to the device virtualization service even after the timer expires, enabling the second device to continuously play both the first and second data packets.

[0028] Secondly, embodiments of this application provide an audio processing apparatus, which may be an electronic device, or a chip or chip system within an electronic device. The audio processing apparatus may include a display unit and a processing unit. When the audio processing apparatus is an electronic device, the display unit may be a display screen. The display unit is used to perform display steps to cause the electronic device to implement an audio processing method described in the first aspect or any possible implementation of the first aspect. When the audio processing apparatus is an electronic device, the processing unit may be a processor. The audio processing apparatus may further include a storage unit, which may be a memory. The storage unit is used to store instructions, and the processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an audio processing method described in the first aspect or any possible implementation of the first aspect. When the audio processing apparatus is a chip or chip system within an electronic device, the processing unit may be a processor. The processing unit executes the instructions stored in the storage unit to cause the electronic device to implement an audio processing method described in the first aspect or any possible implementation of the first aspect. The storage unit can be a storage unit within the chip (e.g., a register, cache, etc.) or a storage unit located outside the chip within the electronic device (e.g., a read-only memory, random access memory, etc.).

[0029] For example, a display unit is used to display images, videos, etc.

[0030] A processing unit is used to execute an audio processing method described in the first aspect or any possible implementation thereof.

[0031] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, the memory for storing code instructions, and the processor for running the code instructions to perform the methods described in the first aspect or any possible implementation of the first aspect.

[0032] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0033] Fifthly, embodiments of this application provide a computer program product including a computer program, which, when run on a computer, causes the computer to perform the methods described in the first aspect or any possible implementation thereof.

[0034] Sixthly, this application provides a chip or chip system including at least one processor and a communication interface. The communication interface and the at least one processor are interconnected via a circuit. The at least one processor is used to run computer programs or instructions to perform the methods described in the first aspect or any possible implementation of the first aspect. The communication interface in the chip can be an input / output interface, pins, or circuits, etc.

[0035] In one possible implementation, the chip or chip system described above in this application further includes at least one memory storing instructions. The memory can be an internal storage unit of the chip, such as a register or cache, or it can be a storage unit of the chip itself (e.g., read-only memory, random access memory, etc.).

[0036] It should be understood that the second to sixth aspects of this application correspond to the technical solutions of the first aspect of this application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation are similar, and will not be repeated here. Attached Figure Description

[0037] Figure 1 This is a schematic diagram of a vehicle-machine interconnection scenario provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0039] Figure 3 A schematic diagram of the software structure of an electronic device provided in an embodiment of this application;

[0040] Figure 4A A schematic diagram of the software structure of another electronic device provided in an embodiment of this application;

[0041] Figure 4B A schematic diagram of the software structure of another electronic device provided in an embodiment of this application;

[0042] Figure 5A schematic diagram illustrating a process for enabling audio virtualization capabilities of an electronic device, as provided in an embodiment of this application;

[0043] Figure 6 A flowchart illustrating an audio processing method provided in an embodiment of this application;

[0044] Figure 7A A schematic diagram of a packet assembly protocol provided for an embodiment of this application;

[0045] Figure 7B A schematic diagram of another packet assembly protocol provided in an embodiment of this application;

[0046] Figure 8 This is a schematic flowchart of another audio processing method provided in an embodiment of this application. Detailed Implementation

[0047] To facilitate a clear description of the technical solutions in the embodiments of this application, some terms and technologies involved in the embodiments of this application will be briefly introduced below:

[0048] 1. Audio virtualization: This refers to the process of implementing the audio function of one electronic device on another electronic device. For example, the audio function can be the function of playing audio data (referred to as audio).

[0049] In some embodiments, audio data may include music audio data, navigation audio data, call audio data, and notification audio data. Examples of notification audio data include alarm clocks, SMS reminders, and push notifications.

[0050] Taking the vehicle-machine interconnection scenario as an example, and taking the electronic device as a mobile phone, the mobile phone is equipped with a music application and has the function of playing music audio data (referred to as music or songs). Users can open the music application and play music audio data through the mobile phone. The mobile phone's speaker does not output the music audio data, but outputs the music audio data through the vehicle's speaker.

[0051] In this scenario, the function of playing music and audio data on a mobile phone is virtualized. Users can enjoy this function through the car's infotainment system, which can enrich the user's driving experience.

[0052] In some embodiments, the vehicle infotainment system can be referred to as the vehicle itself, and vehicle-to-vehicle connectivity refers to the connection between the vehicle infotainment system and a mobile phone.

[0053] 2. Binder Mechanism: This is a mechanism for inter-process communication. Different applications or application components (such as activity components or service components) run in different processes, and the Binder mechanism enables communication between these different processes.

[0054] The Binder mechanism uses interfaces (such as those defined by the Android Interface Definition Language (AIDL)) to allow developers to declare methods for cross-process communication in their code. These interfaces generate corresponding Binder classes during code compilation to implement cross-process communication.

[0055] In some embodiments, interfaces defined by the HAL interface definition language (HIDL) can also achieve inter-process communication through the Binder mechanism.

[0056] In some embodiments, the interface defined by AIDL can be called the AIDL interface, and the interface defined by HIDL can be called the HIDL interface.

[0057] In the embodiments of this application, reference is made to Figure 6 In electronic devices, the sub-thread of the audio HAL can send combined data packets of multiple audio data to the device virtualization service of the electronic device through binder calls, thereby reducing the number of interactions between device processes (or threads).

[0058] 3. Real-time Transport Protocol (RTP): This is a network transmission protocol that can be used by applications (APPs) to transmit multimedia data in single-point or multi-point scenarios. In this embodiment, multimedia data refers to audio data.

[0059] In this context, "single point" refers to transmission from one sender (e.g., application A) to one receiver (e.g., application B), while "multipoint" refers to transmission from one sender (e.g., application A) to at least two receivers (e.g., applications B and C), or transmission from at least two senders (e.g., applications A and B) to one receiver (e.g., application C).

[0060] RTP is widely used in various scenarios, such as video conferencing, online gaming, and real-time streaming. Taking video conferencing as an example, RTP can transmit audio and video data in real time, ensuring synchronization between them during a video conference.

[0061] 4. Other terms

[0062] In the embodiments of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order of execution. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or execution order, and that "first" and "second" do not necessarily imply that they are different.

[0063] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0064] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, and c can be single or multiple.

[0065] 5. Electronic equipment

[0066] The electronic devices in this application embodiment may include handheld devices with audio processing functions, vehicle-mounted devices, etc. For example, some electronic devices include: mobile phones, tablets, PDAs, laptops, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving vehicles, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, cellular phones, cordless phones, session initiation protocol (SIP) phones, wireless local loop (WLL) stations, personal digital assistants (PDAs), handheld devices with wireless communication capabilities, computing devices or other processing devices connected to a wireless modem, in-vehicle devices, wearable devices, terminal devices in 5G networks, or future evolution of public land mobile communication networks. Terminal devices in a network (PLMN), etc., are not limited to this in the embodiments of this application.

[0067] By way of example and not limitation, in this embodiment, the electronic device can also be a wearable device. Wearable devices, also known as wearable smart devices, are a general term for devices that utilize wearable technology to intelligently design and develop everyday wearables, such as glasses, gloves, watches, clothing, and shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothing or accessories. Wearable devices are not merely hardware devices, but also achieve powerful functions through software support, data interaction, and cloud interaction. Broadly speaking, wearable smart devices include those that are feature-rich, large in size, and can achieve complete or partial functions without relying on a smartphone, such as smartwatches or smart glasses, as well as those that focus on a specific type of application function and require the use of other devices such as smartphones, such as various smart bracelets and smart jewelry for vital sign monitoring.

[0068] Furthermore, in this embodiment of the application, the electronic device can also be a terminal device in the Internet of Things (IoT) system. IoT is an important part of the future development of information technology. Its main technical feature is to connect objects to the network through communication technology, thereby realizing an intelligent network of human-machine interconnection and object-to-object interconnection.

[0069] The electronic devices in the embodiments of this application may also be referred to as: terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent, or user device, etc.

[0070] In this embodiment, the electronic device or various network devices include a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on top of the operating system layer. The hardware layer includes hardware such as a central processing unit (CPU), a memory management unit (MMU), and memory (also called main memory). The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software.

[0071] In one possible scenario, a user might play music while driving using the car's infotainment system. For example, using a mobile phone as an example... Figure 1 This is a schematic diagram of a vehicle-machine interconnection scenario provided in an embodiment of this application.

[0072] Reference Figure 1 This scenario can include a mobile phone and an in-vehicle infotainment system. Both the mobile phone and the in-vehicle infotainment system are equipped with in-vehicle applications and can establish a communication connection. The communication connection can be, for example, a wireless fidelity (WiFi) connection, a Bluetooth connection, or a near field communication (NFC) connection. This application does not limit the method by which the mobile phone and the in-vehicle infotainment system establish a communication connection, and can refer to the description in the prior art.

[0073] When a user opens a music app on their phone and plays music audio data (e.g., playing music 100), such as Figure 1As shown in a, the mobile phone can display interface 10, which may include: the song title of music 100, the artist name of music 100, the lyrics of music 100, the playback progress bar of music 100, volume control control 101, playback mode control 102, control to switch to the previous song 103, play / pause control 104, control to switch to the next song 105, and music list control 106, etc.

[0074] When a communication connection is established between the mobile phone and the vehicle's infotainment system, and the user opens an in-vehicle application through the system, the vehicle's display screen can show, for example: Figure 1 The interface 11 shown in b is the interface corresponding to the vehicle system application. The interface 11 may include window 110 and window 111.

[0075] Window 110 may include: an icon 112 for returning to the main interface of the vehicle infotainment system, an icon 113 for the navigation application, an icon 114 for the music application, an icon 115 for the call application, and an icon 116 for the vehicle infotainment system application, etc.

[0076] Window 111 may include a sub-window 117 corresponding to a navigation application, a sub-window 118 corresponding to a music application, and a sub-window 119 corresponding to a call application. In some embodiments, window 111 may include one of the sub-windows shown in the example above.

[0077] The sub-window 117 corresponding to the navigation application may include: driving route information, such as "turn left at 50km to enter bridge A", time required to reach the destination, such as "33min", distance from the destination, such as "14.9km", time of arrival at the destination, such as "14:41pm", and end / start navigation controls, etc. It should be understood that the "End Navigation" option in sub-window 117 represents the end / start navigation control.

[0078] The sub-window 118 corresponding to the music application may include: a music list, such as "My Favorites", "Local Songs", and "Car Radio" lists, the song name of music 100, the artist name of music 100, a control 120 to switch to the previous song, a play / pause control 121, and a control 122 to switch to the next song, etc.

[0079] The sub-window 119 corresponding to the call application may include: an incoming call icon, an answer call icon, a hang-up call icon, and a mute icon.

[0080] In some embodiments, in-vehicle applications can also be described as smartphone-vehicle connectivity applications. In-vehicle applications can be understood as smartphone-vehicle connectivity products that integrate smartphone functionality into the vehicle, allowing users to access and enjoy various applications on their phones while in the vehicle. For example, users can access navigation, music, and calling applications on their phones through in-vehicle applications.

[0081] For example, in a music scenario, in response to a user click Figure 1 As shown in Figure a, the operation of the play / pause control 104 (such as playing music) allows the mobile phone to send a command to the vehicle system to play music 100 and the music 100. In response to the command to play music 100, the vehicle system can control the speakers on the vehicle system to play music 100.

[0082] Or, in response to user clicks Figure 1 As shown in b, the operation of the play / pause control 121 (such as playing music) allows the vehicle system to send a command to the mobile phone to play music 100. This command instructs the mobile phone to play music 100. If the mobile phone agrees to play music 100, it can send the command to play music 100 and the music 100 to the vehicle system. In response to the command to play music 100, the vehicle system can control the speakers on the vehicle system to play music 100.

[0083] For example, in a navigation scenario, when a user sets a navigation route in the navigation application on their mobile phone and clicks the end / start navigation control (such as start navigation), the mobile phone can send a command to the vehicle's infotainment system to play navigation audio data and the navigation audio data. In response to the command to play the navigation audio data, the vehicle's infotainment system can control the vehicle's speakers to play the navigation audio data.

[0084] Or, in response to user clicks Figure 1 The operation of the navigation application icon 113 shown in b, and in response to the user setting a navigation route in the vehicle's navigation application and clicking the start / end / navigation control (such as starting navigation), the vehicle's system can send a command to the mobile phone to play navigation audio data. This command is used to instruct the mobile phone to play navigation audio data. If the mobile phone agrees to play navigation audio data, the mobile phone can send the command to play navigation audio data and the navigation audio data to the vehicle's system. In response to the command to play navigation audio data, the vehicle's system can control the speakers on the vehicle's system to play the navigation audio data.

[0085] Similarly, taking a call scenario as an example, the car's infotainment system can control the speakers on the system to play call audio data.

[0086] Similarly, taking a notification scenario as an example, the vehicle's infotainment system can control the speakers on the system to play notification audio data.

[0087] In some embodiments, the mobile phone receives multiple audio data streams from different applications. The mobile phone can perform mixing, encoding, and other processing on the multiple audio data streams to obtain the audio data to be transmitted, and then send the audio data to be transmitted to the vehicle's infotainment system. Mixing refers to merging (or superimposing) multiple audio data streams into a single audio data stream.

[0088] In some embodiments, a mobile phone receives multiple audio data streams from different applications. When the mobile phone is processing these multiple audio data streams, different processes on the phone (such as sub-threads and device virtualization services in the following embodiments) send the multiple audio data streams at fixed packet transmission intervals. The packet transmission interval refers to the time interval between two transmissions of the multiple audio data streams. The multiple audio data streams may include: music audio data, navigation audio data, call audio data, and notification audio data. For example, the packet transmission interval can be 20 milliseconds (ms), meaning each audio data stream is sent once every 20 ms.

[0089] It should be understood that the sub-threads and device virtualization services in the following embodiments belong to different processes.

[0090] In this example, when the mobile phone processes multiple audio data (such as four audio data), the transmission of each audio data in the multiple audio data between different processes (such as the sub-thread and device virtualization service in the following embodiment) will involve a Binder mechanism call. With a packet transmission interval of 20ms, the mobile phone needs to make 200 Binder mechanism calls per second (50 calls per audio data per second, and 200 calls per four audio data), resulting in a high number of Binder mechanism calls.

[0091] However, each Binder mechanism call requires the central processing unit (CPU) to handle the request, which includes serialization and deserialization of multiple audio data streams, process switching (such as switching between sub-threads and device virtualization services), etc. When the number of Binder mechanism calls is high (e.g., 200 times per second), it consumes a lot of the phone's CPU resources.

[0092] In addition, since the resources of a mobile phone's CPU are limited, and there are other tasks in the phone besides audio data transmission, when audio data transmission and other tasks run in parallel, multiple tasks (such as audio data transmission and other tasks) will compete for the phone's CPU resources, reducing the speed and stability of the phone's audio data transmission, resulting in stuttering when the car's system plays audio data.

[0093] When the phone's CPU is under high load (such as when the phone's CPU is handling many tasks), users can clearly perceive stuttering in each audio data stream, resulting in a poor user experience.

[0094] In the above embodiments, each audio data stream in the multi-channel audio data is sent between different processes of the mobile phone (such as sub-threads and audio device virtualization in the following embodiments), which generates a Binder mechanism call each time. This results in a high number of Binder mechanism calls, reducing the transmission speed and stability of audio data by the mobile phone. For in-vehicle systems, this causes stuttering in the received audio data, affecting user experience and driving safety.

[0095] Accordingly, this application provides an audio processing method, an electronic device, a storage medium, and a chip system. In this method, before sending multiple audio data to the vehicle's infotainment system, the electronic device can perform packet processing on the multiple audio data to obtain a total data packet. Sending the total data packet between different processes or internal modules of the electronic device will only generate one call to the Binder mechanism, which can reduce the number of times the electronic device calls the Binder mechanism internally. This can improve the transmission speed and stability of the electronic device for multiple audio data, effectively avoid stuttering when the other end device (vehicle infotainment system) plays audio data, and improve the user's auditory experience and driving safety.

[0096] Before introducing the audio processing method provided in the embodiments of this application, the structure of the electronic device provided in the embodiments of this application will be first introduced:

[0097] Figure 2 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. (Refer to...) Figure 2 The electronic device 200 may include: a processor 210, an audio module 211, a speaker 211A, a receiver 211B, a microphone 211C, a headphone jack 211D, a sensor module 212, a display screen 213, an internal memory 214, a wireless communication module 215, and an antenna 1, etc. It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 200.

[0098] In this embodiment of the application, the processor 210 can be used to execute the audio processing method provided in this embodiment of the application.

[0099] The speaker 211A can be used to play audio data.

[0100] Microphone 211C can be used for user input of audio data. For example, in a call scenario, and using a mobile phone as an electronic device, the user can input call audio data through microphone 211C.

[0101] The sensor module 212 may include sensors such as a pressure sensor 212A and a touch sensor 212B. The touch sensor may be mounted on the display screen 213, forming a touchscreen. The touch sensor is used to receive user trigger operations on the touchscreen.

[0102] The display screen 213 can be used to display images, videos, etc. The display screen 213 includes a display panel. For example, the display screen 213 can realize [something related to displaying images, videos, etc.]. Figure 1 The display of the interface shown in 'a' is not limited in this embodiment.

[0103] The internal memory 214 can be used to store relevant instructions so that the audio processing method provided in the embodiments of this application can be implemented when the processor 210 calls the relevant instructions.

[0104] The wireless communication module 215 can provide functions such as wireless local area networks (WLAN) (such as Wi-Fi, Bluetooth, or NFC) for use in electronic devices.

[0105] In other embodiments of this application, the electronic device 200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0106] The software system of an electronic device can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application uses a layered architecture as an example to illustrate the software structure of an electronic device. Figure 3 This is a schematic diagram of the software structure of an electronic device provided in an embodiment of this application. The layered architecture divides the software system of the electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, refer to... Figure 3 Electronic devices may include: application layer (APP), application framework layer, hardware abstraction layer (HAL), and kernel layer.

[0107] It is understood that the software structure division of the electronic device in this application embodiment is just an example. This application embodiment does not limit the hierarchical division method in the software system of the electronic device. The modules in each level in the following embodiments are the modules involved in the embodiments of this application. Each level may also include more or fewer modules than shown in the figure, or combine some modules, or split some modules.

[0108] The application layer may include a series of application packages, and runs the application by calling the application programming interface (API) provided by the application framework layer. In this embodiment, reference is made to... Figure 3 The application layer can include music applications, navigation applications, calling applications, and in-vehicle infotainment applications.

[0109] Among them, music apps support playing music audio data, navigation apps support playing navigation audio data, call apps support playing call audio data, and notification apps support playing notification audio data.

[0110] In-vehicle infotainment applications, used to enable interconnection between electronic devices and the vehicle's infotainment system, can be referenced as described above. Figure 1 The description in the text.

[0111] In some embodiments, the application layer may further include a notification application, which may include, but is not limited to, at least one of the following: music applications, navigation applications, calling applications, game applications, and social applications. The notification application may push messages to the user and support playing notification audio data when pushing messages.

[0112] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer may include some predefined functions. In this embodiment, reference is made to... Figure 3 The application framework layer may include: an audio management module and a device virtualization service.

[0113] The audio management module can be used to receive audio data from multiple applications (such as music audio data from a music application, navigation audio data from a navigation application, call audio data from a call application, and notification audio data from a notification application). For example, Figure 4A The audio data input represents audio data from multiple applications.

[0114] In some embodiments, the audio management module can split audio data from multiple applications based on a splitting strategy to obtain multi-stream data. The splitting strategy can be a pre-configured splitting strategy in the electronic device or a splitting strategy from other devices (e.g., an in-vehicle system), and this embodiment of the application does not limit the specific implementation of the strategy.

[0115] For example, refer to Figure 4A The audio management module can divide audio data from multiple applications into four streams based on a splitting strategy and send them to the audio HAL in the following embodiments. The following embodiments use four streams as an example.

[0116] It should be understood that different streams of data can correspond to different channels. For example, stream 1 can correspond to channel 4, stream 2 to channel 3, stream 3 to channel 2, and stream 4 to channel 1. By assigning different streams of data to different channels, parallel processing of multiple streams of data can be achieved.

[0117] Device virtualization services include audio virtualization services (or audio virtualization capabilities), which can be used to process and transmit audio-related data. Audio-related data may include the total data packets described in the following embodiments.

[0118] In some embodiments, the device virtualization service may also include other virtualization services, such as image virtualization services and video virtualization services.

[0119] In some embodiments, the device virtualization service can split the total data packet into multiple sub-data packets.

[0120] In some embodiments, an encoder can be configured in the device virtualization service, which can use the encoder to encode the total data packet (or multiple sub-data packets).

[0121] In some embodiments, the device virtualization service may be referred to as the distributed mobile sensing development platform (DMSDP).

[0122] It should be noted that the aforementioned device virtualization service is not limited to the application framework layer. In some embodiments, the device virtualization service may also reside in other layers. For example, the device virtualization service may reside in the application layer or the hardware abstraction layer.

[0123] In some embodiments, the application framework layer may further include one or more of the following: a display composition system, a window manager, a content provider, a resource manager, a view system, or a notification manager, etc. Figure 3 (not shown in the text), but this is not limited in the embodiments of this application.

[0124] The purpose of the hardware abstraction layer is to abstract hardware, providing a unified interface for upper-layer applications to query hardware devices, or to provide data storage services for upper-layer applications. In this embodiment, reference is made to... Figure 3 The hardware abstraction layer includes: Audio HAL.

[0125] Reference Figure 3The audio HAL can include a main thread and child threads. The main thread can be used to initialize (or create) child threads and cache multi-stream data. Initializing (or creating) child threads includes allocating (or requesting) a buffer for the child thread; the buffer is used to cache multi-stream data from the audio HAL. The child threads are used to perform packet merging of the multi-stream data, and can also be used to send merged data to the device virtualization service via the Binder mechanism. For example, refer to... Figure 4A The child thread can combine multiple streams of data, such as stream 1, stream 2, stream 3, and stream 4.

[0126] In some embodiments, a timer can be configured in the sub-thread, and the duration of the timer is used to indicate the packet sending interval for sending combined data.

[0127] The kernel layer is the layer between hardware and software. It drives the hardware, enabling it to function. The kernel layer may include, but is not limited to, one or more of the following: sensor drivers, display drivers, and Bluetooth drivers.

[0128] In one possible approach, the electronic device may include a hardware layer, which may include one or more of the following: a camera, a liquid crystal display (LCD), a graphics processing unit (GPU), or a central processing unit (CPU), etc. Figure 3 (Not shown in the image).

[0129] In this application embodiment, no specific limitations are made on the software layers involved in the software architecture, the modules contained in the layers, and the functions of the modules.

[0130] The software structure of the electronic device provided in the embodiments of this application has been described in the example above. The following describes the preparations made to implement the audio processing method provided in this application.

[0131] Figure 5 This is a schematic flowchart illustrating an embodiment of this application that enables audio virtualization capabilities in electronic devices. Figure 5 In the corresponding embodiments, the process of enabling the audio virtualization capability of electronic devices involves both electronic devices and vehicle systems. Taking a mobile phone as an example, the mobile phone may include: an audio management module, a vehicle system application, a device virtualization service, and an audio HAL. The vehicle system may include: a device virtualization service and a vehicle system application.

[0132] Reference Figure 5 The process for enabling audio virtualization capabilities of electronic devices provided in this application embodiment may include:

[0133] S501, in response to the user-triggered connection operation, the vehicle application on the mobile phone initializes the device virtualization service of the mobile phone.

[0134] Device virtualization services may include, but are not limited to, audio virtualization capabilities.

[0135] In some embodiments, the connection operation can be an action triggered (clicked) by a user on a mobile phone interface, such as triggering the Bluetooth control identifying the vehicle's infotainment system in the phone's settings application. In response to the action of triggering the Bluetooth control identifying the vehicle's infotainment system, the phone's infotainment system application initializes the phone's device virtualization service.

[0136] In some embodiments, a connection operation may be referred to as a device connection event. A device connection event may be a device connection event acting on a first application of the mobile phone (such as a settings application), or a device connection event initiated from the vehicle's infotainment system (as described in the embodiments below).

[0137] S502, the car infotainment application on the phone can check whether the phone's device virtualization service supports audio virtualization capabilities.

[0138] In some embodiments, audio virtualization capabilities include audio streaming capabilities. Audio streaming capabilities refer to the ability to transmit audio data.

[0139] S503, the phone's device virtualization service queries whether the phone's main thread supports audio splitting.

[0140] Audio offloading refers to the process of dividing audio data from multiple applications into multiple streams based on an offloading strategy. The offloading strategy can be described in the following embodiments.

[0141] S504: The phone's main thread sends support information to the phone's device virtualization service.

[0142] Correspondingly, the phone's device virtualization service can receive support information from the main thread. This support information may include: support for audio streaming.

[0143] In some embodiments, when the main thread of the phone does not support audio streaming, the main thread of the phone can send a message of non-support to the phone's device virtualization service. The non-support message may include: audio streaming is not supported.

[0144] S505 establishes a connection between the mobile phone's device virtualization service and the vehicle's device virtualization service.

[0145] In some embodiments, prior to S505, in response to a user-triggered connection operation, the vehicle's infotainment system application can initialize the device virtualization service of the infotainment system. The description of the vehicle's infotainment system application initializing the device virtualization service can be referenced to the description of the mobile phone's infotainment system application initializing the mobile phone's device virtualization service.

[0146] S506, the mobile phone's device virtualization service sends the mobile phone's device information to the vehicle's device virtualization service.

[0147] Correspondingly, the vehicle's virtualization service can receive device information from the mobile phone. This device information may include: support status.

[0148] S507, the vehicle's device virtualization service sends the vehicle's device information to the mobile phone's device virtualization service.

[0149] Correspondingly, the mobile phone's device virtualization service can receive device information from the vehicle's infotainment system. This device information may include traffic offloading strategies.

[0150] Stream splitting strategy refers to dividing multi-stream data into different streams based on audio data type. Each stream can correspond to at least one audio data type. Audio data types can include: music audio data, navigation audio data, call audio data, and notification audio data, etc.

[0151] In some embodiments, audio data from different streams obtained based on a splitting strategy can be referred to as streaming data.

[0152] For example, consider a streaming strategy where multiple audio data streams are divided into four streams: a music stream, a navigation stream, a call stream, and a notification audio stream. The audio data in the music stream originates from a music application, the audio data in the navigation stream originates from a navigation application, the audio data in the call stream originates from a call application, and the audio data in the notification audio stream originates from a notification application.

[0153] For example, consider a streaming strategy where multiple audio data streams are divided into three streams: a music stream, a navigation stream, and a mixing stream. The audio data in the music stream originates from a music application, the audio data in the navigation stream originates from a navigation application, and the audio data in the mixing stream originates from a call application and a notification application.

[0154] It should be understood that, in the embodiments of this application, the traffic splitting strategies are not exhaustively listed one by one.

[0155] In some embodiments, the in-vehicle device virtualization service can configure different splitting strategies based on business needs, dividing multiple audio data into no more than three or four streams.

[0156] In some embodiments, the splitting strategy may be referred to as the audio data type corresponding to the preset stream type.

[0157] S508, the phone's device virtualization service sends audio capability information to the phone's in-vehicle application.

[0158] The audio capability information may include: whether audio virtualization capability is supported, whether audio splitting is supported, and the device information of the vehicle system.

[0159] Whether audio splitting is supported can be categorized into supported and unsupported cases.

[0160] S509, the car infotainment system application of the mobile phone determines whether both the mobile phone and the car infotainment system support audio splitting based on audio capability information.

[0161] Since the audio capability information includes whether audio splitting is supported, the car's infotainment system can determine whether the phone supports audio splitting based on this information.

[0162] It should be understood that since the device virtualization service is included in the mobile phone, the device virtualization service's support for audio offloading can be understood as the mobile phone supporting audio offloading.

[0163] In some embodiments, the mobile phone's vehicle infotainment application can confirm whether the vehicle infotainment system supports audio splitting based on the device information (such as splitting strategy) in the audio capability information. For example, when the splitting strategy is to divide the data into at least two streams (such as two streams, three streams, and four streams), the mobile phone's vehicle infotainment application can confirm that the vehicle infotainment system supports audio splitting.

[0164] With S510, when both the mobile phone and the vehicle system support audio splitting, the mobile phone's vehicle system application configures the splitting strategy to the audio management module.

[0165] When a splitting strategy is configured in the audio management module, the audio management module can split multiple audio data streams based on the splitting strategy.

[0166] S511 enables audio virtualization capabilities for device virtualization services through mobile phone in-vehicle applications.

[0167] Here, "enabling" can be understood as: granting capabilities. For example, if the audio virtualization capability of the device virtualization service is not enabled in the mobile phone's in-vehicle application, the device virtualization service cannot use the audio virtualization capability.

[0168] S512, the device virtualization service initialization service of the mobile phone sends packets to the main thread at intervals and sets packet merge flags.

[0169] Correspondingly, the main thread can receive the packet sending interval and packet merging flag from the device virtualization service. The packet merging flag is used to instruct the child thread to perform packet merging processing.

[0170] In some embodiments, the packet transmission interval can be pre-configured in the device virtualization service of the mobile phone. For example, the packet transmission interval can be 40ms.

[0171] S513, the main thread initializes (or creates) a child thread.

[0172] The main thread initializing (or creating) a child thread may include: allocating (or creating) resources for the child thread, and configuring relevant information for the child thread to run.

[0173] The resource can be a first buffer, which is used to store multiple stream data (such as multiple first stream data or multiple second stream data in the embodiments below). The first buffer may contain multiple buffer queues, and each buffer queue can store the corresponding stream data. For example, refer to... Figure 4A , Figure 4A Taking the first buffer area as an example, which contains four buffer queues, each buffer queue stores stream data 1, stream data 2, stream data 3, and stream data 4 respectively.

[0174] The relevant information may include, but is not limited to, the duration of the timer (or the packet sending interval).

[0175] In some embodiments, since the main thread is contained within the phone, the initialization (or creation) of a child thread by the main thread can be referred to as the initialization (or creation) of a child thread by the phone, and the allocation (or creation) of a first buffer by the main thread can be referred to as the allocation (or creation) of a first buffer by the phone.

[0176] In this embodiment, when a connection is established between the mobile phone and the vehicle's infotainment system, the two devices can exchange their respective device information. Furthermore, the mobile phone's in-vehicle application can determine whether both the mobile phone and the vehicle's infotainment system support audio splitting based on audio capability information. If both the mobile phone and the vehicle's infotainment system support audio splitting, the mobile phone can configure a splitting strategy in the audio management module. Thus, during subsequent audio processing, the audio management module can perform audio splitting based on the splitting strategy.

[0177] The above example illustrates the preparations made to implement the audio processing method of this application. The following description, in conjunction with specific embodiments, illustrates the audio processing method provided by the embodiments of this application. These embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0178] Figure 6 This is a schematic flowchart illustrating another audio processing method provided in an embodiment of this application. (Refer to...) Figure 6The audio processing method provided in this application embodiment may include:

[0179] S601, the audio management module obtains audio data from multiple applications.

[0180] Once a connection is established between the mobile phone and the vehicle's infotainment system, and both the phone and the system support audio splitting, the audio management module can acquire audio data from multiple applications. These applications may include, but are not limited to, music applications, navigation applications, call applications, and notification applications.

[0181] In some embodiments, audio data from multiple applications may be referred to as the first audio data to be played by the multiple applications.

[0182] S602, the audio management module sends multi-stream data to the main thread based on the splitting strategy and audio data from multiple applications.

[0183] Correspondingly, the main thread can receive multi-stream data from the audio management module.

[0184] The streaming strategy can be referred to the description of the streaming strategy in S507 above. For example, taking the streaming strategy of dividing the data into four streams as an example, the audio management module can divide the audio data from multiple applications into four streams (multi-stream data) based on the streaming strategy, and then the audio management module can send the four streams to the main thread.

[0185] S603, the main thread caches multiple stream data.

[0186] Since each stream in a multi-stream dataset has a different data type, the main thread can use a suitable data structure to cache the corresponding stream data to ensure the integrity of each stream. This data structure can include, but is not limited to, any of the following: Map, Queue, Array, and Linked List.

[0187] Taking the main thread using a Map data structure to cache multiple streams of data as an example, each stream corresponds to a Map. In each Map, the key serves as the identifier for that stream, and the value stores that stream data. Since there is a one-to-one correspondence between the key and the value, in the main thread, the corresponding value can be retrieved by querying the key.

[0188] In some embodiments, the main thread can cache multiple stream data into a first buffer.

[0189] S604, the main thread starts the child thread.

[0190] S605, the child thread retrieves multi-stream data from the cache.

[0191] S606: The sub-thread performs packet merging processing on multiple stream data based on a preset packet merging protocol to obtain a total data packet.

[0192] The preset packet assembly protocols may include packet assembly protocol 1 and packet assembly protocol 2.

[0193] Figure 7A This is a schematic diagram of a packet assembly protocol provided in an embodiment of this application, with reference to... Figure 7A Packet assembly protocol 1 can include, in sequence, packet header information (info), data type (dataType), and data (data).

[0194] The packet header information (info) can include, in sequence: peer device identifier (deviceId), service identifier (serviceId), session identifier (sessionId), and data header (head). Among them, the peer device identifier is used to indicate the peer device, such as the vehicle's infotainment system; the service identifier is used to indicate the corresponding service, which can be a microphone or speaker; and the session identifier is used to support the session connecting the mobile phone and the vehicle's infotainment system.

[0195] The data type (dataType) indicates the data transmission format. Since child threads can transmit data via either the AIDL interface or the HIDL interface, and different interfaces support different transmission formats, this embodiment uses the AIDL interface. Figure 7A The example used is a data type DATA_TYPE_BYTE.

[0196] Data refers to streaming data.

[0197] The sub-thread can perform packet combining on each stream of data separately based on packet assembly protocol 1 to obtain the data packet corresponding to each stream. For example, taking four streams of data as an example, the data packet corresponding to each stream can be data packet 1, data packet 2, data packet 3, and data packet 4.

[0198] Figure 7B A schematic diagram of another packetization protocol provided in this application embodiment is shown below. Figure 7B Packet assembly protocol 2 may include: packet header information (info), data type (dataType), and data.

[0199] and Figure 7A The difference is that the packet header does not include a data header, and the packet header also includes a merge flag and the total length of the data packet.

[0200] In some embodiments, the packet merging flag may be referred to as the first flag, which is used to indicate that the total data packet is merged data.

[0201] The data can include, in sequence, a data subheader and the data itself.

[0202] The data header can include, in sequence, the stream type (type), length, and audio type (reserved[2]). The stream type can include music stream, navigation stream, call stream, and notification audio stream, etc. The audio type (reserved[2]) is used to store usage and version number. Usage is used to identify the type of audio data, and the version number can be used to distinguish different interfaces, such as AIDL interface and HIDL interface.

[0203] In some embodiments, the sub-thread can combine the data packets corresponding to each stream of data according to packet combining protocol 2 to obtain a total data packet. For example, the sub-thread can combine data packets 1, 2, 3, and 4 according to packet combining protocol 2 to obtain a single total data packet. The packet combining process may include steps such as adding packet header information and setting the data transmission format.

[0204] In some embodiments, the total data packet may include, in sequence, a first field, a second field, and a third field. Wherein, refer to Figure 7B The first field records the packet header information (info), the second field records the data type (dataType), and the third field records the data (data). The first field is used to indicate the general information of the multi-stream data in the total data packet, the second field is used to indicate the transmission format of the total data packet, and the third field is used to indicate the multi-stream data in the total data packet, as well as the attribute information of each stream in the multi-stream data.

[0205] S607, the child thread sends the total data packet to the phone's device virtualization service.

[0206] Correspondingly, the phone's device virtualization service can receive the total data packet from the child thread.

[0207] In this embodiment, since the child thread performs packet merging on the multi-stream data, the child thread can send a single total data packet to the device virtualization service of the mobile phone without making multiple calls to the Binder mechanism.

[0208] S608, the child thread deletes the multi-stream data in the cache.

[0209] Since the cache space is limited and multi-stream data will occupy the cache space, after the child thread sends the total data packet to the phone's device virtualization service, the child thread can delete the multi-stream data in the cache and release the cache space.

[0210] S609, the mobile phone's device virtualization service splits the total data packet into multiple sub-data packets.

[0211] In some embodiments, the device virtualization service of the mobile phone can split the total data packet based on a preset packet assembly protocol (such as packet assembly protocol 2) to obtain multiple sub-data packets.

[0212] For example, refer to Figure 4B The term "packet splitting" refers to splitting the total data packet into multiple sub-data packets. These sub-data packets can be, for example, sub-data packet 1, sub-data packet 2, sub-data packet 3, and sub-data packet 4.

[0213] It should be understood that sub-data packet 1, sub-data packet 2, sub-data packet 3, and sub-data packet 4 can be: data packets 1, 2, 3, and 4 obtained by the sub-thread based on the packet assembly protocol 1 in the above embodiment.

[0214] In some embodiments, due to network bandwidth limitations, such as the network not supporting the transmission of large sub-data packets, the device virtualization service can further split the sub-data packet before transmission to accommodate the network's transmission capacity in order to ensure smooth transmission. Here, "large data packet" can be understood as the sub-data packet exceeding the network's bandwidth threshold.

[0215] In addition, during network transmission, sub-data packets may be lost due to network failures or abnormal shutdown of mobile phones. To reduce the risk of sub-data packet loss, the device virtualization service can continue to split the sub-data packets to reduce the risk of loss.

[0216] In S610, the mobile phone's device virtualization service encodes multiple sub-data packets to obtain multiple encoded sub-data packets.

[0217] In some embodiments, to ensure the security of each sub-data packet during transmission, the device virtualization service can encode multiple sub-data packets separately to obtain multiple encoded sub-data packets, and transmit the multi-encoded sub-data packets based on the RTP protocol.

[0218] For example, refer to Figure 4BTaking the device virtualization service splitting the total data packet into four sub-data packets as an example, the device virtualization service can encode multiple (e.g., four) sub-data packets using four encoders, such as encoder 1, encoder 2, encoder 3, and encoder 4. Each encoder is responsible for encoding its corresponding sub-data packet. After encoding, each sub-data packet will produce corresponding encoded sub-data packets, such as output 1, output 2, output 3, and output 4. Since there are four encoded sub-data packets, therefore... Figure 4B The document shows four RTP transmission channels, namely RTP1, RTP2, RTP3, and RTP4, each of which can transmit the corresponding encoded sub-data packets.

[0219] S611, the mobile phone's device virtualization service sends multiple encoded sub-data packets to the vehicle's device virtualization service.

[0220] Correspondingly, the vehicle's device virtualization service can receive multiple encoded sub-data packets.

[0221] In some embodiments, the mobile phone's device virtualization service can simultaneously send multiple encoded sub-data packets to the vehicle's device virtualization service.

[0222] S612, the vehicle's device virtualization service decodes multiple encoded sub-data packets to obtain multiple sub-data packets.

[0223] Reference Figure 3 As described in the text, since the device virtualization service of the vehicle system can be configured with a decoder, the device virtualization service of the vehicle system can use the decoder to decode the multiple encoded sub-data packets to obtain multiple sub-data packets.

[0224] For example, the vehicle-mounted system's device virtualization service can use four decoders to decode the four encoded sub-data packets respectively, resulting in four decoded sub-data packets.

[0225] S613, the vehicle's device virtualization service sends multiple sub-data packets to the vehicle's application.

[0226] Correspondingly, the vehicle-mounted application can receive multiple sub-data packets.

[0227] S614, the vehicle infotainment system's application controls the speaker to play multiple sub-data packets.

[0228] In some embodiments, after S614, the method further includes:

[0229] S615, the audio management module obtains audio data from multiple applications.

[0230] In some embodiments, audio data from multiple applications may be referred to as second audio data to be played by multiple applications.

[0231] The S616 audio management module sends multi-stream data to the main thread based on the splitting strategy and audio data from multiple applications.

[0232] In some embodiments, multiplexed stream data may be referred to as multiplexed second stream data.

[0233] S617, main thread caches multiple stream data.

[0234] S618, the child thread retrieves multi-stream data from the cache.

[0235] S619: The sub-thread performs packet merging processing on multiple stream data based on a preset packet merging protocol to obtain a total data packet.

[0236] In some embodiments, the total data packet may be referred to as the second data packet.

[0237] S620, the child thread sends the total data packet to the phone's device virtualization service.

[0238] S621, the child thread deletes the multi-stream data in the cache.

[0239] S622, the mobile phone's device virtualization service splits the total data packet into multiple sub-data packets.

[0240] S623, the mobile phone's device virtualization service encodes multiple sub-data packets to obtain multiple encoded sub-data packets.

[0241] S624, the mobile phone's device virtualization service sends multiple encoded sub-data packets to the vehicle's device virtualization service.

[0242] S625, the vehicle's device virtualization service decodes multiple encoded sub-data packets to obtain multiple sub-data packets.

[0243] S626, the vehicle's device virtualization service sends multiple sub-data packets to the vehicle's application.

[0244] S627, the vehicle infotainment system's application controls the speaker to play multiple sub-data packets.

[0245] In this embodiment, S615-S617 can refer to the description in S601-S603 above, and S618-S627 can refer to the description in S605-S614 above.

[0246] In this embodiment, the mobile phone's audio management module can split audio data from multiple applications based on a splitting strategy to obtain multi-stream data. The mobile phone's audio management module can then send this multi-stream data to the mobile phone's audio HAL. When the mobile phone's audio HAL receives the multi-stream data from the audio management module, a sub-thread within the audio HAL can combine the multi-stream data according to a preset packet assembly protocol to obtain a total data packet. The sub-thread within the mobile phone's audio HAL can send this total data packet to the mobile phone's device virtualization service through a single Binder mechanism call. In response to the total data packet, the mobile phone's device virtualization service can split the total data packet into multiple sub-data packets. The mobile phone's device virtualization service can then send these multiple sub-data packets to the vehicle's infotainment system, enabling the system to play them. In this method, the sub-thread within the mobile phone's audio HAL sends the total data packet to the mobile phone's device virtualization service without requiring multiple Binder mechanism calls. Furthermore, because the mobile phone's device virtualization service can split the total data packet into multiple sub-data packets, stuttering during audio data transmission to the vehicle's infotainment system can be avoided, thus improving the user experience.

[0247] The above example illustrates the audio processing method provided by this application from the perspective of the interaction between various modules in a mobile phone and a vehicle system. The following embodiments illustrate the audio processing method provided by this application from the perspective of an electronic device (hereinafter referred to as a device).

[0248] Figure 8 A flowchart illustrating another audio processing method provided in an embodiment of this application. (Refer to...) Figure 8 The audio processing method provided in this application embodiment may include:

[0249] S801, when a communication connection is established between the first device and the second device, and multiple applications of the first device have first audio data to be played, the first device splits the first audio data to be played by the multiple applications into multiple first stream data.

[0250] Communication connections may include, but are not limited to: WiFi connection, Bluetooth connection, and NFC connection.

[0251] The first device and the second device are different electronic devices. For example, the first device can be a mobile phone, and the second device can be a vehicle infotainment system, a tablet computer, a conference screen, or a projector.

[0252] The initial audio data to be played from multiple applications may include: music audio data, navigation audio data, call audio data, and notification audio data.

[0253] In some embodiments, the first device may split the first audio data to be played from multiple applications into multiple first stream data based on a splitting strategy, and the multiple first stream data can be "multi-stream data" in S602.

[0254] S802, after obtaining multiple first-stream data, the first device starts the first thread, which is used to merge the multiple first-stream data into the first data packet.

[0255] The first thread can be a sub-thread, and the first data packet can be the total data packet in S606.

[0256] In some embodiments, after the first device starts the first thread, the first thread can merge each of the first stream data in the multiple first stream data into the first data packet, as described in S606.

[0257] S803, the first device splits the first data packet into multiple sub-data packets to be transmitted.

[0258] S803 can be referred to the description in S609.

[0259] S804, the first device sends multiplexed subdata packets to the second device.

[0260] Correspondingly, the second device can receive multiple sub-data packets from the first device.

[0261] In some embodiments, after receiving multiple sub-data packets, the second device can play multiple sub-data packets.

[0262] The embodiments of this application have the same technical effects as the embodiments described above, and can be referred to the descriptions in the embodiments above, which will not be repeated here.

[0263] In some embodiments, the audio processing method provided in this application can be applied to scenarios other than vehicle-to-machine (vehicle-to-phone) interconnection. For example, it can be applied to phone-to-tablet interconnection, phone-to-display interconnection, and tablet-to-display interconnection scenarios, etc., and this application does not limit it in this regard.

[0264] It should be noted that the module names involved in the embodiments of this application can all be defined as other names, as long as they can achieve the function of each module, and no specific restrictions are placed on the module names.

[0265] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0266] The audio processing method according to the embodiments of this application has been described above. The apparatus for performing the above method provided in the embodiments of this application is described below. Those skilled in the art will understand that the methods and apparatus can be combined with and referenced by each other, and the related apparatus provided in the embodiments of this application can perform the steps in the above list sorting method.

[0267] The audio processing method provided in this application can be applied to electronic devices with communication functions. Electronic devices include terminal devices, and the specific device form of the terminal device can be referred to the above-mentioned descriptions, which will not be repeated here.

[0268] This application provides a terminal device, which includes a processor and a memory; the memory stores computer execution instructions; the processor executes the computer execution instructions stored in the memory, causing the terminal device to perform the above-described method.

[0269] This application provides a chip. The chip includes a processor, which is used to call a computer program in memory to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0270] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0271] In one possible implementation, a computer-readable medium may include RAM, ROM, compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage or other magnetic storage devices, or any other medium targeted to carry or to store the required program code in the form of instructions or data structures, and accessible by a computer. Furthermore, any connection is appropriately referred to as a computer-readable medium. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. As used herein, disks and optical discs include optical discs, laser discs, optical discs, Digital Versatile Discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0272] This application provides a computer program product, which includes a computer program that, when run, causes a computer to perform the above-described method.

[0273] This application describes embodiments of methods, apparatus (systems), and computer program products according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0274] The above specific embodiments further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. An audio processing method, characterized in that, Applied to a first device, wherein the first device establishes a communication connection with a second device, the method includes: When multiple applications of the first device have first audio data to be played, the first device splits the first audio data to be played by the multiple applications into multiple first stream data. After receiving the multiple first stream data, the first device starts a first thread, which is used to merge the multiple first stream data into a first data packet; The first device splits the first data packet into multiple sub-data packets to be transmitted; The first device sends the multi-path sub-data packets to the second device.

2. The method according to claim 1, characterized in that, Before the first device starts the first thread, the method further includes: the first device receiving a device connection event and creating the first thread; The creation of the first thread includes: configuring a timer for the first thread; the duration of the timer is used to indicate the interval for sending data packets.

3. The method according to claim 2, characterized in that, The method further includes: During the process of the first device creating the first thread, the first device creates a first buffer, which is used to store multi-stream data.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: After the first thread of the first device merges the multiple first stream data into the first data packet, the first device deletes the multiple first stream data from the first buffer.

5. The method according to any one of claims 1 to 4, characterized in that, The first data packet includes, in sequence: The data consists of a first field, a second field, and a third field; the first field indicates general information about the multiple first stream data in the first data packet, the second field indicates the transmission format of the first data packet, and the third field indicates the multiple first stream data in the first data packet, as well as the attribute information of each first stream data.

6. The method according to claim 5, characterized in that, The common information of the multiple first-stream data includes: the identifier of the second device, the service identifier, the session identifier, the first flag bit, and the length of the first data packet; the first flag bit is used to indicate that the first data packet is a combined data packet. The attribute information of each first stream of data includes the stream type, length, and audio type of the first stream data.

7. The method according to any one of claims 1 to 6, characterized in that, The first device includes an audio management module, which performs audio splitting on the first audio data to be played from the multiple applications to obtain multiple first stream data, including: The audio management module of the first device receives first audio data to be played from the plurality of applications; The audio management module performs audio splitting on the first audio data to be played from the multiple applications according to the audio data type corresponding to the preset stream type, so as to obtain the multiple first stream data. Each first stream of data includes first audio data to be played from at least one application.

8. The method according to claim 7, characterized in that, The first device further includes an Audio Hardware Abstraction Layer (HAL), and the method further includes: The audio management module sends the multi-channel first-stream data to the audio HAL; The first device starts the first thread, including: After receiving the multiple first stream data, the audio HAL starts the first thread.

9. The method according to any one of claims 1 to 8, characterized in that, The first device includes a device virtualization service, and the method further includes: The first thread sends the first data packet to the device virtualization service via a Binder call; The first device splits the first data packet into multiple sub-data packets to be transmitted, including: After receiving the first data packet, the device virtualization service of the first device splits and encodes the first data packet to obtain multiple sub-data packets to be transmitted. The first device sends the multi-path sub-data packets to the second device, including: The first device sends the multi-path sub-data packets to the second device through the device virtualization service.

10. The method according to claim 9, characterized in that, After the first thread sends the first data packet to the device virtualization service via a Binder call, the method further includes: The first thread starts a timer; When the timer expires, the first thread sends a second data packet to the device virtualization service via a Binder call; wherein the second data packet is a data packet that merges multiple second stream data, and the multiple second stream data are multiple stream data corresponding to the second audio data to be played by multiple applications of the first device.

11. An electronic device, characterized in that, The electronic device includes: one or more processors and memory; The memory is coupled to the one or more processors, the memory being used to store computer program code, the computer program code including computer instructions, the one or more processors invoking the computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

12. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system including one or more processors, the one or more processors being used to invoke computer instructions to cause the electronic device to perform the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes computer program code that, when run on an electronic device, causes the electronic device to perform the method as described in any one of claims 1 to 10.

Images