Multimedia data processing method and device, electronic equipment and program product

Abstract

The embodiment of the invention discloses a multimedia data processing method and device, electronic equipment and a program product. The method is applied to the electronic equipment, the electronic equipment comprises a Bluetooth host and a Bluetooth controller, and the method comprises the following steps: the Bluetooth host obtains a multimedia data packet to be transmitted and stores the multimedia data packet in a multimedia buffer area; the Bluetooth host determines and discards one or more first multimedia data packets according to the generation moments corresponding to the multimedia data packets stored in the multimedia buffer area under the condition that the data discarding condition is met; a first duration from a generation moment corresponding to the first multimedia data packet to a current moment is greater than or equal to a duration threshold; and the Bluetooth host transmits the second multimedia data packet stored in the multimedia buffer area to the Bluetooth controller, so that the Bluetooth controller sends the second multimedia data packet. According to the embodiment of the invention, the time delay of the data transmitted by the Bluetooth can be reduced, and the real-time performance of the data is improved.

Description

Technical Field

[0001] This application relates to the field of communication technologies, and particularly to a method, apparatus, electronic device, and program product for processing multimedia data. Background Art

[0002] As a short - range wireless communication technology, Bluetooth communication technology is widely used in various communication scenarios due to its advantages such as low power consumption, low cost, and ease of use. However, in a complex wireless communication environment, Bluetooth signals are easily interfered with, affecting Bluetooth data transmission and unable to meet user requirements. Summary of the Invention

[0003] Embodiments of this application disclose a method, apparatus, electronic device, and program product for processing multimedia data, which can reduce the latency of data transmitted via Bluetooth and improve the real - time performance of the data.

[0004] Embodiments of this application disclose a method for processing multimedia data, which is applied to an electronic device. The electronic device includes a Bluetooth host and a Bluetooth controller, and the Bluetooth host is communicatively connected to the Bluetooth controller. The method includes:

[0005] The Bluetooth host obtains a multimedia data packet to be transmitted and stores the multimedia data packet in a multimedia buffer;

[0006] When the data discard condition is met, the Bluetooth host determines one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer; the first duration from the generation time corresponding to the first multimedia data packet to the current time is greater than or equal to a duration threshold;

[0007] The Bluetooth host discards the one or more first multimedia data packets from the multimedia buffer;

[0008] The Bluetooth host transmits the second multimedia data packet stored in the multimedia buffer to the Bluetooth controller so that the Bluetooth controller sends the second multimedia data packet.

[0009] Embodiments of this application disclose a device for processing multimedia data, which is applied to an electronic device. The electronic device includes a Bluetooth host and a Bluetooth controller, and the Bluetooth host is communicatively connected to the Bluetooth controller. The device includes:

[0010] A storage module, configured to obtain a multimedia data packet to be transmitted through the Bluetooth host and store the multimedia data packet in a multimedia buffer;

[0011] A timeout determination module, configured to determine one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer when the Bluetooth host meets the data discarding condition; the first duration between the generation time corresponding to the first multimedia data packet and the current time is greater than or equal to a duration threshold.

[0012] A discarding module, configured to discard the one or more first multimedia data packets from the multimedia buffer through the Bluetooth host.

[0013] A transmission module, configured to transmit the second multimedia data packets stored in the multimedia buffer to the Bluetooth controller through the Bluetooth host, so that the Bluetooth controller sends the second multimedia data packets.

[0014] An embodiment of the present application discloses an electronic device, including a memory, a processor and a transceiver unit. A computer program is stored in the memory. When the computer program is executed by the processor, the electronic device implements the method as described above.

[0015] An embodiment of the present application discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor in an electronic device, the electronic device implements the method as described above.

[0016] An embodiment of the present application discloses a computer program product, including a computer program. When the computer program is executed by a processor in an electronic device, the electronic device implements the method as described above.

[0017] The method, apparatus, electronic device, and program product for processing multimedia data disclosed in the embodiments of the present application. The Bluetooth host in the electronic device acquires the multimedia data packets to be transmitted and stores them in the multimedia buffer. When the data discarding condition is met, the Bluetooth host determines one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer. The first duration from the generation time of the first multimedia data packet to the current time is greater than or equal to the duration threshold, and the one or more first multimedia data packets are discarded from the multimedia buffer. The Bluetooth host transmits the second multimedia data packets stored in the multimedia buffer to the Bluetooth controller in the electronic device so that the Bluetooth controller sends the second multimedia data packets. In the embodiments of the present application, when the data discarding condition is met, the Bluetooth host detects the first multimedia data packets in the multimedia buffer with a generation time that is too long from the current time (indicating a large delay) and discards them, which can ensure that the second multimedia data packets transmitted to the Bluetooth controller are multimedia data packets with a generation time that is shorter from the current time, thereby ensuring that the delay of the second multimedia data packets sent by the Bluetooth controller is small, reducing the delay of the data transmitted by Bluetooth, and improving the real-time performance of the data. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] To more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0019] Figure 1A Schematic diagram of Bluetooth data transmission in the related art;

[0020] Figure 1B Flowchart of the automatic refresh mechanism supported by the classic Bluetooth protocol in one embodiment;

[0021] Figure 2A Application scenario diagram of the method for processing multimedia data in one embodiment;

[0022] Figure 2B Schematic diagram of the Bluetooth architecture in one embodiment;

[0023] Figure 3 Flowchart of the method for processing multimedia data in one embodiment;

[0024] Figure 4 Flowchart of the Bluetooth host discarding multimedia data packets with too large a delay in one embodiment;

[0025] Figure 5Flowchart of a multimedia data packet discarded by a Bluetooth host with excessive latency in another embodiment;

[0026] Figure 6 Flowchart of a Bluetooth host transmitting a signaling data packet to a Bluetooth controller in one embodiment;

[0027] Figure 7 Schematic diagram of data transmission in one embodiment;

[0028] Figure 8 Flowchart of a Bluetooth host transmitting data to a Bluetooth controller in one embodiment;

[0029] Figure 9 Block diagram of a multimedia data processing device in one embodiment;

[0030] Figure 10 Block diagram of an electronic device in one embodiment. Detailed implementation manners

[0031] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0032] It should be noted that the terms "including" and "having" and any variations thereof in the embodiments of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0033] It can be understood that the terms "first", "second", etc. used in the present application can be used to describe various elements in this article, but these elements are not limited by these terms. These terms are only used to distinguish one element from another. For example, without departing from the scope of the present application, the first multimedia data packet can be called the second multimedia data packet, and similarly, the second multimedia data packet can be called the first multimedia data packet. Both the first multimedia data packet and the second multimedia data packet are multimedia data packets, but they are not the same multimedia data packet. The term "plurality" used in the present application refers to two or more. The term "and / or" used in the present application refers to one of the solutions or any combination of multiple solutions.

[0034] In Bluetooth communication technology, a sending device sends Bluetooth data packets to a peer device. If the peer device receives the Bluetooth data packet, it will send an ACK (Acknowledgment) confirmation signal to the sending device. After receiving the ACK confirmation signal feedback from the peer device, the sending device will send the next Bluetooth data packet. If the sending device does not receive the ACK confirmation signal within a certain period of time, it will retransmit the Bluetooth data packet to the peer device.

[0035] Exemplarily, Figure 1A FIG. is a schematic diagram of Bluetooth data transmission in the related art. As Figure 1A shown, after the sending device sends a Bluetooth data packet to the peer device each time, it will send the next Bluetooth data packet only after receiving the ACK signal feedback from the peer device. If the ACK confirmation signal is not received within a certain period of time, the Bluetooth data packet will be resent to the peer device.

[0036] In a complex wireless communication environment, Bluetooth signals are easily interfered with, and Bluetooth data packets are prone to long-term and frequent retransmissions. Then, the unsent Bluetooth data packets will keep piling up in the sending queue. Even if the subsequent wireless communication environment improves, since the Bluetooth data packets stored in the sending queue have been piled up for a long time, it will cause a large delay in the data sent from the sending device to the peer device. Taking the example of a mobile phone sending voice data to a headset via Bluetooth, the voice data sent by the mobile phone to Bluetooth may be from the previous few seconds, with a large delay and unable to meet the user's real-time requirement for voice data.

[0037] To achieve real-time transmission of data packets, the classic Bluetooth protocol supports an Auto Flush mechanism, which is used to optimize data transmission efficiency, reduce latency, and avoid buffer overflow and other problems. When the Bluetooth Host sends a data packet to the Bluetooth Controller, it can set the boundary flag bit (such as Packet_Boundary_Flag) in the data packet to be automatically flushable (such as automatically-flushable), thereby triggering the Bluetooth Controller to automatically clear the unsuccessfully transmitted data packets in the buffer. The Bluetooth Host will send a flush timeout parameter to the Bluetooth Controller. The Bluetooth Controller will record the arrival time of each data packet sent by the Bluetooth Host. If the Bluetooth Controller detects that the duration (the time difference between the current time and the arrival time) of a certain data packet reaching the Bluetooth Controller exceeds the above flush timeout parameter and has not been successfully sent to the peer device, the data packet will be discarded, thereby reducing data latency.

[0038] Figure 1BFlowchart of the automatic refresh mechanism supported by the classic Bluetooth protocol in an embodiment. As Figure 1B shown, the Bluetooth Controller sends data packets, such as packets of types DV (Data Voice), DM (Data Medium-rate), or DH (Data High-rate). The Bluetooth Controller can determine whether an ACK confirmation signal corresponding to the sent data packet is received. If the ACK confirmation signal is received, the SEQN (Sequential Number) can be updated, and then the next new data packet can be sent. If the Bluetooth Controller does not receive the ACK confirmation signal corresponding to the sent data packet, it can determine whether to perform a Flush operation. If the Flush operation is not performed, the data packet can be retransmitted. If the Flush operation is performed, the data packets in the buffer that have reached the Bluetooth Controller for a duration exceeding the above refresh timeout parameter can be cleared, and a data packet without a payload can be sent to the peer device; if not, the original data packet can be retransmitted.

[0039] However, for the BLE (Bluetooth Low Energy) protocol, the above automatic refresh mechanism is not supported. If the Bluetooth Controller does not receive the ACK confirmation signal, it will keep retransmitting until the ACK confirmation signal feedback from the peer device is received, and then it will send the next packet, resulting in the accumulation of data packets in the buffer, causing a large delay in the transmitted data, poor data real-time performance, and unable to meet the user's requirements for data real-time performance.

[0040] The embodiments of the present application provide a method, device, electronic device, and program product for processing multimedia data, which can reduce the delay of the data transmitted via Bluetooth and improve the real-time performance of the data.

[0041] Figure 2A Application scenario diagram of the method for processing multimedia data in an embodiment. The method for processing multimedia data can be applied to the electronic device 210.

[0042] The electronic device 210 can include but is not limited to devices such as mobile phones, wearable devices (such as smart watches, smart glasses, etc.), vehicle-mounted terminals, tablet computers, laptop computers, and PCs (Personal Computers).

[0043] The electronic device 210 can establish a Bluetooth communication connection with the peer device 220, and the peer device 220 can include but is not limited to mobile phones, wearable devices, vehicle-mounted terminals, tablet computers, laptop computers, PCs, headphones, smart speakers, and smart home devices.

[0044] Among them, the electronic device 210 can establish a BLE connection with the peer device 220, or establish a classic Bluetooth connection, or establish both a BLE connection and a classic Bluetooth connection at the same time, etc., which is not limited here.

[0045] Figure 2B It is a schematic diagram of a Bluetooth architecture in an embodiment. As Figure 2B shown, the electronic device 210 includes a Bluetooth Host and a Bluetooth Controller, and the Bluetooth Host and the Bluetooth Controller are important components of the Bluetooth protocol stack.

[0046] The Bluetooth Host can provide an interface to the application programs in the application layer, allowing the application programs in the application layer to communicate with external devices through the Bluetooth protocol stack. The Bluetooth Host is mainly responsible for managing the high-level protocols in the Bluetooth protocol stack. For example, one or more of L2CAP (Logical Link Control and Adaptation Protocol), SDP (Session Description Protocol), ATT (Attribute Protocol), and GATT (Generic Attribute Profile).

[0047] The Bluetooth Controller can be responsible for handling the hardware operations and link layer functions in Bluetooth communication, can manage the link layer, is responsible for processes such as the establishment, maintenance, and disconnection of Bluetooth links, and can also be responsible for the reception and transmission of wireless signals, etc.

[0048] The Bluetooth Host and the Bluetooth Controller can be communicatively connected. For example, the Bluetooth Host and the Bluetooth Controller can interact through the HCI (Host Controller Interface).

[0049] Optionally, the Bluetooth Host and the Bluetooth Controller can be respectively deployed on different chips. For example, the Bluetooth Host can be deployed on the AP (Application Processor) chip of the electronic device, and the Bluetooth Controller can be deployed on the Bluetooth SoC (System on Chip).

[0050] Optionally, the Bluetooth Host and the Bluetooth Controller can also be deployed in the same chip.

[0051] The Bluetooth Host may have a multimedia buffer, which can be used to store multimedia data packets to be transmitted to the Bluetooth Controller, such as voice data packets generated by a call, audio data packets generated by playing music, etc.

[0052] Optionally, the Bluetooth Host may also have a signaling buffer, which can be used to store signaling data packets to be transmitted to the Controller. The signaling data packet can carry information such as instructions and commands sent by an application program at the application layer. The instructions, commands, etc. can be used to control the peer device to perform corresponding operations.

[0053] The Bluetooth Controller may have a transmit buffer, which is used to store data packets to be sent to the peer device, including but not limited to multimedia data packets, signaling data packets, etc.

[0054] In the embodiment of the present application, the Bluetooth Host can obtain the multimedia data packets to be transmitted and store the multimedia data packets to be transmitted in the multimedia buffer. When the Bluetooth Host meets the data discard condition, it can determine one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer. The first duration from the generation time of the first multimedia data packet to the current time is greater than or equal to the duration threshold. The Bluetooth Host can discard the one or more first multimedia data packets from the multimedia buffer, so as to discard the first multimedia data packets with larger latency in the multimedia buffer. The Bluetooth Host can transmit the second multimedia data packets stored in the multimedia buffer to the Bluetooth Controller, and the Bluetooth Controller then sends the second multimedia data packets to the peer device 220. Since the multimedia data packets with larger latency in the multimedia buffer of the Bluetooth Host are discarded, it can be ensured that the second multimedia data packets transmitted by the Bluetooth Host to the Bluetooth Controller are data packets with smaller latency, so as to ensure that the data latency of the Bluetooth Controller sent to the peer device 220 is smaller, reduce the latency of the data transmitted by Bluetooth, and improve the real-time performance of the data.

[0055] It should be noted that the peer device 220 may also include a Bluetooth Host and a Bluetooth Controller. The Bluetooth Controller of the peer device 220 can receive the data packets sent by the Bluetooth Controller of the electronic device 210 and then transmit the received data packets to the Bluetooth Host for processing.

[0056] As Figure 3 shown, in one embodiment, a method for processing multimedia data is provided, which can be applied to the above-mentioned electronic device. The method may include the following steps:

[0057] Step 310, the Bluetooth host obtains the multimedia data packet to be transmitted and stores the multimedia data packet in the multimedia buffer.

[0058] The multimedia data packet to be transmitted may refer to the multimedia data packet in the Bluetooth Host that waits to be sent to the Bluetooth Controller and then sent to the peer device through the Bluetooth Controller. The multimedia data packet may carry multimedia data, and the multimedia data may include, but is not limited to, one or more of video data, audio data, and file data.

[0059] As an optional implementation manner, the Bluetooth Host may obtain the multimedia data that the application program in the application layer needs to send, and encapsulate the multimedia data through protocols such as A2DP to obtain the multimedia data packet to be transmitted.

[0060] The Bluetooth Host may have a multimedia buffer, and the multimedia buffer may be used to store the multimedia data packets in the Bluetooth Host that are to be transmitted to the Bluetooth Controller. After the Bluetooth Host obtains the multimedia data packet to be transmitted, it may store the multimedia data packet to be transmitted in the multimedia buffer.

[0061] When the Bluetooth Host obtains the multimedia data packet to be transmitted, it may also obtain the generation time corresponding to the multimedia data packet to be transmitted, and the generation time may refer to the generation time of the multimedia data carried in the multimedia data packet.

[0062] Optionally, the generation time corresponding to the multimedia data packet may be the encoding completion time corresponding to the multimedia data carried in the multimedia data packet. Taking an audio data packet as an example, after obtaining the original audio data from the application program, it is necessary to encode the original audio data through an audio encoder to obtain audio data in formats such as SBC (Subband Codec) and AAC (Advanced Audio Coding) suitable for Bluetooth transmission, and then the Bluetooth Host encapsulates the audio data through protocols such as A2DP to obtain the audio data packet to be transmitted. Then, the generation time corresponding to the audio data packet may be the time when the audio encoder encodes to obtain the audio data (i.e., the encoding completion time).

[0063] Optionally, the generation time corresponding to the multimedia data packet may be the acquisition time corresponding to the multimedia data carried in the multimedia data packet, and the acquisition time may refer to the time when the application program acquires the multimedia data. Taking the multimedia data carried in the multimedia data packet as voice data as an example, the generation time corresponding to the multimedia data packet may be the time when the application program acquires the voice segment corresponding to the multimedia data packet through an audio acquisition device such as a microphone.

[0064] As an implementation, the multimedia data sent by the application layer to the Bluetooth Host may carry a timestamp corresponding to the generation time. When the Bluetooth Host obtains the multimedia data, it may extract the timestamp to obtain the generation time corresponding to the multimedia data. After encapsulating the multimedia data through protocols such as A2DP to obtain a multimedia data packet, the generation time may be stored corresponding to the multimedia data packet. For example, the generation time may be stored corresponding to the packet identifier of the multimedia data packet, so as to facilitate subsequent detection of whether the delay of the multimedia data packet is too large.

[0065] In some embodiments, the multimedia buffer may include a buffer queue, and the multimedia data packets in the multimedia buffer may be arranged in a certain order. The Bluetooth Host may sequentially transmit the multimedia data packets in the multimedia buffer to the Bluetooth Controller according to this arrangement order.

[0066] For example, the multimedia data packets in the multimedia buffer may be arranged in the order from the earliest to the latest of the entry times when the multimedia data packets enter the multimedia buffer. The earlier the entry time of the multimedia data packet at the front, the longer the storage duration in the multimedia buffer, and it will be preferentially transmitted to the Bluetooth Controller, avoiding excessive delay caused by the multimedia data packet being stored in the multimedia buffer for too long.

[0067] Again, for example, the multimedia data packets in the multimedia buffer may be arranged in the order from the earliest to the latest of the generation times corresponding to the multimedia data packets. The earlier the generation time corresponding to the multimedia data packet at the front.

[0068] Step 320: When the Bluetooth host meets the data discarding condition, determine one or more first multimedia data packets according to the generation times corresponding to the multimedia data packets stored in the multimedia buffer; the first duration from the generation time corresponding to the first multimedia data packet to the current time is greater than or equal to the duration threshold.

[0069] The data discarding condition may refer to the condition that triggers the Bluetooth Host to detect whether there are multimedia data packets to be discarded in the multimedia buffer. The multimedia data packets to be discarded include those with excessive delay. Therefore, the data discarding condition may also refer to the condition that triggers the Bluetooth Host to detect whether there are multimedia data packets with excessive delay in the multimedia buffer.

[0070] Among them, the multimedia data packets with excessive time delay may include the multimedia data packets whose first duration from the generation moment corresponding to them to the current moment is greater than or equal to the duration threshold. That the first duration from the generation moment corresponding to the multimedia data packet to the current moment is greater than or equal to the duration threshold indicates that the current moment is relatively far from the generation time of the multimedia data carried in the multimedia data packet, which means that the multimedia data carried in the multimedia data packet has a large time delay and poor real-time performance.

[0071] The data discard condition can be preset. When the data discard condition is met, the Bluetooth Host can be triggered to detect the multimedia data packets stored in the multimedia buffer, and determine whether there are multimedia data packets to be discarded in the multimedia buffer.

[0072] Optionally, the data discard condition can be a condition indicating that the Bluetooth Host can transmit multimedia data packets to the Bluetooth Controller. When this data discard condition is met, the Bluetooth Host will transmit multimedia data packets to the Bluetooth Controller. Then, before transmitting the multimedia data packets to the Bluetooth Controller, the Bluetooth Host can detect whether there are multimedia data packets to be discarded in the multimedia buffer.

[0073] Optionally, the Bluetooth Host can detect whether there are multimedia data packets to be discarded in the multimedia buffer according to a preset detection period. Then, the data discard condition can be a condition indicating that the preset detection period is satisfied.

[0074] Exemplarily, the data discard condition may include that the second duration from the current moment to the target detection moment reaches the target time interval. The target detection moment can be the moment when the Bluetooth Host last detected whether there is a first multimedia data packet (a multimedia data packet to be discarded) in the multimedia buffer. The target time interval can be the cycle duration of the preset detection period. For example, the Bluetooth Host can detect whether there are multimedia data packets to be discarded in the multimedia buffer every 1 second, then the target time interval can be 1 second; another example, the Bluetooth Host can detect whether there are multimedia data packets to be discarded in the multimedia buffer every 2 seconds, then the target time interval can be 2 seconds, which is not limited here. In this way, the Bluetooth Host can regularly detect and discard the multimedia data packets with excessive time delay in the multimedia buffer, so as to ensure that the time delay of the multimedia data packets stored in the multimedia buffer is small, and when it can transmit multimedia data packets to the Bluetooth Controller, there is no need to detect again, improving the data transmission efficiency between the Bluetooth Host and the Bluetooth Controller.

[0075] When the data discarding condition is met, the Bluetooth Host can calculate the first duration from the generation time corresponding to the multimedia data packet stored in the multimedia buffer to the current time according to the generation time of the multimedia data packet, and determine whether the delay of the multimedia data packet is too large according to the first duration corresponding to the multimedia data packet. If the first duration corresponding to the multimedia data packet is greater than or equal to the duration threshold, it is considered that the delay of the multimedia data packet is too large, so as to determine one or more first multimedia data packets in the multimedia buffer. The first multimedia data packet refers to the multimedia data packet that needs to be discarded in the multimedia buffer.

[0076] Further, the determined one or more first multimedia data packets can be all the multimedia data packets in the multimedia buffer whose corresponding first durations are greater than or equal to the duration threshold, that is, the Bluetooth Host can discard all the multimedia data packets with too large delays in the multimedia buffer.

[0077] The determined one or more first multimedia data packets can also be some of the multimedia data packets in the multimedia buffer whose corresponding first durations are greater than or equal to the duration threshold, that is, the Bluetooth Host can discard some of the multimedia data packets with too large delays in the multimedia buffer.

[0078] In the embodiment of the present application, the first duration is calculated using the generation time corresponding to the multimedia data packet. The first duration can accurately measure the duration from the generation time of the multimedia data, so as to accurately determine whether the multimedia data carried in the multimedia data packet has too large a delay, improving the accuracy of the delay determination of the multimedia data.

[0079] The above time threshold can be set according to actual needs. As an implementation manner, for multimedia data packets of different data types, the time threshold can be the same.

[0080] As another implementation manner, for multimedia data packets of different data types, different duration thresholds can be corresponding.

[0081] The data type corresponding to the multimedia data packet can be used to indicate the type of the multimedia data stored in the multimedia data packet. For example, the data type can include a voice type, an audio type of an audio player, an audio type of a video player, etc. Among them, the voice type is used to indicate that the multimedia data stored in the multimedia data packet is voice data, the audio type of the audio player is used to indicate that the multimedia data stored in the multimedia data packet is the audio played by the audio player, the audio type of the video player is used to indicate that the multimedia data stored in the multimedia data packet is the audio played by the video player, etc., but not limited thereto.

[0082] The real-time nature of multimedia data packets of different data types varies. The higher the real-time nature of a data type, the higher the user's requirement for the real-time nature of the multimedia data of that data type. For example, the real-time nature of the voice type can be greater than that of the audio type of an audio player; the real-time nature of the voice type can be greater than that of the audio type of a video player; the real-time nature of the audio type of a video player (to ensure audio-visual synchronization) can be greater than that of the audio type of an audio player, etc., but not limited to this.

[0083] The duration threshold can be negatively correlated with the real-time nature of the data type. The greater the real-time nature of the data type, the higher the user's requirement for the real-time nature of the multimedia data of that data type, and the lower the tolerance for the generated delay, then the duration threshold can be smaller.

[0084] Taking the data type corresponding to the multimedia data packet including the first data type or the second data type as an example, the real-time nature of the first data type can be greater than that of the second data type. For example, the first data type can be the voice type, and the second data type can be the audio type of an audio player. Then the duration threshold corresponding to the first data type can be less than or equal to the duration threshold corresponding to the audio type of the second data type.

[0085] Optionally, the user has different requirements for the playback quality of multimedia data of different data types. The duration threshold can also be positively correlated with the playback quality requirements corresponding to the data type. The higher the playback quality requirements corresponding to the data type, the greater the duration threshold. Then the multimedia data packet corresponding to that data type is less likely to be discarded, thereby reducing the frequency of the multimedia data packet corresponding to that data type being discarded and ensuring the playback quality of the multimedia data of that data type. For example, the playback quality requirements corresponding to the audio type of an audio player are higher than those corresponding to the voice type. Then the duration threshold corresponding to the audio type of the audio player can be greater than the duration threshold corresponding to the voice type.

[0086] In this embodiment, different duration thresholds can be used to detect multimedia data packets with excessive delay in the multimedia buffer according to the real-time nature of the data type corresponding to the multimedia data packet, adapting to the user's requirements for the real-time nature of multimedia data of different data types, and taking into account the user's requirements for the playback quality of multimedia data of different data types, further improving the user experience.

[0087] Step 330, the Bluetooth host discards the above one or more first multimedia data packets from the multimedia buffer.

[0088] The Bluetooth Host can perform a Flush operation on the multimedia buffer to discard the above determined one or more first multimedia data packets from the multimedia buffer.

[0089] Step 340: The Bluetooth host transmits the second multimedia data packet stored in the multimedia buffer to the Bluetooth controller, so that the Bluetooth controller sends the second multimedia data packet.

[0090] The second multimedia data packet refers to the remaining multimedia data packets in the multimedia buffer. Optionally, the second multimedia data packet can be a multimedia data packet with a smaller time delay. For example, the second multimedia data packet can be a multimedia data packet whose generation time is less than the time threshold from the current time. The Bluetooth Host transmits the second multimedia data packet stored in the multimedia buffer to the Bluetooth Controller, and the Bluetooth Controller can add the second multimedia data packet to the transmission buffer and wait to be sent to the peer device. The second multimedia data packet has a smaller time delay, so as to reduce the time delay of the multimedia data sent by the Bluetooth Controller to the peer device and improve the real-time performance of the data.

[0091] It should be noted that if the Bluetooth Host only discards some multimedia data packets with too large time delays in the multimedia buffer, the second multimedia data may also be a multimedia data packet whose generation time is greater than or equal to the time threshold from the current time. However, since most of the multimedia data packets with too large time delays have been discarded, the probability of multimedia data packets with too large time delays among the remaining multimedia data packets in the multimedia buffer is reduced. Therefore, the real-time performance of the multimedia data sent by the Bluetooth Controller to the peer device can still be improved.

[0092] In some embodiments, a BLE connection can be established between the electronic device and the peer device, and the Bluetooth Controller can send the above-mentioned second multimedia data packet to the peer device through the BLE connection with the peer device. This solution can be applied to the BLE communication scenario. Since in the related art, the BLE communication protocol does not support the automatic refresh mechanism, therefore, through the method of multimedia data provided in the embodiments of the present application, a time-based Flush mechanism supported by the BLE communication protocol is provided, which can reduce the time delay of the multimedia data packets transmitted by the Bluetooth Host to the Bluetooth Controller, thereby reducing the time delay of the data transmitted in the BLE communication and improving the real-time performance of the data.

[0093] It should be noted that the method of multimedia data provided in the embodiments of the present application can also be applied to classic Bluetooth communication, which is not limited here.

[0094] In an embodiment of the present application, when the Bluetooth Host meets the data discarding condition, it detects the first multimedia data packet in the multimedia buffer whose generation time is too long from the current time (indicating a large delay), and discards it, which can ensure that the second multimedia data packet transmitted to the Bluetooth Controller is a multimedia data packet with a short generation time from the current time, so as to ensure that the delay of the second multimedia data packet sent by the Bluetooth Controller is small, reduce the delay of the data transmitted by Bluetooth, and improve the real-time performance of the data.

[0095] In some embodiments, the data discarding condition may be a condition indicating that the Bluetooth Host can transmit multimedia data packets to the Bluetooth Controller. The data discarding condition may include: the Bluetooth host receives the first information sent by the Bluetooth controller, and the first information is used to indicate that there is free space in the transmission buffer in the Bluetooth controller.

[0096] After the Bluetooth Controller successfully sends the data packet (such as a multimedia data packet) in the transmission buffer to the peer device, the transmission buffer will be updated. The successfully sent data packet goes out of the transmission buffer, and the space occupied by the successfully sent data packet in the transmission buffer becomes free, and new data can be stored again.

[0097] When there is free space in the transmission buffer of the Bluetooth Controller, and the free space is the space where data can be written in the transmission buffer, the Bluetooth Controller can send the first information to the Bluetooth Host. After receiving the first information, the Bluetooth Host can send the multimedia data packet stored in the multimedia buffer to the Bluetooth Controller according to the first information.

[0098] Optionally, the Bluetooth Controller can send the first information to the Bluetooth Host every time it successfully sends a data packet to the peer device, so that the Bluetooth Host immediately sends a new multimedia data packet to the Bluetooth Controller and stores it in the transmission buffer.

[0099] Optionally, the Bluetooth Controller can also send the first information to the Bluetooth Host at regular intervals. For example, the Bluetooth Controller sends the first information to the Bluetooth Host every 1 second or every 1.5 seconds, etc., so that the Bluetooth Host regularly sends a new multimedia data packet to the Bluetooth Controller and stores it in the transmission buffer, which can reduce the communication overhead.

[0100] When the Bluetooth Host receives the first information sent by the Bluetooth Controller, it indicates that the Bluetooth Host needs to transmit multimedia data packets to the Bluetooth Controller. Before transmitting the multimedia data packets to the Bluetooth Controller, the Bluetooth Host can detect whether there are multimedia data packets to be discarded in the multimedia buffer, and discard the first multimedia data packet with an excessive delay detected. Detect the multimedia buffer according to the transmission requirements of the Bluetooth Controller and discard the first multimedia data packet with an excessive delay, so as to ensure that the delay of each multimedia data packet transmitted from the Bluetooth Host to the Bluetooth Controller is small, reduce the delay of the multimedia data packets transmitted from the Bluetooth Host to the Bluetooth Controller, and improve the real-time performance of the data.

[0101] When the Bluetooth Host triggers the detection of whether there are multimedia data packets to be discarded in the multimedia buffer according to the first information sent by the Bluetooth Controller, the Bluetooth Host can sequentially determine whether the multimedia data packets are to be discarded or transmitted to the Bluetooth Controller according to the arrangement order of each multimedia data packet in the multimedia buffer. As an implementation, as Figure 4 shown, the above method may further include the following steps:

[0102] Step 402, the Bluetooth host receives the first information sent by the Bluetooth controller.

[0103] The description of step 402 can refer to the relevant description in the above embodiment and will not be repeated here.

[0104] Step 404, the Bluetooth host determines whether the first duration from the generation time corresponding to the multimedia data packet arranged at the head in the multimedia buffer to the current time is greater than or equal to the duration threshold. If so, step 406 is executed; if not, step 408 is executed.

[0105] When the Bluetooth Host receives the first information sent by the Bluetooth Controller, the Bluetooth Host can calculate the first duration from the generation time corresponding to the multimedia data packet arranged at the head to the current time, and determine whether the first duration corresponding to the multimedia data packet arranged at the head is greater than or equal to the duration threshold.

[0106] Step 406, the Bluetooth host determines that the multimedia data packet arranged at the head is the first multimedia data packet and discards the first multimedia data packet.

[0107] If the first duration corresponding to the multimedia data packet at the head of the queue is greater than or equal to the duration threshold, it indicates that the delay of the multimedia data packet at the head of the queue is too large, and it has been a relatively long time since the generation time of the multimedia data in the multimedia data packet at the head of the queue. Therefore, the Bluetooth Host can discard the multimedia data packet at the head of the queue from the multimedia buffer.

[0108] After discarding the multimedia data packet at the head of the queue from the multimedia buffer, the Bluetooth Host can continue to determine whether the first duration corresponding to the next multimedia data packet at the head of the queue is greater than or equal to the duration threshold until it is determined that the first duration corresponding to the multimedia data packet at the head of the queue is less than the duration threshold, that is, a multimedia data packet with a smaller delay is found.

[0109] Step 408, the Bluetooth host uses the multimedia data packet at the head of the queue as the second multimedia data packet and transmits it to the Bluetooth controller.

[0110] If the first duration corresponding to the multimedia data packet at the head of the queue is less than the duration threshold, it indicates that the delay of the multimedia data packet at the head of the queue is small, and it has been a relatively short time since the generation time of the multimedia data in the multimedia data packet at the head of the queue. Therefore, the Bluetooth Host can transmit the multimedia data packet at the head of the queue to the Bluetooth Controller and send it out through the Bluetooth Controller.

[0111] In some embodiments, the first information can also be used to indicate the size of the space corresponding to the free space.

[0112] The size of the space corresponding to the free space can be used to represent the amount of data that can be stored in the free space in the transmission buffer. Exemplarily, this amount of data can be represented by the number of bytes (Bytes) that the free space can store, or by the number of data packets that the free space can store, or by the number of samples (Samples) or the playback duration corresponding to the multimedia data that the free space can store.

[0113] As an implementation manner, the Bluetooth Controller can send the first information to the Bluetooth Host at regular intervals. The first information can include the amount of data successfully sent by the Bluetooth Controller within this regular interval. The amount of data successfully sent by the Bluetooth Controller within this regular interval is the size of the space corresponding to the free space, and can also be understood as the amount of data that the Bluetooth Host can transmit to the Bluetooth Controller in this round of data transmission.

[0114] For example, the first information may include the number of data packets successfully sent by the Bluetooth Controller within a certain period of time. Then, the number of multimedia data packets that the Bluetooth Host can transmit to the Bluetooth Controller in this round is the same as the number of successfully sent data packets.

[0115] For another example, the first information may include the total number of bytes successfully sent by the Bluetooth Controller within a certain period of time. Then, the total number of bytes corresponding to the multimedia data packets that the Bluetooth Host can transmit to the Bluetooth Controller in this round should be less than or equal to the total number of bytes successfully sent.

[0116] For another example, the first information may include the total playback duration corresponding to the multimedia data carried in each multimedia data packet successfully sent by the Bluetooth Controller within a certain period of time. Then, the total playback duration corresponding to the multimedia data carried in the multimedia data packets that the Bluetooth Host can transmit to the Bluetooth Controller in this round should be less than or equal to the total playback duration corresponding to the multimedia data carried in each successfully sent multimedia data packet.

[0117] After the Bluetooth Host uses the multimedia data packet arranged at the top as the second multimedia data packet and transmits it to the Bluetooth Controller, the Bluetooth Host can determine whether the total data volume corresponding to the second multimedia data packet transmitted to the Bluetooth Controller in this round reaches the size of the free space mentioned above.

[0118] If the total data volume corresponding to the second multimedia data packet transmitted by the Bluetooth Host to the Bluetooth Controller in this round has reached the size of the free space, it means that there is no free space in the Bluetooth Controller's transmission buffer to store the multimedia data packets transmitted by the Bluetooth Host. Then, the Bluetooth Host stops sending multimedia data packets to the Bluetooth Controller and waits for the Bluetooth Controller to send the first information next time.

[0119] If the total data volume corresponding to the second multimedia data packet transmitted by the Bluetooth Host to the Bluetooth Controller in this round has not reached the size of the free space, it means that the Bluetooth Host can continue to transmit multimedia data packets to the Bluetooth Controller. Then, the Bluetooth Host can determine whether the first duration from the generation time of the next multimedia data packet arranged at the top in the multimedia buffer to the current time is greater than or equal to the duration threshold, that is, re-execute steps 404 - 408 until the total data volume corresponding to the second multimedia data packet transmitted to the Bluetooth Controller in this round reaches the size of the free space.

[0120] Optionally, if the size of the free space is represented by the number of bytes, the number of samples, the playback duration, etc., since the number of bytes corresponding to each multimedia data packet, the number of samples of the multimedia data carried by the multimedia data packet, the playback duration, etc. may be different, there may be a risk of buffer overflow in the transmission buffer.

[0121] As an implementation, when the Bluetooth Host determines that the total data volume corresponding to the second multimedia data packet to be transmitted to the Bluetooth Controller in this round is less than the size of the free space, it can further calculate the sum value between the total data volume corresponding to the second multimedia data packet to be transmitted to the Bluetooth Controller in this round and the data volume of the next multimedia data packet arranged at the top, and determine whether the sum value is greater than the size of the free space. If the sum value is greater than the size of the free space, it means that there is a risk of buffer overflow in the transmission buffer of the Bluetooth Controller when sending the next multimedia data packet arranged at the top. Then the Bluetooth Host can stop sending multimedia data packets to the Bluetooth Controller and wait for the Bluetooth Controller to send the first message next time.

[0122] If the sum value is less than or equal to the size of the free space, it means that the free space in the transmission buffer of the Bluetooth Controller can still store the next multimedia data packet arranged at the top. The Bluetooth Host can continue to transmit multimedia data packets to the Bluetooth Controller. Then the Bluetooth Host can determine whether the first duration from the generation time corresponding to the next multimedia data packet arranged at the top in the multimedia buffer to the current time is greater than or equal to the duration threshold until the sum value between the total data volume corresponding to the second multimedia data packet to be transmitted to the Bluetooth Controller in this round and the data volume of the next multimedia data packet arranged at the top is greater than the size of the free space.

[0123] In this implementation, the Bluetooth Host can transmit multimedia data packets to the Bluetooth Controller according to the size of the free space feedback by the Bluetooth Controller, and can discard the first multimedia data packets with too large delay during the transmission process, and only transmit the second multimedia data packets with smaller delay to the Bluetooth Controller, effectively reducing the delay of the multimedia data packets transmitted by the Bluetooth Host to the Bluetooth Controller, thereby improving the real-time performance of the multimedia data transmitted by the Bluetooth Controller to the peer device.

[0124] In the embodiments of the present application, when the Bluetooth Host needs to transmit multimedia data packets to the Bluetooth Controller, it can detect whether there are first multimedia data packets in the multimedia buffer that need to be discarded and have a large time delay, and perform the detection in sequence according to the arrangement order of each multimedia data packet in the multimedia buffer. Each time the Bluetooth Host discards a small number of first multimedia data packets, so the user's perception of the lack of multimedia data is not obvious and will not have much impact on the user's overall playback experience. It can improve the real-time performance of data transmission while ensuring the user's playback experience of multimedia data.

[0125] In some embodiments, the Bluetooth Host can discard multiple first multimedia data packets with excessive time delay in the multimedia buffer at one time. For example Figure 5 As shown, the above method may further include the following steps:

[0126] Step 502, when the Bluetooth host meets the data discard condition, determine a target number of first multimedia data packets according to the generation time corresponding to each multimedia data packet stored in the multimedia buffer; the first duration between the generation time corresponding to the first multimedia data packet and the current time is greater than or equal to the duration threshold.

[0127] Step 504, the Bluetooth host discards the target number of first multimedia data packets from the multimedia buffer.

[0128] When the data discard condition is met, the Bluetooth Host can traverse each multimedia data packet stored in the multimedia buffer, calculate the first duration between the generation time corresponding to each multimedia data packet and the current time, and determine whether the first duration corresponding to each multimedia data packet is greater than or equal to the duration threshold. The Bluetooth Host determines a target number of first multimedia data packets and discards the target number of first multimedia data packets from the multimedia buffer.

[0129] Among them, the target number can be a number set in advance according to actual needs. For example, the target number can be 3, 5, etc., but not limited thereto.

[0130] Further, after the Bluetooth Host determines whether the first duration between the generation time corresponding to each multimedia data packet in the multimedia buffer and the current time is greater than or equal to the duration threshold, it can count the number of multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold. If the number of multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold is less than the target number, the Bluetooth Host can determine all multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold as the first multimedia data packets.

[0131] In the case where the number of multimedia data packets in the multimedia buffer, whose corresponding first duration is greater than or equal to the duration threshold, is less than the target number, it indicates that the number of multimedia data packets with excessive delay in the multimedia buffer is small. Then, all multimedia data packets in the multimedia buffer, whose corresponding first duration is greater than or equal to the duration threshold, can be discarded to ensure that the delay of the remaining multimedia data packets in the multimedia buffer is small, further ensuring the real-time performance of data transmission.

[0132] If the number of multimedia data packets in the multimedia buffer, whose corresponding first duration is greater than or equal to the duration threshold, is greater than the target number, the Bluetooth Host can select and discard the target number of multimedia data packets from them.

[0133] Optionally, in the case where the number of multimedia data packets in the multimedia buffer, whose corresponding first duration is greater than or equal to the duration threshold, is greater than the target number, the target number of first multimedia data packets can be the multimedia data packets whose generation times are arranged in the first target number.

[0134] The Bluetooth Host can select and discard the multimedia data packets, whose corresponding first duration is greater than or equal to the duration threshold and whose generation times are arranged in the first target number in the order from the earliest to the latest. In the case where there are many multimedia data packets with excessive delay in the multimedia buffer, only the target number of multimedia data packets are discarded each time, which can control the number of multimedia data packets discarded each time and avoid excessive loss of multimedia data packets, resulting in excessive missing parts of multimedia data and affecting the user's playback experience of multimedia data. Further, the multimedia data packets whose generation times are arranged in the first target number can be discarded, that is, the multimedia data packets with the largest delay are discarded, so as to ensure the real-time performance of multimedia data.

[0135] As another implementation manner, the Bluetooth Host can also sequentially determine whether the corresponding first duration of each multimedia data packet in the multimedia buffer is greater than or equal to the duration threshold. If so, the multimedia data packet can be discarded as the first multimedia data packet, and the judgment of the next multimedia data packet is carried out until the number of discarded multimedia data packets reaches the target number, or all multimedia data packets in the multimedia buffer are traversed.

[0136] That is to say, the Bluetooth Host can adopt the method of detecting whether the corresponding first duration of the multimedia data packet is greater than or equal to the duration threshold while discarding the multimedia data packets whose corresponding first duration is greater than or equal to the duration threshold, or can adopt the method of first determining all multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold, and then selecting and discarding the target number of multimedia data packets. The embodiments of the present application do not make specific limitations on this.

[0137] As an implementation, for multimedia data packets of different data types, the target quantity can be the same.

[0138] In some embodiments, for multimedia data packets of different data types, different target quantities can be corresponding. Optionally, the real-time nature of multimedia data packets of different data types is different, and the target quantity can have a positive correlation with the real-time nature of the data type. The greater the real-time nature of the data type, it indicates that the user has a higher requirement for the real-time nature of the multimedia data of this data type, and has a lower tolerance for the generated time delay. Then the target quantity can be larger, and the Bluetooth Host discards more multimedia data packets each time, further improving the real-time nature of the multimedia data.

[0139] Taking the data types corresponding to multimedia data packets including a first data type or a second data type as an example, the real-time nature of the first data type can be greater than that of the second data type. For example, the first data type can be a voice type, and the second data type can be an audio type of an audio player. Then the target quantity corresponding to the voice type can be greater than or equal to the target quantity corresponding to the second data type.

[0140] Optionally, the user has different requirements for the playback quality of multimedia data of different data types, and the target quantity can have a negative correlation with the playback quality requirements corresponding to the data type. The higher the playback quality requirements corresponding to the data type, the smaller the target quantity, and the Bluetooth Host discards smaller multimedia data packets each time, thereby ensuring the playback quality of the multimedia data of this data type.

[0141] In the embodiments of the present application, the Bluetooth Host can discard multiple first multimedia data packets with excessive time delays in the multimedia buffer at one time, thereby avoiding the Bluetooth Host frequently detecting whether there are multimedia data packets to be discarded in the multimedia buffer, and avoiding the Bluetooth Host discarding multimedia data packets too frequently, which can reduce the operating overhead of the Bluetooth Host and improve the processing efficiency of multimedia data.

[0142] In some embodiments, in addition to transmitting multimedia data packets to the Bluetooth Controller, the Bluetooth Host can also transmit signaling data packets to the Bluetooth Controller. As Figure 6 shown, the above method can further include the following steps:

[0143] Step 602, the Bluetooth host obtains the signaling data packet to be transmitted and stores the signaling data packet in the signaling buffer.

[0144] The Bluetooth Host can have a multimedia buffer and a signaling buffer. The multimedia buffer is used to store multimedia data packets, and the signaling buffer is used to store signaling data packets. The multimedia buffer and the signaling buffer are different storage spaces.

[0145] The signaling data packet can be a data packet carrying signaling, and the signaling can include commands, instructions, etc.

[0146] As a specific implementation, when an application program at the application layer needs to send signaling to a peer device, it can transmit the signaling to be sent to the Bluetooth Host. The Bluetooth Host can process the signaling through protocols such as AVRCP (Audio / Video Remote Control Profile) to generate a signaling data packet. The Bluetooth Host then stores the signaling data packet in the signaling buffer and then transmits it to the Bluetooth Controller, which is sent out by the Bluetooth Controller.

[0147] Step 604, the Bluetooth host transmits the signaling data packet stored in the signaling buffer to the Bluetooth controller according to the size relationship between the first priority corresponding to the signaling data packet stored in the signaling buffer and the second priority corresponding to the multimedia data packet stored in the multimedia buffer, so that the Bluetooth controller sends the signaling data packet.

[0148] The Bluetooth Controller can store the multimedia data packets and signaling data packets to be sent through the same sending buffer. When the space in the sending buffer of the Bluetooth Controller is limited, the Bluetooth Host can transmit multimedia data packets and / or signaling data packets to the Bluetooth Controller according to the respective priorities of the multimedia data packets and signaling data packets.

[0149] When there are signaling data packets stored in the signaling buffer and multimedia data packets stored in the multimedia buffer, the Bluetooth Host can compare the first priority corresponding to the signaling data packet with the second priority corresponding to the multimedia data packet stored in the multimedia buffer to determine the size relationship between the two. If the first priority corresponding to the signaling data packet is higher than the second priority corresponding to the multimedia data packet, the signaling data packet can be preferentially transmitted to the Bluetooth Controller, and after the transmission of the signaling data packet is completed, the multimedia data packet is transmitted to the Bluetooth Controller. If the first priority corresponding to the signaling data packet is lower than the second priority corresponding to the multimedia data packet, the multimedia data packet can be preferentially transmitted to the Bluetooth Controller, and then the signaling data packet is transmitted to the Bluetooth Controller.

[0150] Optionally, the priorities corresponding to the multimedia data packets and the signaling data packets respectively can be pre-configured. For example, it can be pre-set that the first priority corresponding to the signaling data packet is higher than the second priority corresponding to the multimedia data packet. In this case, when there is a signaling data packet stored in the signaling buffer, the Bluetooth Host will always give priority to transmitting the signaling data packet to the Bluetooth Controller.

[0151] Optionally, the priorities corresponding to the multimedia data packets and the signaling data packets respectively can also be determined by the application program. The application program can allocate the priorities corresponding to the multimedia data packets and the signaling data packets according to the application scenario. As an implementation, the application program can allocate the priorities corresponding to the multimedia data packets and the signaling data packets according to the data type of the multimedia data to be sent and the control function corresponding to the signaling to be sent.

[0152] For example, assuming that the multimedia data to be sent is voice data and the control function corresponding to the signaling to be sent is to adjust the volume, the application program can allocate the second priority corresponding to the multimedia data packet to be higher than the first priority corresponding to the signaling data packet, give priority to transmitting the voice data, and then perform the volume adjustment control.

[0153] Another example, assuming that the multimedia data to be sent is audio data played by an audio player and the control function corresponding to the signaling to be sent is to switch to the next song, the application program can allocate the first priority corresponding to the signaling data packet to be higher than the second priority corresponding to the multimedia data packet, give priority to controlling the song switch, and then transmit the audio data.

[0154] It should be noted that since there are many multimedia data packets stored in the multimedia buffer, when the second priority corresponding to the multimedia data packet is higher than the first priority corresponding to the signaling data packet, the Bluetooth Host does not necessarily have to wait for all the multimedia data packets to be transmitted before transmitting the signaling data packet. The Bluetooth Host can allocate the ratio between the multimedia data packets and the signaling data packets to be transmitted in this round according to the size of the free space feedback by the Bluetooth Controller. Among them, the proportion of the transmitted multimedia data packets can be greater than or equal to the proportion of the signaling data packets. The Bluetooth Host can first transmit the second multimedia data packet to the Bluetooth Controller according to the proportion of the multimedia data packets, and then transmit the signaling data packet to the Bluetooth Controller according to the proportion of the signaling data packets, which can take into account the priorities corresponding to the multimedia data packets and the signaling data packets respectively, and ensure that the peer device can receive the signaling in time and execute the corresponding control function.

[0155] Exemplarily, Figure 7 is a schematic diagram of data transmission in an embodiment. As Figure 7As shown, an application in an electronic device and a Bluetooth Host can respectively establish a data channel and a signaling channel. The data channel can be used to transmit multimedia data, and the signaling channel can be used to transmit signaling. The application can send multimedia data to the Bluetooth Host through the data channel. The Bluetooth Host generates multimedia data packets to be transmitted and stores them in the multimedia buffer. The application can send signaling to the Bluetooth Host through the signaling channel. The Bluetooth Host generates signaling packets to be transmitted and stores them in the signaling buffer. When the Bluetooth Host receives the first information sent by the Bluetooth Controller and determines that data can be transmitted to the Bluetooth Controller, it can transmit multimedia data packets and / or signaling packets to the Bluetooth Controller according to the priorities corresponding to the multimedia data packets and the signaling packets respectively.

[0156] Further, as Figure 8 shown, before sending a multimedia data packet to the Bluetooth Controller, the Bluetooth Host can first determine whether the first duration between the generation time corresponding to the multimedia data packet and the current time is greater than or equal to a duration threshold. If the first duration corresponding to the multimedia data packet is greater than or equal to the duration threshold, it means that the delay of the multimedia data packet is too large, so the multimedia data packet can be discarded, and it can be determined whether the first duration corresponding to the next multimedia data packet is greater than or equal to the duration threshold. If the first duration corresponding to the multimedia data packet is less than the duration threshold, it means that the delay of the multimedia data packet is small, so the multimedia data packet can be transmitted to the Bluetooth Controller.

[0157] After obtaining the multimedia data packets and / or signaling packets transmitted by the Bluetooth Host, the Bluetooth Controller can store the multimedia data packets and / or signaling packets in the transmission buffer and then send them to the peer device. Further, after the Bluetooth Controller of the peer device receives the data packets sent by the electronic device, it can store them in the reception buffer and then transmit the data packets stored in the reception buffer to the Bluetooth Host. The Bluetooth Host of the peer device can parse the data packets sent by the Bluetooth Controller to obtain the multimedia data or signaling carried in the data packets, and then store the parsed multimedia data in the multimedia buffer or store the parsed signaling in the signaling buffer. The Bluetooth Host of the peer device can send the multimedia data stored in the multimedia buffer to the application through the data channel, so that the application can play the multimedia data. The Bluetooth Host of the peer device can also send the signaling stored in the signaling buffer to the application through the signaling channel, so that the application can perform corresponding control functions according to the signaling.

[0158] In an embodiment of the present application, the Bluetooth Host of the electronic device may have a multimedia buffer and a signaling buffer, which are respectively used to store multimedia data and signaling. The Bluetooth Host may transmit multimedia data packets and / or signaling data packets to the Bluetooth Controller according to the magnitude relationship between the priorities corresponding to the multimedia data packets and the signaling data packets, so as to meet the requirement of the electronic device to transmit multimedia data and signaling through the Bluetooth connection with the peer device, ensuring the real-time nature of the multimedia data and the timeliness of the signaling control, and improving the user experience.

[0159] As Figure 9 shown, in one embodiment, a multimedia data processing device 900 is provided, which can be applied to the above-mentioned electronic device. The multimedia data processing device 900 may include a storage module 910, a timeout determination module 920, a discard module 930, and a transmission module 940.

[0160] The storage module 910 is used to obtain the multimedia data packets to be transmitted through the Bluetooth host and store the multimedia data packets in the multimedia buffer.

[0161] The timeout determination module 920 is used to determine one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer when the data discard condition is met through the Bluetooth host; the first duration from the generation time corresponding to the first multimedia data packet to the current time is greater than or equal to the duration threshold.

[0162] In one embodiment, the data type corresponding to the multimedia data packet includes a first data type or a second data type. The real-time nature of the first data type is greater than that of the second data type, and the duration threshold corresponding to the first data type is less than or equal to the duration threshold corresponding to the second data type.

[0163] The discard module 930 is used to discard the one or more first multimedia data packets from the multimedia buffer through the Bluetooth host.

[0164] The transmission module 940 is used to transmit the second multimedia data packets stored in the multimedia buffer to the Bluetooth controller through the Bluetooth host, so that the Bluetooth controller sends the second multimedia data packets.

[0165] In one embodiment, the data discard condition includes: the Bluetooth host receives the first information sent by the Bluetooth controller, and the first information is used to indicate that there is free space in the send buffer in the Bluetooth controller.

[0166] In one embodiment, the timeout determination module 920 is further configured to determine, via the Bluetooth host, whether a first duration from a generation time corresponding to the multimedia data packet arranged at the head in the multimedia buffer to the current time is greater than or equal to a duration threshold; if the first duration corresponding to the multimedia data packet arranged at the head is greater than or equal to the duration threshold, determine, via the Bluetooth host, the multimedia data packet arranged at the head as the first multimedia data packet.

[0167] The transmission module 940 is further configured to, if the first duration corresponding to the multimedia data packet arranged at the head is not greater than the duration threshold, transmit, via the Bluetooth host, the multimedia data packet arranged at the head as the second multimedia data packet to the Bluetooth controller.

[0168] In one embodiment, the first information is further configured to indicate a size of the free space corresponding thereto.

[0169] The timeout determination module 920 is further configured to, if a total data volume corresponding to the second multimedia data packets transmitted by the Bluetooth host to the Bluetooth controller in this round does not reach the size of the free space corresponding thereto, determine, via the Bluetooth host, whether a first duration from a generation time corresponding to the next multimedia data packet arranged at the head in the multimedia buffer to the current time is greater than or equal to the duration threshold until the total data volume reaches the size of the free space corresponding thereto.

[0170] In one embodiment, the data discard condition includes: a second duration from the current time to a target detection time reaches a target time interval; the target detection time is a time when the Bluetooth host last detected whether there is a first multimedia data packet in the multimedia buffer.

[0171] In one embodiment, the timeout determination module 920 is further configured to determine, via the Bluetooth host, a target number of first multimedia data packets according to generation times corresponding to respective multimedia data packets stored in the multimedia buffer.

[0172] In one embodiment, in the case where the number of multimedia data packets corresponding to a first duration greater than or equal to the duration threshold in the multimedia buffer is greater than the target number, the target number of first multimedia data packets are multimedia data packets with generation times arranged in the top target number.

[0173] In one embodiment, the timeout determination module 920 is further configured to determine, via the Bluetooth host, whether a first duration from a generation time corresponding to each multimedia data packet in the multimedia buffer to the current time is greater than or equal to the duration threshold; if the number of multimedia data packets corresponding to a first duration greater than or equal to the duration threshold in the multimedia buffer is less than the target number, determine, via the Bluetooth host, all multimedia data packets corresponding to a first duration greater than or equal to the duration threshold in the multimedia buffer as the first multimedia data packets.

[0174] In one embodiment, the storage module 910 is further configured to obtain a signaling data packet to be transmitted through a Bluetooth host and store the signaling data packet in a signaling buffer.

[0175] The transmission module 940 is further configured to, through the Bluetooth host, based on the magnitude relationship between the first priority corresponding to the signaling data packet stored in the signaling buffer and the second priority corresponding to the multimedia data packet stored in the multimedia buffer, transmit the signaling data packet stored in the signaling buffer to the Bluetooth controller, so that the Bluetooth controller sends the signaling data packet.

[0176] In one embodiment, the transmission module 940 is further configured to use a Bluetooth Low Energy (BLE) connection with a peer device through the Bluetooth controller to send a second multimedia data packet to the peer device.

[0177] In the embodiment of the present application, when the Bluetooth host meets the data discarding condition, it detects and discards the first multimedia data packet in the multimedia buffer whose generation time is too long from the current time (indicating a large delay exists), which can ensure that the second multimedia data packet transmitted to the Bluetooth controller is a multimedia data packet with a shorter generation time from the current time. Thus, it can ensure that the delay of the second multimedia data packet sent by the Bluetooth controller is small, reduce the delay of the data transmitted via Bluetooth, and improve the real-time performance of the data.

[0178] Figure 10 It is a structural block diagram of an electronic device in one embodiment. As Figure 10 shown, the electronic device 1000 may include one or more of the following components: a processor 1010, and a memory 1020 coupled to the processor 1010, where the memory 1020 may store one or more computer programs, and when the one or more computer programs are configured to be executed by the one or more processors 1010, the electronic device 1000 can implement the methods described in the above embodiments.

[0179] The processor 1010 may include one or more processing cores. The processor 1010 connects various parts within the entire electronic device 1000 through various interfaces and lines. By running or executing instructions, programs, code sets, or instruction sets stored in the memory 1020, and by calling data stored in the memory 1020, it performs various functions of the electronic device 1000 and processes data. Optionally, the processor 1010 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), or programmable logic array (PLA). The processor 1010 may integrate a combination of one or several of a central processing unit (CPU), a graphics processing unit (GPU), and a modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the display content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 1010 and may be implemented separately through a communication chip.

[0180] The memory 1020 may include a random access memory (RAM) and may also include a read-only memory (ROM). The memory 1020 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1020 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for implementing at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area may also store data created during the use of the electronic device 1000.

[0181] The electronic device 1000 may further include a transceiver unit, which may include, but is not limited to, a Bluetooth module, etc., and may be used to provide Bluetooth communication functions.

[0182] An embodiment of this application discloses a computer-readable storage medium that stores a computer program. When the computer program is executed by a processor in an electronic device, the electronic device is enabled to implement the methods described in the above various embodiments.

[0183] An embodiment of the present application discloses a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor in an electronic device, the electronic device can implement the methods described in the above embodiments.

[0184] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. Among them, the storage medium can be a magnetic disk, an optical disc, a ROM, etc.

[0185] Any reference to a memory, storage, database, or other medium as used herein may include non-volatile and / or volatile memory. Suitable non-volatile memory may include ROM, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which is used as an external cache. By way of illustration and not limitation, RAM can be in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), memory bus direct RAM (Rambus DRAM, RDRAM), and direct memory bus dynamic RAM (DirectRambus DRAM, DRDRAM).

[0186] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that the specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics may be combined in one or more embodiments in any suitable manner. Those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to the present application. It should be noted that "a plurality of" in the present application includes "two or more".

[0187] In various embodiments of the present application, it should be understood that the magnitudes of the serial numbers of the above processes do not necessarily mean the inevitable sequence of execution. The execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0188] The units described as separate components above may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0189] In addition, in each embodiment of the present application, each functional unit may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of a software functional unit.

[0190] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0191] The above has introduced in detail a method, device, electronic device and program product for processing multimedia data disclosed in the embodiments of the present application. Specific examples are used in this article to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application. At the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for processing multimedia data, characterized in that: Applied to an electronic device, the electronic device comprises a Bluetooth host and a Bluetooth controller, the Bluetooth host is communicatively connected with the Bluetooth controller; the method comprises: The Bluetooth host obtains a multimedia data packet to be transmitted, and stores the multimedia data packet in a multimedia buffer; The Bluetooth host determines, when a data discard condition is met, one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer; the first duration between the generation time corresponding to the first multimedia data packet and the current time is greater than or equal to the duration threshold; The Bluetooth host discards the one or more first multimedia data packets from the multimedia buffer; The Bluetooth host transmits the second multimedia data packet stored in the multimedia buffer to the Bluetooth controller, so that the Bluetooth controller sends the second multimedia data packet.

2. The method according to claim 1, characterized in that The data discarding conditions include: The Bluetooth host receives first information sent by the Bluetooth controller, where the first information is used to indicate that there is free space in a sending buffer in the Bluetooth controller.

3. The method according to claim 2, characterized in that The Bluetooth host determines one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer, including: The Bluetooth host determines whether a first duration from the generation time of the multimedia data packet arranged first in the multimedia buffer to the current time is greater than or equal to a duration threshold; If the first duration corresponding to the multimedia data packet arranged in the first place is greater than or equal to the duration threshold, the Bluetooth host determines that the multimedia data packet arranged in the first place is the first multimedia data packet; The Bluetooth host transmits the second multimedia data packet stored in the multimedia buffer to the Bluetooth controller, including: If the first duration corresponding to the multimedia data packet arranged in the first place is not greater than the duration threshold, the Bluetooth host transmits the multimedia data packet arranged in the first place as the second multimedia data packet to the Bluetooth controller.

4. The method according to claim 3, characterized in that The first information is also used to indicate the size of the space corresponding to the free space; the method further includes: If the total data volume corresponding to the second multimedia data packet transmitted by the Bluetooth host to the Bluetooth controller in this round does not reach the space size corresponding to the free space, the Bluetooth host determines whether the first time duration from the generation moment corresponding to the next first multimedia data packet in the multimedia buffer to the current moment is greater than or equal to the time duration threshold, until the total data volume reaches the space size corresponding to the free space.

5. The method according to claim 1, characterized in that The data discarding conditions include: The second time length from the current moment to the target detection moment reaches the target time interval; the target detection moment is the moment when the Bluetooth host last detected whether there is a first multimedia data packet in the multimedia buffer.

6. The method according to any one of claims 1 to 2 and 5, characterized in that: The Bluetooth host determines one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer, including: The Bluetooth host determines a target number of first multimedia data packets according to generation times corresponding to respective multimedia data packets stored in the multimedia buffer.

7. The method according to claim 6, characterized in that In the multimedia buffer, when the number of multimedia data packets whose corresponding first duration is greater than or equal to the duration threshold is greater than the target number, the first multimedia data packets of the target number are the multimedia data packets arranged in front of the target number at the time of generation.

8. The method according to claim 6, characterized in that The Bluetooth host determines one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer, including: The Bluetooth host determines whether a first duration from a generation time corresponding to each multimedia data packet in the multimedia buffer to the current time is greater than or equal to a duration threshold; If the number of multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold is less than the target number, the Bluetooth host determines all multimedia data packets in the multimedia buffer whose corresponding first duration is greater than or equal to the duration threshold as first multimedia data packets.

9. The method according to any one of claims 1 to 5, 7 to 8, characterized in that: The method further comprises: The Bluetooth host obtains a signaling data packet to be transmitted, and stores the signaling data packet in a signaling buffer; The Bluetooth host transmits the signaling data packet stored in the signaling buffer to the Bluetooth controller based on the relationship between the first priority corresponding to the signaling data packet stored in the signaling buffer and the second priority corresponding to the multimedia data packet stored in the multimedia buffer, so that the Bluetooth controller sends the signaling data packet.

10. The method according to any one of claims 1 to 5, 7 to 8, characterized in that: The data type corresponding to the multimedia data packet includes a first data type or a second data type, the real-time performance of the first data type is greater than the real-time performance of the second data type, and the duration threshold corresponding to the first data type is less than or equal to the duration threshold corresponding to the second data type.

11. The method according to any one of claims 1 to 5, 7 to 8, characterized in that: The Bluetooth controller sending the second multimedia data packet includes: The Bluetooth controller sends the second multimedia data packet to the opposite device through a Bluetooth low energy BLE connection between the controller and the opposite device.

12. A multimedia data processing device, characterized in that: Applied to electronic equipment, the electronic equipment includes a Bluetooth host and a Bluetooth controller, the Bluetooth host is communicatively connected with the Bluetooth controller; the device includes: A storage module, used for acquiring a multimedia data packet to be transmitted through the Bluetooth host, and storing the multimedia data packet in a multimedia buffer; A timeout determination module, configured to determine, by the Bluetooth host, when a data discard condition is met, one or more first multimedia data packets according to the generation time corresponding to the multimedia data packets stored in the multimedia buffer; the generation time corresponding to the first multimedia data packet is greater than or equal to a time threshold from the current time; a discarding module, configured to discard the one or more first multimedia data packets from the multimedia buffer through the Bluetooth host; The transmission module is used to transmit the second multimedia data packet stored in the multimedia buffer to the Bluetooth controller through the Bluetooth host, so that the Bluetooth controller sends the second multimedia data packet.

13. An electronic device, characterized in that: The electronic device comprises a memory, a processor and a transceiver unit. The memory stores a computer program. When the computer program is executed by the processor, the electronic device implements the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The method comprises a computer program, and when the computer program is executed by a processor in an electronic device, the electronic device implements the method according to any one of claims 1 to 11.

Images