An electronic device and a data processing method
By offloading the decoding and rendering process of multimedia streams to the intellectual property core, the problem of high power consumption when electronic devices play multimedia data is solved, achieving the effects of reducing power consumption and improving processing flexibility.
Patent Information
- Application Number
- CN202510643851.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-16
- Publication Date
- 2026-11-17
AI Technical Summary
Electronic devices consume a lot of power when playing multimedia data, and how to reduce power consumption is an urgent problem to be solved.
The decoding and rendering process of multimedia streams is offloaded to the intellectual property core, reducing the load on the central processing unit and reducing power consumption by utilizing the specific functional design of the intellectual property core.
By offloading multimedia streaming processing to the intellectual property core, the load on the central processing unit is reduced, the power consumption of electronic devices is decreased, and the processing flexibility and device efficiency in multimedia playback are improved.
Smart Images

Figure CN123123640A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to an electronic device and a data processing method. Background Technology
[0002] With the development of terminal technology, the functions of electronic devices are becoming increasingly diverse, especially the playback of multimedia data (such as audio and video data), which is gradually becoming a common function of electronic devices. Electronic devices can play multimedia data through various installed applications.
[0003] Typically, electronic devices need to perform extensive calculations on multimedia data and drive corresponding output devices to play the multimedia data. Reducing the significant power consumption generated by electronic devices when playing multimedia data is a pressing issue that needs to be addressed. Summary of the Invention
[0004] This application provides an electronic device and a data processing method that offloads a significant portion of the workload from a first processing unit to a second processing unit. This eliminates the binding of multimedia stream decoding and display processes to the first processing unit, thereby reducing the load on the first processing unit.
[0005] To achieve the above objectives, this application adopts the following technical solution:
[0006] In a first aspect, this application provides a data processing method applied to an electronic device. The electronic device has a multimedia application installed and includes a first processing unit and a second processing unit. The method includes: the first processing unit running the multimedia application; the second processing unit acquiring a multimedia stream from the multimedia application; and the second processing unit processing the multimedia stream and outputting the processing result. The processing may be decoding and rendering, or rendering.
[0007] Thus, the embodiments of this application can utilize the second processing unit to process the multimedia stream. After the multimedia application provides the multimedia stream to the intellectual property core, it does not need to pay attention to the subsequent playback process of the multimedia stream.
[0008] On the one hand, when the first processing unit is the central processing unit (CPU) and the second processing unit is the intellectual property core (IPC), the CPU does not need to process the multimedia stream, thus offloading a large portion of the CPU's workload to the IPC. This eliminates the binding between the multimedia stream decoding and display processes and the CPU, reducing the CPU's workload.
[0009] Furthermore, the intellectual property core has a single function, allowing it to be designed for specific functions such as decoding, without requiring complex computing power. Therefore, when performing the same workload, the intellectual property core's processing unit and control logic are relatively simple, resulting in less power consumption. This reduces the overall power consumption of the electronic device. On the other hand, the multimedia streams proposed in this application include audio and video streams; different types of data streams can be processed by the intellectual property core, enhancing the device's processing flexibility during multimedia playback.
[0010] In one possible implementation, the multimedia stream includes a video stream, and before the second processing unit acquires the multimedia stream from the multimedia application, the method further includes:
[0011] The first processing unit obtains the first capability information of the second processing unit, which is used to characterize whether the second processing unit supports decoding and / or displaying the video stream.
[0012] Based on the first capability information, the first processing unit determines the first execution strategy for the second processing unit to process the video stream.
[0013] Thus, in this embodiment, the first processing unit may include a first system service. This first system service can utilize acquired hardware capability information to select the most suitable decoding and rendering method for the video stream, fully leveraging the advantages of the hardware. Simultaneously, given the varying hardware performance of different devices, developing strategies based on hardware capability information enables the system service to better adapt to different devices, ensuring the normal operation of the playback process and providing a relatively consistent user experience across various hardware platforms. Furthermore, the hardware performance differences between different devices are only perceptible at the system level, not at the multimedia application level, guaranteeing consistent multimedia application experience across devices and platforms.
[0014] In one possible implementation, if the first capability information indicates that the video stream is supported for decoding and display, the second processing unit decodes and renders the multimedia stream.
[0015] When the first capability information indicates that it supports decoding the video stream but does not support displaying the video stream, the second processing unit decodes the multimedia stream as a video stream, and the first processing unit controls the second processing unit to render the video stream.
[0016] Thus, in this embodiment of the application, when the first processing unit is a central processing unit (CPU) and the second processing unit is an intellectual property core (IPC), the video stream processing can be offloaded to the IPC, eliminating the need for CPU involvement. Simultaneously, the IPC has a single function, allowing it to be designed for specific tasks such as decoding, without requiring complex computational capabilities. Therefore, when performing the same workload, the IPC's computational unit and control logic are relatively simple, resulting in lower power consumption. This reduces the overall power consumption of the electronic device.
[0017] In one possible implementation, the second processing unit includes a third processing unit and a fourth processing unit, wherein the third processing unit is used to decode the video stream and the fourth processing unit is used to render the decoded video stream.
[0018] The third processing unit sends the decoded video stream to the fourth processing unit.
[0019] Thus, in this embodiment, when the third processing unit is a video decoding core and the fourth processing unit is a video rendering core, the video decoding core can send the decoded video stream to the video rendering core. The decoding and rendering processes are clearly separated, improving the processing efficiency of the video stream. Furthermore, due to this functional separation, when upgrades or improvements to the video decoding or rendering functions are needed, the video decoding core and video rendering core can be independently optimized and updated without affecting the functionality of other parts. This helps reduce system complexity and improves system maintainability and scalability.
[0020] In one possible implementation, the first processing unit includes a rendering control module; the rendering control module acquires the decoded video stream and sends the decoded video stream to the second processing unit so that the second processing unit renders the video stream.
[0021] Thus, in this embodiment, when the fourth processing unit is a video rendering core, the video rendering core is a hardware unit specifically designed for video rendering, possessing powerful parallel processing capabilities and an optimized architecture. The rendering control module sends the decoded video stream to the video rendering core, which can utilize its hardware characteristics for efficient rendering processing, greatly improving the speed and frame rate of video rendering and making video playback smoother. Simultaneously, allowing the video rendering core to handle video rendering tasks frees the first processing unit from complex graphics processing. This allows the first processing unit to focus on other important system tasks, thereby improving the overall system performance and response speed.
[0022] In one possible implementation, the multimedia stream includes an audio stream, prior to the second processing unit acquiring the multimedia stream from the multimedia application.
[0023] The first processing unit obtains the second capability information of the second processing unit, which is used to characterize whether the second processing unit supports decoding of audio streams;
[0024] Based on the second capability information, the first processing unit determines the second execution strategy for the second processing unit to process the audio stream;
[0025] Specifically, when the second capability information indicates support for decoding the audio stream, the second processing unit decodes and renders the audio stream.
[0026] Thus, in this embodiment, when the first processing unit includes a second system service, the second system service can obtain hardware capability information and select the most suitable decoding and rendering method for the audio stream, fully leveraging the advantages of the hardware. This enables the system service to better adapt to different devices, ensuring that the playback process runs normally on various hardware platforms and providing a relatively consistent user experience. Furthermore, the hardware performance differences between different devices are only perceived on the system side, not on the multimedia application side, ensuring consistent multimedia application experience across devices and platforms.
[0027] In one possible implementation, the second processing unit includes a fifth processing unit and a sixth processing unit, wherein the fifth processing unit is used to decode the audio stream and the sixth processing unit is used to render the decoded audio stream.
[0028] The fifth processing unit sends the decoded audio stream to the sixth processing unit.
[0029] Thus, in this embodiment, when the fifth processing unit is an audio decoding core and the sixth processing unit is an audio rendering core, the audio decoding core can send the decoded audio stream to the audio rendering core. The decoding and rendering processes are clearly separated, improving the processing efficiency of the audio stream. Furthermore, due to this functional separation, when upgrades or improvements to the audio decoding or rendering functions are needed, the audio decoding core and audio rendering core can be independently optimized and updated without affecting the functionality of other parts. This helps reduce system complexity and improves system maintainability and scalability.
[0030] In one possible implementation, the first processing unit receives a control operation on the multimedia stream, the control operation being used to control the playback of the multimedia stream, adjust the playback speed of the multimedia stream, adjust the playback progress, or adjust the playback volume of the multimedia stream.
[0031] In one possible implementation, the multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. During the output of the processing result by the second processing unit, the second processing unit uses the audio timestamp of the audio stream to synchronize the audio processing process and the video processing process. The second processing unit then outputs the synchronized processing result.
[0032] Thus, the embodiments of this application can perform the audio-visual synchronization process through the second processing unit. The audio-visual synchronization function can be quickly implemented with only simple configuration of the second processing unit, such as the intellectual property core, without relying on the central processing unit. This reduces the load on the central processing unit and the power consumption of the entire electronic device while ensuring synchronized audio and video playback. Furthermore, the embodiments of this application can decouple the audio stream, video stream, and audio-visual synchronization logic, providing flexibility in the audio-visual synchronization process.
[0033] In one possible implementation, the second processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit. During the synchronization of audio processing and video processing using the audio timestamp of the audio stream, the sixth processing unit synchronizes the audio timestamp to the fourth processing unit via the seventh processing unit. The fourth processing unit calculates the expected playback time of the video stream using the audio timestamp and the video timestamp. Based on the expected playback time, the fourth processing unit synchronizes the audio processing result and the video processing result.
[0034] Thus, in this embodiment of the application, during the audio-visual synchronization process in the second processing unit, the audio rendering core and the video rendering core can synchronize audio timestamps through a synchronization core. The synchronization core can coordinate the working rhythm of the audio rendering core and the video rendering core based on the audio timestamps. During playback, it ensures that the rendering speeds of audio and video match, avoiding stuttering or frame skipping caused by audio or video rendering being too fast or too slow, thereby providing a smoother playback experience.
[0035] In one possible implementation, the multimedia stream includes a video stream, the second processing unit includes a fourth processing unit and a seventh processing unit, the processing result includes a video processing result, and the second processing unit is configured to output the processing result during the process.
[0036] When the multimedia application supports audio playback, the seventh processing unit receives the audio timestamp from the first processing unit.
[0037] The seventh processing unit sends the audio timestamp to the fourth processing unit.
[0038] The fourth processing unit uses the audio timestamp and the video timestamp of the video stream to calculate the expected playback time of the video stream.
[0039] The fourth processing unit outputs the video processing results based on the expected playback time.
[0040] Thus, even when the multimedia application supports self-developed audio playback, the second processing unit, such as the intellectual property core, can synchronize the audio timestamps with the multimedia application, ensuring synchronized audio and video playback. This improves the stability and flexibility of the audio-visual synchronization process.
[0041] In one possible implementation, the multimedia stream includes an audio stream, and the second processing unit includes a sixth processing unit and a seventh processing unit.
[0042] When the multimedia application supports video playback, the sixth processing unit synchronizes the audio timestamp of the audio stream to the first processing unit through the seventh processing unit.
[0043] Thus, even when the multimedia application supports self-developed video playback, the second processing unit, such as the intellectual property core, can synchronize audio timestamps with the multimedia application in this embodiment. This ensures synchronized audio and video playback and improves the stability and flexibility of the audio-visual synchronization process.
[0044] In one possible implementation, the sixth processing unit synchronizes the audio timestamps according to a preset time interval, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed or progress of the multimedia stream, or adjusting the playback volume of the multimedia stream, the sixth processing unit synchronizes the audio timestamps.
[0045] Thus, the embodiments of this application can adapt the synchronization timing of audio timestamps according to different scenarios, ensuring normal multimedia playback while supporting speed adjustment, progress adjustment, and updating audio timestamps to ensure consistent audio and video output. This meets complex playback needs and enhances the user experience.
[0046] Secondly, embodiments of this application also provide an electronic device equipped with a multimedia application. The electronic device includes a first processing unit and a second processing unit. The first processing unit is configured to run the multimedia application.
[0047] The second processing unit is configured to acquire a multimedia stream from a multimedia application. The second processing unit is also configured to process the multimedia stream and output the processing result; wherein the processing includes decoding and rendering, or rendering.
[0048] In one possible implementation, the multimedia stream includes a video stream, and the first processing unit is configured to acquire first capability information of the second processing unit, the first capability information being used to characterize whether the second processing unit supports decoding and / or displaying the video stream.
[0049] The first processing unit is also configured to determine a first execution strategy for the second processing unit to process the video stream based on the first capability information.
[0050] In one possible implementation, if the first capability information indicates that the video stream is supported for decoding and display, the second processing unit is configured to decode and render the video stream.
[0051] When the first capability information indicates that it supports decoding the video stream but does not support displaying the video stream, the second processing unit is configured to decode the video stream, and the first processing unit is configured to control the second processing unit to render the video stream.
[0052] In one possible implementation, the first processing unit includes a rendering control module.
[0053] The rendering control module is configured to acquire the decoded video stream and send the decoded video stream to the second processing unit so that the second processing unit renders the video stream.
[0054] In one possible implementation, the multimedia stream includes an audio stream.
[0055] The first processing unit is configured to acquire second capability information of the second processing unit, the second capability information being used to characterize whether the second processing unit supports decoding of audio streams.
[0056] The first processing unit is also configured to determine a second execution strategy for the second processing unit to process the audio stream based on the second capability information.
[0057] Wherein, if the second capability information indicates support for decoding the audio stream, the second processing unit is configured to decode and render the audio stream.
[0058] In one possible implementation, the second processing unit includes a fifth processing unit and a sixth processing unit, wherein the fifth processing unit is used to decode the audio stream and the sixth processing unit is used to render the decoded audio stream.
[0059] The fifth processing unit is configured to send the decoded audio stream to the sixth processing unit.
[0060] In one possible implementation, the first processing unit is further configured to receive control operations on the multimedia stream, the control operations being used to control the playback of the multimedia stream, adjust the playback speed of the multimedia stream, adjust the playback progress, or adjust the playback volume of the multimedia stream.
[0061] In one possible implementation, the multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result.
[0062] The second processing unit is configured to synchronize the audio processing and video processing processes using the audio timestamps of the audio stream.
[0063] The second processing unit is also configured to output the synchronized processing result.
[0064] In one possible implementation, the second processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit;
[0065] The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit via the seventh processing unit;
[0066] The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream.
[0067] The fourth processing unit is also configured to synchronize the audio processing results and the video processing results according to the expected playback time.
[0068] In one possible implementation, the multimedia stream includes a video stream, the second processing unit includes a fourth processing unit and a seventh processing unit, and the processing result includes a video processing result.
[0069] When the multimedia application supports audio playback, the seventh processing unit is configured to receive an audio timestamp from the first processing unit.
[0070] The seventh processing unit is also configured to send audio timestamps to the fourth processing unit.
[0071] The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream.
[0072] The fourth processing unit is also configured to output video processing results based on the desired playback time.
[0073] In one implementation, the multimedia stream includes an audio stream, and the second processing unit includes a sixth processing unit and a seventh processing unit. If the multimedia application supports video playback, the sixth processing unit is configured to synchronize the audio timestamps of the audio stream to the first processing unit via the seventh processing unit.
[0074] In one possible implementation, the sixth processing unit is configured to synchronize audio timestamps according to a preset time interval, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed, playback progress, or adjusting the playback volume of the multimedia stream, the sixth processing unit is configured to synchronize audio timestamps.
[0075] The beneficial effects of the technical solution in the second aspect above can be referred to the beneficial effects of the technical solution in the first aspect.
[0076] Thirdly, embodiments of this application also provide a processing unit configured to acquire a multimedia stream from a multimedia application. The processing unit is further configured to process the multimedia stream and output a processing result; wherein the processing is decoding and rendering, or rendering.
[0077] In one possible implementation, the multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. The processing unit is configured to synchronize the audio processing process and the video processing process using the audio timestamp of the audio stream.
[0078] The processing unit is also configured to output the synchronized processing result.
[0079] In one possible implementation, the processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit.
[0080] The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit via the seventh processing unit. The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream. The fourth processing unit is also configured to synchronize the audio processing result and the video processing result based on the expected playback time.
[0081] The beneficial effects of the technical solution in the third aspect above can be referred to the beneficial effects of the technical solution in the first aspect.
[0082] Fourthly, this application also provides an electronic device, which includes a memory, one or more processors, and a display; the display is coupled to the memory and the processor; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the method described in the first aspect above.
[0083] Fifthly, this application also provides a computer-readable storage medium storing instructions that, when executed on an electronic device, cause the electronic device to perform the method described in the first aspect above.
[0084] Sixthly, this application also provides a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the method described in the first aspect above. Attached Figure Description
[0085] Figure 1 A schematic diagram of the hardware structure of a mobile phone provided in an embodiment of this application;
[0086] Figure 2This is a schematic diagram of the structure of a processing unit provided in an embodiment of this application;
[0087] Figure 3 A schematic diagram of the software structure of a mobile phone provided in an embodiment of this application;
[0088] Figure 4 A schematic diagram of the details page of a video application provided in an embodiment of this application;
[0089] Figure 5 An interactive illustration of a system service, multimedia application, and intellectual property core provided in this application embodiment. Figure 1 ;
[0090] Figure 6 An interactive illustration of a system service, multimedia application, and intellectual property core provided in this application embodiment. Figure 2 ;
[0091] Figure 7 A flowchart illustrating a configuration task provided in an embodiment of this application;
[0092] Figure 8 An interactive illustration of a system service, multimedia application, and intellectual property core provided in this application embodiment. Figure 3 ;
[0093] Figure 9 An interactive illustration of a system service, multimedia application, and intellectual property core provided in this application embodiment. Figure 4 ;
[0094] Figure 10 An interactive illustration of a synchronization process provided in this application embodiment. Figure 1 ;
[0095] Figure 11 An interactive illustration of a synchronization process provided in this application embodiment. Figure 2 ;
[0096] Figure 12 An interactive illustration of a synchronization process provided in this application embodiment. Figure 3 ;
[0097] Figure 13 A flowchart illustrating a data processing method provided in an embodiment of this application;
[0098] Figure 14 This is a schematic diagram of the structure of a mobile phone provided in an embodiment of this application. Detailed Implementation
[0099] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of this application, unless otherwise stated, " / " indicates that the objects before and after are in an "or" relationship. For example, A / B can represent A or B. "And / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. Furthermore, in the description of this application, unless otherwise stated, "multiple" refers to two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. In addition, in order to clearly describe the technical solutions of the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish the same or similar items with basically the same function and effect.
[0100] Those skilled in the art will understand that the terms "first," "second," etc., do not limit the quantity or order of execution, and that "first," "second," etc., are not necessarily different. Furthermore, in some embodiments of this application, words such as "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or description. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner for ease of understanding.
[0101] Furthermore, the device architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of device architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0102] With the development of terminal technology, electronic devices need to perform a large amount of computation on multimedia data and drive corresponding output devices during playback. This results in a high load on the processor of electronic devices, leading to significant power consumption. Reducing the power consumption of electronic devices during multimedia playback is a pressing issue that needs to be addressed.
[0103] To address the aforementioned technical problems, embodiments of this application provide an electronic device. The electronic device is equipped with a multimedia application and includes a first processing unit and a second processing unit. The first processing unit is configured to run the multimedia application. The second processing unit is configured to acquire a multimedia stream from the multimedia application. The second processing unit is further configured to process the multimedia stream and output a processing result; wherein the processing includes decoding and rendering, or rendering.
[0104] Thus, the embodiments of this application can utilize the second processing unit to process the multimedia stream. After the multimedia application provides the multimedia stream to the intellectual property core, it does not need to pay attention to the subsequent playback process of the multimedia stream.
[0105] On the one hand, when the first processing unit is the central processing unit (CPU) and the second processing unit is the intellectual property core (IPC), the CPU does not need to process the multimedia stream, thus offloading a large portion of the CPU's workload to the IPC. This eliminates the binding between the multimedia stream decoding and display processes and the CPU, reducing the CPU's workload.
[0106] Furthermore, the intellectual property core has a single function, allowing it to be designed for specific functions such as decoding, without requiring complex computing power. Therefore, when performing the same workload, the intellectual property core's processing unit and control logic are relatively simple, resulting in less power consumption. This reduces the overall power consumption of the electronic device. On the other hand, the multimedia streams proposed in this application include audio and video streams; different types of data streams can be processed by the intellectual property core, enhancing the device's processing flexibility during multimedia playback.
[0107] In some embodiments of this application, the above-described electronic device can be applied to a data processing method. The data processing method may include: a first processing unit running a multimedia application; a second processing unit acquiring a multimedia stream from the multimedia application; and the second processing unit processing the multimedia stream and outputting a processing result; wherein the processing involves decoding and rendering, or rendering.
[0108] In some embodiments, the electronic device may be a stationary device or a portable device. The operating system installed on the electronic device includes, but is not limited to, […]. Alternatively, other operating systems may be used. This application does not limit the specific type of electronic device or the operating system installed.
[0109] For example, electronic devices may include, but are not limited to, mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), wearable devices, artificial intelligence (AI) devices, and other similar devices.
[0110] Taking a mobile phone as an example, the electronic device in this embodiment can... Figure 1 This is achieved through mobile phones.
[0111] like Figure 1 As shown, the mobile phone 100 may include: a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, and an intellectual property core 196.
[0112] The aforementioned sensor module 180 may include sensors such as a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, and a bone conduction sensor 180M.
[0113] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the mobile phone 100. In other embodiments of this application, the mobile phone 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.
[0114] Processor 110 may include one or more processing units, such as application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more system-on-a-chip (SoCs).
[0115] The controller can serve as the central nervous system and command center of the mobile phone 100. Based on the instruction operation code and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.
[0116] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.
[0117] The wireless communication function of mobile phone 100 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor.
[0118] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover one or more communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.
[0119] The mobile communication module 150 can support 2G / 3G / 4G / 5G / future communication networks or more advanced communication standards, which is not limited in this application.
[0120] The wireless communication module 160 can provide solutions for wireless communication applications on the mobile phone 100, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.
[0121] The mobile phone 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.
[0122] The display screen 194 is used to display a user interface, etc. The display screen 194 includes a display panel. In some embodiments, the mobile phone 100 may include one or N displays screens 194, where N is a positive integer greater than 1.
[0123] For example, display screen 194 can show the playback interface of a multimedia application. The user can trigger the playback interface to initiate the playback of the multimedia stream in the multimedia application.
[0124] The mobile phone 100 can achieve shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.
[0125] The mobile phone 100 can achieve audio functions such as music playback and recording through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.
[0126] In some embodiments, Intellectual Property (IP) core 196 is an abbreviation for an integrated circuit core with intellectual property rights, or simply IP core. The IP core can be located in the System-on-Chip (SOC) of the mobile phone 100. An IP core is a pre-designed or even verified integrated circuit, device, or component with a specific function. Based on the hardware description level implementation of the IP core, it can be divided into three categories: soft core, hard core, and solid core. Soft core: primarily describes functionality and is submitted to the user in HDL (Hardware Description Language) text form. Solid core: primarily describes structure, falling between soft and hard cores, and is provided to the user in the form of a gate-level netlist. Hard core: based on physical description and verified through process technology, is provided to the user as a circuit physical structure mask diagram and a complete set of process documents. The second processing unit mentioned above in this application embodiment can be an IP core.
[0127] It should be noted that the intellectual property core refers to an integrated circuit design module with specific functions that has been repeatedly verified. It usually exists in the form of hardware description language (HDL) code, layout or other forms, and can be integrated into the on-chip system of the mobile phone 100 to realize specific functions, such as decoding core, rendering core and synchronization core.
[0128] In some embodiments of this application, the aforementioned electronic device may include a multi-core operating system. A multi-core operating system is an operating system that runs on a multi-core processing unit and is capable of utilizing the computing power of multiple cores. Here, multi-core refers to integrating two or more complete computing engines (kernels) within a single processing unit. It is understood that a multi-core operating system includes multiple processing cores. It should be noted that a processing core can also be referred to as a processing unit.
[0129] Multi-core operating systems employ a "divide and conquer" strategy. By dividing tasks, threaded applications can fully utilize multiple execution cores and execute more tasks within a given timeframe. A multi-core processing unit is a single chip (also known as a "silicon core") that can be directly inserted into a single processing unit socket. However, the operating system utilizes all relevant resources, treating each of its execution cores as a separate logical processing unit. By dividing tasks among two execution cores, a multi-core processing unit can execute more tasks within a given clock cycle.
[0130] Multi-core technology enables servers to process tasks in parallel, which previously required multiple processing units. Multi-core systems are more scalable and can incorporate more powerful processing performance into a more compact form factor that consumes less power and generates less heat from computation.
[0131] See Figure 2 , Figure 2 This is a schematic diagram of the structure of a processing unit provided in an embodiment of this application. Figure 2 As shown, the processor described above is a multi-core processor, which includes a main processing core and one or more slave processing cores coupled to the main processing core. Figure 3 Taking multiple slave processing cores as an example, this includes slave processing core 1 (Core1), slave processing core 2 (Core2), slave processing core 3 (Core3), slave processing core 4 (Core4), and so on. Each of these slave processing cores includes a configuration interface, namely configuration interface 1, configuration interface 2, configuration interface 3, configuration interface 4, and so on. The master processing core can configure and control the corresponding slave processing cores through these configuration interfaces. Optionally, the master processing core and one or more slave processing cores may reside in one chip. For example, the master processing core and one or more slave processing cores may reside in one chip, or the master processing core may reside in one chip, and some or all of the one or more slave processing cores may reside in another chip. This embodiment does not specifically limit this. It is understood that the master processing core and the multiple slave processing cores can communicate via a bus or other means. Figure 2 The connections shown do not impose restrictions on the coupling relationships between them.
[0132] In some embodiments of this application, the main processing core can be a first processing unit, such as a central processing unit, and the secondary processing core can be a second processing unit, such as an intellectual property core. Of course, the number of intellectual property cores can be one or more, and this application does not limit this.
[0133] During the playback of multimedia streams in multimedia applications, the intellectual property core can be used to process the multimedia stream, such as decoding and rendering. The intellectual property core can also be used to display the processing results of the multimedia stream to achieve playback.
[0134] Figure 3 This is a schematic diagram of the software structure of the mobile phone 100 according to an embodiment of this application.
[0135] The software architecture of Mobile Phone 100 can be divided into several layers. In some embodiments, from bottom to top, these layers are: kernel layer, system service layer, framework layer, and application layer. Layers communicate with each other through software interfaces. System functions can be tailored, added, or combined at the subsystem level depending on the deployment scenario of different device forms. Each subsystem can also be tailored, added, or combined at the functional level.
[0136] The kernel layer includes the kernel abstraction layer, the kernel subsystem, and the driver subsystem.
[0137] The system service layer comprises the core capabilities of the system, providing services to applications through the framework layer. This layer includes, but is not limited to, the following subsystems:
[0138] The system's basic capability subsystems provide the foundational capabilities for the operation, scheduling, and migration of distributed applications across multiple devices. These subsystems may include a distributed soft bus, distributed data management, distributed task scheduling, and the Ark multi-language runtime. They also include a multi-modal input subsystem, a graphics subsystem, a security subsystem, and an AI business subsystem.
[0139] Basic software service subsystems: These provide common and general software services. They may include an event notification subsystem, a telephone service subsystem, a multimedia subsystem, etc.
[0140] Enhanced Software Service Subsystem Set: Provides differentiated, enhanced software services tailored to different devices. This set may include proprietary business subsystems for smart screens, wearables, and IoT, among others.
[0141] Hardware service subsystem set: Provides hardware services. The hardware service subsystem set may include location service subsystem, user IAM (Identity and Access Management) subsystem, wearable proprietary hardware service subsystem, biometric identification subsystem, IoT proprietary hardware service subsystem, etc.
[0142] Distributed task scheduling enables distributed service management (discovery, synchronization, registration, and invocation), supporting remote startup, remote invocation, remote connection, and migration of applications across devices.
[0143] Distributed data management enables data synchronization, data storage, data sharing, and data access across all scenarios and devices.
[0144] First System Service: The first system service can also be called the video componentized playback system service. The video componentized playback system service is used to provide the relevant capabilities for playing video streams.
[0145] Second system service: The first system service can also be called the audio componentized playback system service, which is used to provide the relevant capabilities for playing audio streams.
[0146] In some examples, both the first and second system services support audio-visual synchronization during video and audio stream playback. Audio-visual synchronization refers to the precise timing match between sound and image, meaning that the playback progress of audio and video is consistent, with no instance of sound preceding or lagging behind the image.
[0147] In one possible implementation, the first system service and the second system service can support offloading the audio-visual synchronization process to the intellectual property core.
[0148] For example, an intellectual property core can use the audio timestamps of an audio stream and the video timestamps of a video stream for synchronization.
[0149] In another possible implementation, when the multimedia application is a third-party application and supports self-developed audio playback, the first system service also supports the audio-visual synchronization requirements of the third-party application.
[0150] For example, the first system service can synchronize the audio timestamps of audio streams from multimedia applications to the intellectual property core, which can then calculate the expected playback time of the video stream based on the audio timestamps to achieve audio-visual synchronization.
[0151] In another possible implementation, when the multimedia application is a third-party application and supports self-developed video playback, the second system service also supports the audio-visual synchronization requirements of the third-party application.
[0152] For example, the first system service can synchronize the audio timestamp of the audio stream from the intellectual property core to a multimedia application, which can then calculate the expected playback time of the video stream based on the audio timestamp to achieve audio-visual synchronization.
[0153] In some embodiments, the first system service and the second system service support offloading the decoding and rendering process of the multimedia stream to the intellectual property core. The first system service and the second system service also support offloading the display process of the multimedia stream to the intellectual property core.
[0154] The framework layer provides application programming interfaces (APIs) and programming frameworks for applications in the application layer. The framework layer includes: the ArkUI framework (which provides a complete infrastructure for UI development of system applications, including UI functions such as components, layouts, animations, and interactive events, as well as a real-time interface preview tool), the user application framework, and the Ability framework (an Ability is a lightweight application; the Ability framework schedules and manages the operation and lifecycle of Abilities). Different devices may have different operating systems, and the APIs they support may also differ.
[0155] The framework layer can also include componentized video playback interfaces and componentized audio playback interfaces. Multimedia applications can achieve smooth playback of multimedia streams by calling the componentized video playback interfaces and / or componentized audio playback interfaces.
[0156] Applications can include system apps and extended / third-party apps. System apps can include the desktop, control bar, settings, contacts, input method, gallery, etc., while extended / third-party apps can include social apps, travel apps, etc.
[0157] The data processing method proposed in this application can be applied to multimedia playback scenarios.
[0158] In some examples, users can use multimedia applications on their mobile phones to experience audio and video playback services. Multimedia applications include audio applications, video applications, educational applications, conferencing applications, and live streaming applications, etc. This application does not limit the specific application type corresponding to the multimedia application.
[0159] For example, consider a multimedia application, specifically a video application. See [link / reference] Figure 4 In (A), the phone can display the video application's details page. Users can select the video they want to watch on the details page, such as First Video. See also... Figure 4 In (B), the mobile phone can respond to the user's selection operation by displaying the playback details page of the first video and playing the first video. Specifically, the mobile phone can obtain the video stream of the first video from the server and display the playback details page of the first video based on the video stream.
[0160] It should be noted that during the playback of the first video, the phone supports user control over the display area of the video, such as controlling full-screen playback, adjusting playback quality, enabling screen mirroring, and controlling brightness. The phone also supports playback control of the first video, including adjusting playback speed, volume, whether playback has started, and switching between videos.
[0161] In some embodiments of this application, the multimedia playback scenarios described above are not limited to controlling the playback of a multimedia stream (such as playing a multimedia stream for the first time or playing a multimedia stream after pausing), but may also include scenarios such as adjusting the playback speed or progress of the multimedia stream, adjusting the playback volume of the multimedia stream, and adjusting the playback effects of the multimedia. It should be noted that the embodiments of this application do not limit specific application scenarios.
[0162] In some embodiments of this application, the first processing unit includes an operating system that needs to obtain a multimedia stream from a multimedia application. The second processing unit can obtain the multimedia stream from the multimedia application from the first processing unit.
[0163] In one possible implementation, see [link to implementation details]. Figure 5In step (A), the system service sends a request to the multimedia application to retrieve its multimedia stream. The multimedia application then receives the multimedia stream from the system service. It should be noted that the multimedia application can also proactively send multimedia streams to the system service.
[0164] It should be noted that the raw multimedia data acquired by multimedia applications can be an interleaved data stream, meaning that the audio stream and the data stream are interleaved together. Multimedia applications can decapsulate the raw multimedia data to obtain a non-interleaved multimedia stream, which includes both audio and video streams.
[0165] In another possible implementation, the second processing unit can obtain multimedia data of the multimedia application from the first processing unit. The second processing unit may also include a decapsulation kernel. See also Figure 5 In (B), a multimedia application can respond to a retrieval request by sending raw multimedia data to a system service. The system service can then send the multimedia data to the intellectual property core, where a decapsulation core can decapsulate the multimedia data to obtain a non-interleaved multimedia stream.
[0166] In another possible implementation, the second processing unit can also obtain multimedia data from the server, and the second processing unit can decapsulate the multimedia data to obtain a multimedia stream.
[0167] For example, a multimedia application can send the media identifier of a multimedia stream to a system service, and the system service can then send the media identifier of the multimedia stream to the intellectual property core. See also Figure 5 In (C), the intellectual property core may include a download core and a decompression core. The download core can obtain multimedia data from the server based on the media identifier of the multimedia stream. The download core can synchronize the multimedia data to the decompression core, and the decompression core can decompress the multimedia data to obtain a non-interleaved multimedia stream. The multimedia may include video streams and audio streams. It should be noted that the embodiments of this application do not specifically limit the implementation method of the intellectual property core obtaining the multimedia stream.
[0168] In some embodiments, see continue to see Figure 5 In (A), the first processing unit may further include a synchronization module. The synchronization module is used to synchronize the audio timestamps of the audio stream to enable subsequent audio-visual synchronization.
[0169] In some embodiments of this application, the first processing unit includes system services, which may include a first system service and a second system service. Specifically, the central processing unit includes an application layer and a system service layer. The application layer includes multimedia applications, and the system service layer includes a video componentized playback interface and a first system service, and / or an audio componentized playback interface and a second system service.
[0170] See Figure 6 (A) and Figure 6 In embodiment (B) of this application, a video componentized playback interface and an audio componentized playback interface are provided to support playback services of multimedia applications. Multimedia applications can call a first system service through the video componentized playback interface, and multimedia applications can also call a second system service through the audio componentized playback interface. The first system service can also call back to the multimedia application based on the video componentized playback interface, and the second system service can also call back to the multimedia application based on the audio componentized playback interface, to achieve the transmission of multimedia streams.
[0171] Thus, the embodiments of this application can provide a componentized playback interface to realize multimedia stream playback functionality and its customization. The provided componentized playback interface can be reused, improving development efficiency. Furthermore, although the underlying multimedia stream playback implementation methods differ across operating systems, a unified componentized playback interface can be defined to implement playback functionality on different system platforms, achieving cross-platform playback.
[0172] In some embodiments of this application, during the playback of a media stream, the multimedia stream is typically encoded data, and the multimedia stream needs to undergo decoding, rendering, and output processes.
[0173] For video streams, hardware decoding and hardware display (also known as hardware pass-through display) can be performed on the video stream, or hardware decoding and software display can be performed on the video stream, or software decoding and software display can be performed on the video stream.
[0174] Hardware decoding can be understood as the intellectual property core receiving the video stream from the system service and decoding it. Software decoding can be understood as the system service decoding the video stream.
[0175] Hardware display can be understood as: the intellectual property rights check the decoded video stream, render it, and then display it.
[0176] Software-based video delivery can be understood as follows: On the system server side, the system service receives the decoded video stream from the intellectual property core and controls the intellectual property core to perform subsequent rendering and delivery. On the intellectual property core side, the intellectual property core is controlled by the system service to perform subsequent rendering and delivery. In other words, hardware-based video delivery involves the intellectual property core itself performing rendering and delivery. Software-based video delivery involves the system service controlling the intellectual property core to perform rendering and delivery.
[0177] For audio streams, hardware decoding and hardware output can be performed, or software decoding and software output can be performed. The hardware decoding, software decoding, software output, and hardware output of audio streams are the same as those for video streams, and will not be repeated here.
[0178] The following explanation uses the example of a mobile phone being able to perform hardware decoding and hardware display of multimedia streams.
[0179] The system service can send multimedia streams from multimedia applications to the intellectual property core. The intellectual property core processes the multimedia streams, obtaining the results through decoding and rendering. The intellectual property core then outputs the processing results.
[0180] For example, see [link to example]. Figure 5 In (A), the second processing unit may include a first target processing unit and a second target processing unit. The first target processing unit may be a decoding core, and the second target processing unit may be a rendering core. The decoding core can decode the multimedia stream and send the decoded multimedia stream to the rendering core. The rendering core renders the decoded multimedia stream to obtain the processing result. The rendering core can send the processing result to the output device. It should be noted that... Figure 5 (B) and Figure 5 The decoding and rendering cores in (C) function the same as described above. Figure 5 The description of (A) in the text will not be repeated here.
[0181] Furthermore, the second processing unit also includes a seventh processing unit, such as a synchronization core, which is used to synchronize the audio timestamps of the audio stream to achieve subsequent audio-visual synchronization.
[0182] In some embodiments of this application, the first target processing unit may include a third processing unit and a fifth processing unit, and the second target processing unit may include a fourth processing unit and a sixth processing unit. That is, the intellectual property core can process the audio stream and the video stream independently. The decoding core includes an audio decoding core and a video decoding core, and the rendering core may include an audio rendering core and a video rendering core. Specifically, the third processing unit is a video decoding core, the fourth processing unit is a video rendering core, the fifth processing unit is an audio decoding core, and the sixth processing unit is an audio rendering core.
[0183] See also Figure 6In (A), the first system service can send a video stream to the video decoding core. After decoding the video stream, the video decoding core obtains the decoded video stream. The video decoding core can then send the decoded video stream to the video rendering core. The video rendering core renders the decoded video stream to obtain the processing result. The video rendering core sends the processing result to the display screen so that the display screen shows the image corresponding to the video stream.
[0184] See also Figure 6 In section (B), the second system service can send an audio stream to the audio decoding core. After decoding the audio stream, the audio decoding core obtains the decoded audio stream. The audio decoding core can then send the decoded audio stream to the audio rendering core. The audio rendering core renders the decoded audio stream to obtain the processing result. The audio rendering core sends the processing result to the speaker so that the speaker can play the sound corresponding to the audio stream.
[0185] The data processing method proposed in the embodiments of this application will be described in detail below.
[0186] In some embodiments of this application, the mobile phone may play only video data while playing multimedia data. For example, some animations or long videos do not require accompanying audio. Therefore, when the multimedia stream is a video stream, the mobile phone can process the video stream, obtain the processing result, and display the result.
[0187] In some examples, before the second processing unit acquires the multimedia stream of the multimedia application, the first processing unit may acquire first capability information of the second processing unit. This first capability information characterizes whether the second processing unit supports decoding and / or displaying the video stream. Based on the first capability information, the first processing unit may determine a first execution strategy for the second processing unit to process the video stream.
[0188] In other words, before processing the video stream, the mobile phone needs to query the capabilities of the intellectual property core from the hardware interface and adaptively configure the execution strategy for video stream decoding, rendering, and display tasks.
[0189] For example, after a user performs control operations on a video stream (such as playing the video), the multimedia application can use the video component-based playback interface to call the first system service to implement the playback process for the video stream. The first system service can query the capabilities of the intellectual property core from the hardware interface and adaptively configure the execution strategies for video stream decoding, rendering, and display tasks.
[0190] Furthermore, if the first capability information indicates that the video stream is supported for decoding and display, the second processing unit decodes and renders the multimedia stream.
[0191] When the first capability information indicates that it supports decoding the video stream but does not support displaying the video stream, the second processing unit decodes the multimedia stream as a video stream, and the first system service controls the second processing unit to render the video stream.
[0192] For example, the first processing unit includes a first system service, and the second processing unit includes an intellectual property core. The first system service determines a first execution strategy based on the first capability information.
[0193] See Figure 7 In step (A), the first system service queries the first capability information of the intellectual property core and determines whether hardware decoding is supported based on the first capability information. If the intellectual property core supports hardware decoding, the hardware decoding task of the intellectual property core is configured. After configuring the hardware decoding task of the intellectual property core, the first system service determines whether the intellectual property core supports hardware display based on the first capability information. If the intellectual property core supports hardware display, the hardware display task of the intellectual property core is configured. If the intellectual property core does not support hardware display, the software display task of the first system service is configured.
[0194] Similarly, if the intellectual property core determines that it does not support hardware decoding, the first system service configures the software decoding task for the first system service. After configuring the software decoding task for the intellectual property core, the first system service configures the software display task for the first system service.
[0195] Thus, this embodiment of the application can obtain hardware capability information and select the most suitable decoding and rendering method for the video stream, fully leveraging the advantages of the hardware. Simultaneously, given the varying hardware performance of different devices, developing strategies based on hardware capability information enables system services to better adapt to different devices, ensuring the normal operation of the playback process and providing a relatively consistent user experience across various hardware platforms. Furthermore, the differences in hardware performance between different devices are only perceptible at the system level, not at the multimedia application level, guaranteeing a consistent experience for multimedia applications across devices and platforms.
[0196] In some embodiments of this application, after determining a first execution strategy, the first system service obtains a video stream from a multimedia application and calls corresponding resources to process the video stream based on the first execution strategy.
[0197] In one possible implementation, where the first capability information indicates support for decoding and displaying the video stream, the first execution strategy is used to instruct the intellectual property authority to decode and render the video stream.
[0198] For example, after the first system service obtains the video stream, it sends the video stream to the intellectual property core. The intellectual property core can decode the video stream to obtain a decoded video stream. The intellectual property core then renders the decoded video stream to obtain the processing result.
[0199] Specifically, the intellectual property core includes a video decoding core, a video rendering control core, and a video rendering core. The video decoding core can send the decoded video stream to the video rendering core. See also Figure 8 In step (A) of the first system service, after the data input module acquires the video stream, it sends the video stream to the video decoding core. The video decoding core decodes the video stream to obtain the decoded video stream. The video decoding core then sends the decoded video stream to the video rendering core through the video rendering control core. The video rendering core renders the decoded video stream to obtain the processing result. The video rendering core sends the processing result to the display screen so that the display screen shows the image corresponding to the video stream.
[0200] Thus, on the one hand, the embodiments of this application can offload the video stream processing to the intellectual property core, eliminating the need for a central processing unit. On the other hand, the intellectual property core has a single function, allowing it to be designed for specific functions such as decoding, without requiring complex computing power. Therefore, when performing the same workload, the intellectual property core's processing unit and control logic are relatively simple, resulting in less power consumption. This reduces the overall power consumption of the electronic device.
[0201] On the other hand, the decoupling of the above-mentioned decoding and rendering processes allows the intellectual property core to more flexibly optimize the decoding and rendering processes separately according to the characteristics and capabilities of the hardware, thereby achieving better performance and lower power consumption.
[0202] In another possible implementation, where the first capability information indicates that decoding of the video stream is not supported but display of the video stream is supported, the first execution strategy is used to instruct the rendering of the video stream to be controlled by the intellectual property core.
[0203] Specifically, see Figure 8 In step (B), after the data input module in the first system service acquires the video stream, it sends the video stream to the software decoding module in the first system service. The software decoding module decodes the video stream to obtain the decoded video stream. The software decoding module then sends the decoded video stream to the video rendering control core. Next, the video rendering control core sends the decoded video stream to the video rendering core. The video rendering core renders the decoded video stream to obtain the processing result. The video rendering core sends the processing result to the display screen so that the display screen shows the image corresponding to the video stream.
[0204] Thus, in this embodiment, the rendering process of the video stream can be offloaded to the intellectual property core, eliminating the need for the central processing unit (CPU). Utilizing the intellectual property core for rendering results in lower power consumption. This reduces the CPU load and consequently lowers the overall power consumption of the electronic device.
[0205] In another possible implementation, where the first capability information indicates that decoding of the video stream is supported but display of the video stream is not, the first execution strategy is used to instruct the intellectual property authority to encode the video stream and also to instruct the first system service to control the intellectual property authority to render the video stream.
[0206] Specifically, the first processing unit includes a rendering control module. The rendering control module can acquire the decoded video stream and send it to the second processing unit, enabling the second processing unit to render the video stream. In other words, while the first processing unit is controlling the second processing unit to render the video stream, the rendering control module can send the decoded video stream to the fourth processing unit (video rendering core) within the second processing unit, thereby controlling the fourth processing unit to render the video stream.
[0207] See Figure 8 In step (C), after the data input module in the first system service obtains the video stream, it sends the video stream to the video decoding core. The video decoding core decodes the video stream to obtain the decoded video stream. The video decoding core sends the decoded video stream to the rendering control module in the first system service. The rendering control module sends the decoded video stream to the video rendering core, which renders the decoded video stream to obtain the processing result. The video rendering core sends the processing result to the display screen so that the display screen shows the image corresponding to the video stream.
[0208] Thus, in this embodiment, the decoding and rendering processes of the video stream can be offloaded to the intellectual property core. Although the central processing unit (CPU) participates in controlling the rendering process, the actual decoding and rendering are performed by the intellectual property core. Using the intellectual property core for the rendering process results in lower power consumption. This eliminates the complete binding of the decoding, rendering, and display processes to the CPU. While reducing the CPU load, the power consumption of the entire electronic device is also reduced.
[0209] In some embodiments of this application, during multimedia data playback, the mobile phone may play only audio data, such as a song. Therefore, when the multimedia stream is an audio stream, the mobile phone can process the audio stream, obtain the processing result, and then play the processed result.
[0210] In some examples, before the second processing unit acquires the multimedia stream of the multimedia application, the first processing unit may acquire second capability information of the second processing unit. This second capability information characterizes whether the second processing unit supports decoding the audio stream. Based on the second capability information, the first processing unit determines a second execution strategy for the second processing unit to process the audio stream.
[0211] Wherein, if the second capability information indicates support for decoding the audio stream, the second processing unit can decode and render the audio stream.
[0212] In other words, before processing the audio stream, the mobile phone needs to query the capabilities of the intellectual property core from the hardware interface and adaptively configure the execution strategy for audio stream decoding, rendering, and display tasks.
[0213] For example, a multimedia application can utilize an audio component-based playback interface to call a second system service to implement a playback process for an audio stream. The second system service can query the capabilities of the intellectual property core from the hardware interface and adaptively configure the execution strategies for audio stream decoding, rendering, and playback tasks.
[0214] Specifically, the second system service can obtain the second capability information of the intellectual property core, which is used to characterize whether the intellectual property core supports decoding audio streams. Based on the second capability information, the second system service determines the second execution strategy.
[0215] See Figure 7 In step (B), the second system service queries the second capability information of the intellectual property core and determines whether hardware decoding is supported based on the second capability information. If it is determined that the intellectual property core supports hardware decoding, the hardware decoding task and hardware output task of the intellectual property core are configured. If it is determined that the intellectual property core does not support hardware decoding, the software decoding task and software output task of the second system service are configured.
[0216] Thus, this embodiment of the application can obtain hardware capability information and select the most suitable decoding and rendering method for the audio stream, fully leveraging the advantages of the hardware. This enables system services to better adapt to different devices, ensuring that the playback process runs normally on various hardware platforms and providing a relatively consistent user experience. Furthermore, the hardware performance differences between different devices are only perceived on the system side, not on the multimedia application side, ensuring a consistent experience for multimedia applications across devices and platforms.
[0217] In some embodiments of this application, after determining the second execution strategy, the second system service can obtain an audio stream from the multimedia application and call corresponding resources to process the audio stream based on the second execution strategy.
[0218] For example, when the second capability information indicates support for decoding the audio stream, the second execution strategy is used to instruct the intellectual property authority to decode and render the audio stream.
[0219] For example, after the second system service obtains the audio stream, it sends the audio stream to the intellectual property core. The intellectual property core can decode the audio stream to obtain the decoded audio stream. The intellectual property core then renders the decoded audio stream to obtain the processing result.
[0220] In one possible implementation, see [link to implementation details]. Figure 9 In step (A) of the second system service, after the data input module obtains the audio stream, it sends the audio stream to the audio decoding core. The audio decoding core decodes the audio stream to obtain the decoded audio stream. The audio decoding core sends the decoded audio stream to the audio rendering core. The audio rendering core renders the decoded audio stream to obtain the processing result. The audio rendering core sends the processing result to the speaker so that the speaker plays the sound corresponding to the audio stream.
[0221] Thus, on the one hand, the embodiments of this application can offload the audio stream decoding and rendering process to the intellectual property core, eliminating the need for a central processing unit (CPU). On the other hand, utilizing the computing power of the intellectual property core to complete the same workload requires relatively simple processing units and control logic, resulting in lower power consumption. This not only reduces the load on the CPU but also lowers the overall power consumption of the electronic device.
[0222] In another possible implementation, see Figure 9 In section (B), after the data input module in the second system service obtains the audio stream, it sends the audio stream to the software decoding module. The software decoding module decodes the audio stream to obtain the decoded audio stream. The software decoding module then sends the decoded audio stream to the rendering control module. The rendering control module can then send the decoded audio stream to the audio rendering core. The audio rendering core renders the decoded audio stream to obtain the processing result. The audio rendering core sends the processing result to the speaker so that the speaker plays the sound corresponding to the audio stream.
[0223] Based on the above, the first processing unit can receive control operations on the multimedia stream. These control operations are used to control the playback of the multimedia stream, adjust its playback speed, playback progress, or adjust its playback volume. The first processing unit (such as a system service) can respond to these control operations by configuring tasks based on the source and capability information of the multimedia stream. The source information includes the media content information of the multimedia stream (such as data format, sampling rate, number of channels, resolution, and frame rate) and the encoding information of the multimedia stream (such as encoding format and encryption method). In other words, whenever a user controls the playback of multimedia, the system service needs to obtain the capability information and configure tasks based on the source information of the currently playing multimedia stream.
[0224] It is understood that, in the process of acquiring capability information, the aforementioned system services can receive control operations on multimedia streams and acquire capability information in response to these control operations. For example, the first system service receives a control operation on a video stream and acquires first capability information in response to the control operation. As another example, the second system service receives a control operation on an audio stream and acquires second capability information in response to the control operation.
[0225] In some embodiments of this application, the system service can detect the multimedia type in the multimedia stream after receiving it.
[0226] For example, when the multimedia stream includes an audio stream, the second system service can obtain second capability information. When the multimedia stream includes a video stream, the first system service can obtain first capability information. When the multimedia stream includes both audio and video streams, the first system service can obtain first capability information, and the second system service can obtain second capability information.
[0227] Thus, the multimedia application in this application embodiment can call different system services through different componentized audio and video playback interfaces to achieve independent audio stream unloading and playback, and video stream unloading and display. Of course, when the multimedia stream includes both audio and video streams, the first and second system services can jointly execute audio stream unloading and playback, and video stream unloading and display. This satisfies diverse needs for multimedia playback and improves the flexibility of multimedia playback.
[0228] In some embodiments of this application, during multimedia data playback, the mobile phone can play video and audio data, such as animations and long videos that match the sound. Therefore, when the multimedia stream includes both audio and video streams, the mobile phone can process the audio and video streams, obtain processing results, and output the processing results.
[0229] It should be noted that when the multimedia stream includes audio stream and video stream, the specific process of the mobile phone processing the audio stream and video stream can be referred to the above embodiments, and will not be repeated here.
[0230] When multimedia applications support componentized video and audio playback within the operating system, the intellectual property core can process audio and video streams to obtain the processing results. The intellectual property core can also perform audio-visual synchronization before outputting the processing results.
[0231] In some embodiments of this application, the processing results include audio processing results and video processing results. The second processing unit can use the audio timestamp of the audio stream to synchronize the audio processing process and the video processing process. The second processing unit can also output the synchronized processing results.
[0232] For example, the second processing unit is an intellectual property core, which includes an audio rendering core, a video rendering core, and a synchronization core. The audio rendering core can synchronize audio timestamps to the video rendering core through the synchronization core. The video rendering core uses the audio timestamps and the video timestamps of the video stream to calculate the expected playback time of the video stream. The video rendering core can also synchronize the audio processing results and the video processing results according to the expected playback time.
[0233] See Figure 10 In the process, the first system service synchronizes the video stream to the video decoding core, which decodes the video stream to obtain the decoded video stream. The second system service synchronizes the audio stream to the audio decoding core, which decodes the audio stream to obtain the decoded audio stream. The audio rendering core renders the decoded audio stream to obtain the audio processing result. The audio rendering core synchronizes the audio timestamp to the video rendering core via a synchronization core. The video rendering core uses the audio timestamp and the video stream's video timestamp to calculate the expected playback time of the video stream. Based on the expected playback time, the video rendering core synchronizes the audio processing result and the video processing result. For example, the video rendering core renders and displays the decoded video stream based on the expected playback time.
[0234] In one possible implementation, the video rendering kernel can obtain the audio timestamp and the system time.
[0235] The audio timestamp marks the time position of each frame of audio data in the audio stream. The system time is the operating system's current real-time time. The video timestamp marks the time position of each frame of video data in the video stream.
[0236] The video rendering kernel can calculate the expected playback time based on the audio timestamp, video timestamp, and system time. The sum of the difference between the video timestamp and the audio timestamp and the system time equals the expected playback time.
[0237] It should be noted that audio-visual synchronization is required during the decoding of each frame of a video stream. This application does not limit the specific method for calculating the expected playback time.
[0238] Thus, this embodiment of the application can perform audio-visual synchronization through the intellectual property core. Audio-visual synchronization can be quickly achieved with simple configuration of the intellectual property core, without relying on the central processing unit. While ensuring synchronized audio and video playback, it reduces the load on the central processing unit and lowers the power consumption of the entire electronic device. Furthermore, this embodiment of the application can decouple the audio stream, video stream, and audio-visual synchronization logic, providing flexibility in the audio-visual synchronization process.
[0239] In some embodiments of this application, the multimedia application may support component-based playback of video or audio components within the operating system. That is, the multimedia application may be a third-party application that supports self-developed audio or video playback.
[0240] In one possible implementation, where the multimedia application supports video component playback in the operating system, the intellectual property core needs to actively obtain the audio timestamp of the multimedia application and perform an audio-visual synchronization process based on the audio timestamp.
[0241] In other words, when multimedia applications support audio playback, the synchronization core can receive audio timestamps from the first system service. The synchronization core can then send these audio timestamps to the video rendering core. The video rendering core uses the audio timestamps and the video stream's timestamps to calculate the expected playback time of the video stream. Based on the expected playback time, the video rendering core outputs the video processing result.
[0242] Specifically, if the multimedia application supports audio playback, see [link to relevant documentation]. Figure 11 The first system service includes a synchronization module, which sends update callbacks to the adaptation module in the multimedia application. The adaptation module responds to the update callbacks by sending audio timestamps to the synchronization module. The synchronization module can then synchronize the audio timestamps to the synchronization core.
[0243] The synchronization core receives the audio timestamp from the synchronization module in the first system service. The synchronization core then sends the audio timestamp to the video rendering core. The video rendering core uses the audio timestamp and the video timestamp of the video stream to calculate the expected playback time of the video stream. The video rendering core outputs the video processing result based on the expected playback time.
[0244] It should be noted that the process by which the video rendering kernel calculates the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream can be found in the above embodiments, and will not be repeated here.
[0245] See also Figure 11When a multimedia application supports audio playback, its self-developed audio playback module can utilize the system audio rendering module in the operating system to control the audio rendering core for rendering. For example, the system audio rendering module sends the decoded audio stream to the audio rendering core and then outputs the audio processing result. This application does not limit the specific implementation method of a multimedia application supporting self-developed audio playback.
[0246] Thus, even when the multimedia application supports self-developed audio playback, the intellectual property core can be used to synchronize audio timestamps with the multimedia application, ensuring synchronized audio and video playback. This improves the stability and flexibility of the audio-visual synchronization process.
[0247] In one possible implementation, where the multimedia application supports componentized audio playback within the operating system, the intellectual property core needs to actively synchronize the audio timestamps of the audio stream to assist in the self-developed video playback process of the multimedia application. Specifically, the audio rendering core synchronizes the audio timestamps of the audio stream to the second system service via the synchronization core.
[0248] Specifically, when multimedia applications support audio playback, the first system service includes a synchronization module. See also Figure 12 The second system service synchronizes the audio stream to the audio decoding core, which decodes the audio stream to obtain the decoded audio stream. The audio rendering core renders the decoded audio stream to obtain the audio processing result. The audio rendering core synchronizes the audio timestamp of the audio stream to the synchronization module in the second system service through the synchronization core.
[0249] Therefore, when the multimedia application subsequently performs audio-visual synchronization, the adaptation module in the multimedia application can obtain the audio timestamp from the synchronization module. The adaptation module can synchronize the audio timestamp to the self-developed video playback module, which can then synchronize the audio timestamp to the video rendering core through the system video rendering module. This enables the video rendering core to perform subsequent synchronization processes and output the video processing results to the display screen.
[0250] It should be noted that during the above audio-visual synchronization process, before the next audio timestamp synchronization, all video frames to be rendered are synchronized with the latest synchronized audio timestamp.
[0251] Thus, even when the multimedia application supports self-developed video playback, the intellectual property core can be used to synchronize audio timestamps with the multimedia application in this embodiment, ensuring synchronized audio and video playback and improving the stability and flexibility of the audio-visual synchronization process.
[0252] In some embodiments of this application, as described above, the user can perform control operations on the multimedia stream. These control operations are used to control the playback of the multimedia stream, adjust its playback speed, playback progress, or adjust its playback volume.
[0253] In other words, users can initiate playback of a video or audio stream, resume playback after pausing, adjust playback speed, adjust playback progress, or adjust volume. It should be noted that after a user performs a control operation on the multimedia stream, the phone can respond to that operation and execute the technical steps provided in the above embodiments.
[0254] Thus, this application embodiment supports independent playback control, speed control, progress control, and volume control for audio or video streams. Of course, when the multimedia stream includes both audio and video streams, they can also be controlled together. This adapts to different scenario requirements. Furthermore, the consistent execution logic across different devices and hardware platforms ensures that the multimedia application displays consistent functionality, avoiding functional abnormalities due to device differences and improving stability.
[0255] In some embodiments of this application, during the audio-visual synchronization process, the audio rendering core synchronizes the audio timestamp according to a preset time interval, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed or progress of the multimedia stream, or adjusting the playback volume of the multimedia stream, the audio rendering core synchronizes the audio timestamp.
[0256] For example, the audio rendering core can synchronize audio timestamps at fixed intervals, such as synchronizing audio timestamps every frame.
[0257] As another example, if a multimedia application supports a self-developed playback process, the first system service can also obtain audio timestamps from the multimedia application at fixed intervals.
[0258] As another example, in the case of controlling playback as described above, the audio rendering core can also synchronize audio timestamps.
[0259] Thus, the embodiments of this application can adapt the synchronization timing of audio timestamps according to different scenarios, ensuring normal multimedia playback while supporting speed adjustment, progress adjustment, and updating audio timestamps to ensure consistent audio and video output. This meets complex playback needs and enhances the user experience.
[0260] It should be noted that the rendering core, decoding core, control core, and synchronization core involved in the aforementioned intellectual property cores can also be referred to as rendering devices, decoding devices, control devices, and synchronization devices. This application does not specifically define their names in the embodiments.
[0261] Figure 13 A flowchart of a data processing method is shown.
[0262] S1301, The first processing unit runs multimedia applications.
[0263] S1302, The second processing unit acquires the multimedia stream of the multimedia application.
[0264] S1303, the second processing unit processes the multimedia stream and outputs the processing result. The processing may involve decoding and rendering, or rendering.
[0265] When the first processing unit is the central processing unit (CPU) and the second processing unit is the intellectual property core (IPC), the CPU does not need to process multimedia streams, thus offloading a significant portion of the CPU's workload to the IPC. This eliminates the binding between the multimedia stream decoding and display processes and the CPU, reducing the CPU's load.
[0266] In some embodiments, the multimedia stream includes a video stream, and the method further includes, before the second processing unit acquires the multimedia stream of the multimedia application:
[0267] The first processing unit obtains the first capability information of the second processing unit, which is used to characterize whether the second processing unit supports decoding and / or displaying the video stream.
[0268] Based on the first capability information, the first processing unit determines the first execution strategy for the second processing unit to process the video stream.
[0269] In some embodiments, the multimedia stream includes an audio stream. The first processing unit acquires second capability information of the second processing unit, the second capability information being used to characterize whether the second processing unit supports decoding the audio stream;
[0270] Based on the second capability information, the first processing unit determines the second execution strategy for the second processing unit to process the audio stream;
[0271] Specifically, if the second capability information indicates support for decoding the audio stream, the second processing unit decodes and renders the audio stream. The process for determining the execution strategy described above can be found in [reference needed]. Figure 7 The specific embodiments shown are as follows.
[0272] Thus, in this embodiment, the first processing unit can utilize the acquired hardware capability information to select the most suitable decoding and rendering method for the multimedia stream, fully leveraging the advantages of the hardware. This ensures that the playback process runs normally on various hardware platforms and provides a relatively consistent user experience. Furthermore, the hardware performance differences between different devices are only perceived on the system side, not on the multimedia application side, guaranteeing a consistent experience for multimedia applications across devices and platforms.
[0273] In some embodiments, the processing result includes audio processing result and video processing result. The second processing unit uses the audio timestamp of the audio stream to synchronize the audio processing process and the video processing process; the second processing unit outputs the synchronized processing result. The above audio-visual synchronization process can be referred to... Figure 10 The specific embodiments shown are as follows.
[0274] Thus, the embodiments of this application can perform the audio-visual synchronization process through the second processing unit. The audio-visual synchronization function can be quickly implemented with only simple configuration of the second processing unit, such as the intellectual property core, without relying on the central processing unit. This reduces the load on the central processing unit and the power consumption of the entire electronic device while ensuring synchronized audio and video playback. Furthermore, the embodiments of this application can decouple the audio stream, video stream, and audio-visual synchronization logic, providing flexibility in the audio-visual synchronization process.
[0275] In some embodiments, the multimedia stream includes a video stream, and the second processing unit includes a fourth processing unit and a seventh processing unit.
[0276] When the multimedia application supports audio playback, the seventh processing unit receives the audio timestamp from the first processing unit. The seventh processing unit then sends the audio timestamp to the fourth processing unit. The fourth processing unit uses the audio timestamp and the video timestamp of the video stream to calculate the expected playback time of the video stream. The fourth processing unit outputs the video processing result based on the expected playback time. The above audio-visual synchronization process can be referenced... Figure 11 The specific embodiments shown are as follows.
[0277] Thus, even when the multimedia application supports self-developed audio playback, the second processing unit, such as the intellectual property core, can synchronize the audio timestamps with the multimedia application, ensuring synchronized audio and video playback. This improves the stability and flexibility of the audio-visual synchronization process.
[0278] In some embodiments, the second processing unit includes a sixth processing unit and a seventh processing unit. When the multimedia application supports video playback, the sixth processing unit synchronizes the audio timestamp of the audio stream to the first processing unit via the seventh processing unit. The above-described audio-visual synchronization process can be found in [reference needed]. Figure 12 The specific embodiments shown are as follows.
[0279] Thus, even when the multimedia application supports self-developed video playback, the second processing unit, such as the intellectual property core, can synchronize audio timestamps with the multimedia application in this embodiment. This ensures synchronized audio and video playback and improves the stability and flexibility of the audio-visual synchronization process.
[0280] In some solutions, multiple embodiments of this application can be combined, and the combined solution can be implemented. Optionally, some operations in the process of each method embodiment may be combined, and / or the order of some operations may be changed. Furthermore, the execution order between the steps of each process is merely exemplary and does not constitute a limitation on the execution order between steps; other execution orders are also possible. It is not intended to indicate that the execution order is the only possible order in which these operations can be performed.
[0281] Those skilled in the art will conceive of various ways to reorder the operations described in the embodiments of this application. Furthermore, it should be noted that process details involved in one embodiment of this application are similarly applicable to other embodiments, or different embodiments can be combined.
[0282] Furthermore, some steps in the method embodiments can be equivalently replaced with other possible steps. Alternatively, some steps in the method embodiments may be optional and can be deleted in certain use cases. Or, other possible steps may be added to the method embodiments.
[0283] Furthermore, the various method embodiments can be implemented individually or in combination.
[0284] This application also provides an electronic device, such as the aforementioned mobile phone, etc. Figure 14 As shown, the mobile phone may include one or more processors 1210, memory 1220 and communication interface 1230.
[0285] The memory 1220, communication interface 1230, and processor 1210 are coupled together. For example, the memory 1220, communication interface 1230, and processor 1210 can be coupled together via bus 1240.
[0286] The communication interface 1230 is used for data transmission with other devices. The memory 1220 stores computer program code. The computer program code includes computer instructions, which, when executed by the processor 1210, cause the electronic device to perform the relevant method steps in the embodiments of this application.
[0287] Processor 1210 may be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with this disclosure. The processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.
[0288] Bus 1240 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The aforementioned bus 1240 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 14 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0289] This application also provides an electronic device, which includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the electronic device performs the relevant method steps in the above method embodiments.
[0290] This application also provides an electronic device equipped with a multimedia application, which includes a first processing unit and a second processing unit.
[0291] The first processing unit is configured to run multimedia applications.
[0292] The second processing unit is configured to acquire the multimedia stream of a multimedia application.
[0293] The second processing unit is also configured to process the multimedia stream and output the processing result; wherein the processing is decoding and rendering, or the processing is rendering.
[0294] In one possible implementation, the multimedia stream includes a video stream, and the first processing unit is configured to acquire first capability information of the second processing unit, the first capability information being used to characterize whether the second processing unit supports decoding and / or displaying the video stream.
[0295] The first processing unit is also configured to determine a first execution strategy for the second processing unit to process the video stream based on the first capability information.
[0296] In one possible implementation, if the first capability information indicates that the video stream is supported for decoding and display, the second processing unit is configured to decode and render the video stream.
[0297] When the first capability information indicates that it supports decoding the video stream but does not support displaying the video stream, the second processing unit is configured to decode the video stream, and the first processing unit is configured to control the second processing unit to render the video stream.
[0298] In one possible implementation, the first processing unit includes a rendering control module.
[0299] The rendering control module is configured to acquire the decoded video stream and send the decoded video stream to the second processing unit so that the second processing unit renders the video stream.
[0300] In one possible implementation, the multimedia stream includes an audio stream, and the first processing unit is configured to acquire second capability information of the second processing unit, the second capability information being used to characterize whether the second processing unit supports decoding the audio stream.
[0301] The first processing unit is also configured to determine a second execution strategy for the second processing unit to process the audio stream based on the second capability information.
[0302] Wherein, if the second capability information indicates support for decoding the audio stream, the second processing unit is configured to decode and render the audio stream.
[0303] In one possible implementation, the second processing unit includes a fifth processing unit and a sixth processing unit, wherein the fifth processing unit is used to decode the audio stream and the sixth processing unit is used to render the decoded audio stream.
[0304] The fifth processing unit is configured to send the decoded audio stream to the sixth processing unit.
[0305] In one possible implementation, the first processing unit is further configured to receive control operations on the multimedia stream, the control operations being used to control the playback of the multimedia stream, adjust the playback speed of the multimedia stream, adjust the playback progress, or adjust the playback volume of the multimedia stream.
[0306] In one possible implementation, the multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result.
[0307] The second processing unit is configured to synchronize the audio processing process and the video processing process using the audio timestamp of the audio stream;
[0308] The second processing unit is also configured to output the synchronized processing result.
[0309] In one possible implementation, the second processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit.
[0310] The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit via the seventh processing unit.
[0311] The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream.
[0312] The fourth processing unit is also configured to synchronize the audio processing results and the video processing results according to the expected playback time.
[0313] In one possible implementation, the multimedia stream includes a video stream, the second processing unit includes a fourth processing unit and a seventh processing unit, and the processing result includes a video processing result.
[0314] When the multimedia application supports audio playback, the seventh processing unit is configured to receive an audio timestamp from the first processing unit.
[0315] The seventh processing unit is also configured to send an audio timestamp to the fourth processing unit;
[0316] The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream.
[0317] The fourth processing unit is also configured to output video processing results based on the desired playback time.
[0318] In one possible implementation, the multimedia stream includes an audio stream, and the second processing unit includes a sixth processing unit and a seventh processing unit.
[0319] When the multimedia application supports video playback, the sixth processing unit is configured to synchronize the audio timestamp of the audio stream to the first processing unit through the seventh processing unit.
[0320] In one possible implementation, the sixth processing unit is configured to synchronize audio timestamps according to a preset time interval, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed, playback progress, or adjusting the playback volume of the multimedia stream, the sixth processing unit is configured to synchronize audio timestamps.
[0321] This application embodiment also provides a processing unit configured to acquire a multimedia stream of a multimedia application.
[0322] The processing unit is also configured to process the multimedia stream and output the processing results; wherein the processing is decoding and rendering, or the processing is rendering.
[0323] In one possible implementation, the multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. The processing unit is configured to synchronize the audio processing process and the video processing process using the audio timestamp of the audio stream.
[0324] The processing unit is also configured to output the synchronized processing result.
[0325] In one possible implementation, the processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit.
[0326] The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit via the seventh processing unit. The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream. The fourth processing unit is also configured to synchronize the audio processing result and the video processing result based on the expected playback time.
[0327] The beneficial effects of the above-mentioned electronic devices and processing units can be seen in the beneficial effects described in the specific embodiments above.
[0328] This application also provides a communication device, which includes a memory and one or more processors; the memory is coupled to the processors; wherein the memory stores computer program code, which includes computer instructions, and when the computer instructions are executed by the processor, the communication device performs the relevant method steps in the above method embodiments.
[0329] This application also provides a computer-readable storage medium storing computer program code. When the processor executes the computer program code, the electronic device executes the relevant method steps in the above method embodiments.
[0330] This application also provides a computer program product containing instructions that, when executed on a computer or processor, cause the computer or processor to perform the relevant method steps as described in the above method embodiments.
[0331] This application also provides a chip system, including: a processor coupled to a memory, the memory being used to store programs or instructions, and when the program or instructions are executed by the processor, the chip system enables the methods in any of the above method embodiments.
[0332] Optionally, the chip system may contain one or more processors. These processors can be implemented in hardware or software. When implemented in hardware, the processor can be a logic circuit, an integrated circuit, etc. When implemented in software, the processor can be a general-purpose processor, implemented by reading software code stored in memory.
[0333] Optionally, the chip system may contain one or more memories. The memory may be integrated with the processor or disposed separately from it; this application embodiment does not limit this. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or disposed separately on different chips. This application embodiment does not specifically limit the type of memory or the arrangement of the memory and processor.
[0334] For example, the chip system can be a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a system-on-chip (SoC), a central processing unit (CPU), a network processor (NP), a digital signal processor (DSP), a micro controller unit (MCU), a programmable logic device (PLD), or other integrated chips.
[0335] The electronic devices, computer storage media, or computer program products provided in this application are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.
[0336] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0337] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0338] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units, located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.
[0339] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0340] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the contributing parts, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0341] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
Claims
1. A data processing method applied to electronic devices, characterized in that, The electronic device includes a first processing unit and a second processing unit, and the method includes: The first processing unit runs the multimedia application; The second processing unit acquires the multimedia stream of the multimedia application; The second processing unit processes the multimedia stream and outputs the processing result; The processing can be either decoding and rendering, or rendering.
2. The method according to claim 1, characterized in that, The multimedia stream includes a video stream. Before the second processing unit acquires the multimedia stream of the multimedia application, the method further includes: The first processing unit obtains the first capability information of the second processing unit, which is used to characterize whether the second processing unit supports decoding and / or displaying the video stream; Based on the first capability information, the first processing unit determines a first execution strategy for the second processing unit to process the video stream.
3. The method according to claim 2, characterized in that, The method further includes: When the first capability information indicates support for decoding and displaying the video stream, the second processing unit decodes and renders the video stream; When the first capability information indicates that decoding of the video stream is supported but display of the video stream is not supported, the second processing unit decodes the video stream, and the first processing unit controls the second processing unit to render the video stream.
4. The method according to claim 3, characterized in that, The first processing unit includes a rendering control module, which controls the second processing unit to render the video stream, including: The rendering control module acquires the decoded video stream and sends the decoded video stream to the second processing unit so that the second processing unit renders the video stream.
5. The method according to claim 1, characterized in that, The multimedia stream includes an audio stream. Before the second processing unit acquires the multimedia stream of the multimedia application, the method further includes: The first processing unit obtains the second capability information of the second processing unit, which is used to characterize whether the second processing unit supports decoding the audio stream; Based on the second capability information, the first processing unit determines a second execution strategy for the second processing unit to process the audio stream; Wherein, if the second capability information indicates support for decoding the audio stream, the second processing unit decodes and renders the audio stream.
6. The method according to claim 5, characterized in that, The second processing unit includes a fifth processing unit and a sixth processing unit, wherein the fifth processing unit is used to decode the audio stream, and the sixth processing unit is used to render the decoded audio stream; the method further includes: The fifth processing unit sends the decoded audio stream to the sixth processing unit.
7. The method according to any one of claims 1-6, characterized in that, The method further includes: The first processing unit receives control operations on the multimedia stream, the control operations being used to control the playback of the multimedia stream, adjust the playback speed or playback progress of the multimedia stream, or adjust the playback volume of the multimedia stream.
8. The method according to claim 1, characterized in that, The multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. The second processing unit outputs the processing result, including: The second processing unit uses the audio timestamp of the audio stream to synchronize the audio processing process and the video processing process; The second processing unit outputs the synchronized processing result.
9. The method according to claim 8, characterized in that, The second processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit. The second processing unit uses the audio timestamp of the audio stream to synchronize the audio processing process and the video processing process, including: The sixth processing unit synchronizes the audio timestamp to the fourth processing unit through the seventh processing unit; The fourth processing unit uses the audio timestamp and the video timestamp of the video stream to calculate the expected playback time of the video stream; The fourth processing unit synchronizes the audio processing results and the video processing results according to the expected playback time.
10. The method according to claim 1, characterized in that, The multimedia stream includes a video stream, the second processing unit includes a fourth processing unit and a seventh processing unit, the processing result includes a video processing result, and the second processing unit outputs the processing result, including: If the multimedia application supports audio playback, the seventh processing unit receives an audio timestamp from the first processing unit; The seventh processing unit sends the audio timestamp to the fourth processing unit; The fourth processing unit uses the audio timestamp and the video timestamp of the video stream to calculate the expected playback time of the video stream; The fourth processing unit outputs the video processing result based on the desired playback time.
11. The method according to claim 1, characterized in that, The multimedia stream includes an audio stream, the second processing unit includes a sixth processing unit and a seventh processing unit, and the method further includes: When the multimedia application supports video playback, the sixth processing unit synchronizes the audio timestamp of the audio stream to the first processing unit through the seventh processing unit.
12. The method according to claim 9 or 11, characterized in that, The sixth processing unit synchronizes the audio timestamp according to a preset time interval, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed or playback progress of the multimedia stream, or adjusting the playback volume of the multimedia stream, the sixth processing unit synchronizes the audio timestamp.
13. An electronic device, characterized in that, The electronic device is equipped with a multimedia application and includes a first processing unit and a second processing unit. The first processing unit is configured to run the multimedia application; The second processing unit is configured to acquire the multimedia stream of the multimedia application; The second processing unit is further configured to process the multimedia stream and output the processing result; wherein the processing is decoding and rendering, or the processing is rendering.
14. The electronic device according to claim 13, characterized in that, The multimedia stream includes a video stream. The first processing unit is configured to acquire first capability information of the second processing unit. The first capability information is used to characterize whether the second processing unit supports decoding and / or displaying the video stream. The first processing unit is further configured to determine a first execution strategy for the second processing unit to process the video stream based on the first capability information.
15. The electronic device according to claim 14, characterized in that, When the first capability information indicates support for decoding and displaying the video stream, the second processing unit is configured to decode and render the video stream. When the first capability information indicates that decoding of the video stream is supported but display of the video stream is not supported, the second processing unit is configured to decode the video stream, and the first processing unit is configured to control the second processing unit to render the video stream.
16. The electronic device according to claim 15, characterized in that, The first processing unit includes a rendering control module; The rendering control module is configured to acquire the decoded video stream and send the decoded video stream to the second processing unit so that the second processing unit renders the video stream.
17. The electronic device according to claim 13, characterized in that, The multimedia stream includes an audio stream. The first processing unit is configured to acquire second capability information of the second processing unit, wherein the second capability information is used to characterize whether the second processing unit supports decoding of audio streams; The first processing unit is further configured to determine a second execution strategy for the second processing unit to process the audio stream based on the second capability information; Wherein, if the second capability information indicates support for decoding the audio stream, the second processing unit is configured to decode and render the audio stream.
18. The electronic device according to claim 17, characterized in that, The second processing unit includes a fifth processing unit and a sixth processing unit. The fifth processing unit is used to decode the audio stream, and the sixth processing unit is used to render the decoded audio stream. The fifth processing unit is configured to send the decoded audio stream to the sixth processing unit.
19. The electronic device according to any one of claims 13-18, characterized in that, The first processing unit is further configured to receive control operations on the multimedia stream, the control operations being used to control the playback of the multimedia stream, adjust the playback speed or playback progress of the multimedia stream, or adjust the playback volume of the multimedia stream.
20. The electronic device according to claim 13, characterized in that, The multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. The second processing unit is configured to synchronize the audio processing process and the video processing process using the audio timestamp of the audio stream; The second processing unit is also configured to output the synchronized processing result.
21. The electronic device according to claim 20, characterized in that, The second processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit; The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit through the seventh processing unit; The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream. The fourth processing unit is further configured to synchronize the audio processing result and the video processing result according to the expected playback time.
22. The electronic device according to claim 13, characterized in that, The multimedia stream includes a video stream, the second processing unit includes a fourth processing unit and a seventh processing unit, and the processing result includes a video processing result. When the multimedia application supports audio playback, the seventh processing unit is configured to receive an audio timestamp from the first processing unit. The seventh processing unit is further configured to send the audio timestamp to the fourth processing unit; The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream. The fourth processing unit is further configured to output the video processing result according to the expected playback time.
23. The electronic device according to claim 13, characterized in that, The multimedia stream includes an audio stream, and the second processing unit includes a sixth processing unit and a seventh processing unit; When the multimedia application supports video playback, the sixth processing unit is configured to synchronize the audio timestamp of the audio stream to the first processing unit through the seventh processing unit.
24. The electronic device according to claim 21 or 23, characterized in that, The sixth processing unit is configured to synchronize the audio timestamps at preset time intervals, and / or, when controlling the playback of the multimedia stream, adjusting the playback speed or playback progress of the multimedia stream, or adjusting the playback volume of the multimedia stream, the sixth processing unit is configured to synchronize the audio timestamps.
25. A processing unit, characterized in that, The processing unit is configured to acquire the multimedia stream of a multimedia application; The processing unit is further configured to process the multimedia stream and output the processing result; wherein the processing is decoding and rendering, or the processing is rendering.
26. The processing unit according to claim 25, characterized in that, The multimedia stream includes an audio stream and a video stream, and the processing result includes an audio processing result and a video processing result. The processing unit is configured to synchronize the audio processing process and the video processing process using the audio timestamp of the audio stream; The processing unit is also configured to output the synchronized processing result.
27. The processing unit according to claim 26, characterized in that, The processing unit includes a fourth processing unit, a sixth processing unit, and a seventh processing unit; The sixth processing unit is configured to synchronize the audio timestamp to the fourth processing unit through the seventh processing unit; The fourth processing unit is configured to calculate the expected playback time of the video stream using the audio timestamp and the video timestamp of the video stream. The fourth processing unit is further configured to synchronize the audio processing result and the video processing result according to the expected playback time.
28. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is coupled to the processor and is used to store a computer program that, when executed by the processor, causes the electronic device to perform the method as described in any one of claims 1-12.
29. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the electronic device to perform the method as described in any one of claims 1-12.
30. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it causes the electronic device to perform the method as described in any one of claims 1-12.
