A media data processing method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a method, device, electronic device and storage medium for processing media data, which are applied to a first processing module in an electronic device, and the electronic device also includes a second processing module, a first operating system is running in the first processing module, and a second operating system is running in the second processing module, and the method includes: obtaining the media data to be processed and the hardware resource information of the second processing module; when the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, so that the second processing module uses a decoder established by the second operating system to decode and process the media data. Through the scheme of the embodiments of the present application, it is possible to achieve reasonable allocation of tasks, improve data processing efficiency, and improve data processing efficiency in multi-system devices.

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a media data processing method, device, electronic device and storage medium. Background Art

[0002] At present, electronic devices generally need to run an operating system, such as an Android system or a Linux system, etc. The operating system can be used to obtain and process data to be processed. For example, when processing media data, the operating system can be used to decode the media data.

[0003] However, most of the current electronic devices are single operating system devices. When decoding media data through such electronic devices, the processing module not only needs to decode the media data, but also needs to run its own other system software, resulting in fewer resources that can be allocated to media data decoding, resulting in low decoding efficiency. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a media data processing method, device, electronic device and storage medium to improve the decoding efficiency of media data. The specific technical solution is as follows:

[0005] In a first aspect of an embodiment of the present application, a media data processing method is provided, which is applied to a first processing module in an electronic device, wherein the electronic device further includes a second processing module, wherein a first operating system runs in the first processing module, and a second operating system runs in the second processing module, wherein the method includes:

[0006] Acquire the media data to be processed and the hardware resource information of the second processing module;

[0007] When the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

[0008] Optionally, the method further includes:

[0009] When the hardware resources of the second processing module do not meet the first preset resource condition, the media data is decoded using a decoder established by the first operating system.

[0010] Optionally, when the hardware resources of the second processing module do not meet the first preset resource condition, decoding the media data by using a decoder established by the first operating system includes:

[0011] In the case where the hardware resources of the second processing module do not meet the first preset resource condition, determining whether the GPU resources of the first processing module meet the second preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data, and the media data includes image data;

[0012] If the second preset resource condition is met, processing the image data using a decoder established by the first operating system based on GPU resources;

[0013] If the second preset resource condition is not met, the image data is processed using a decoder established by the first operating system based on CPU resources.

[0014] Optionally, the media data includes image data; and the method further includes:

[0015] The rendering position information of the image data is acquired, and the rendering position information is sent to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information.

[0016] Optionally, the first processing module and the second processing module are connected via a network port link and a media interface link;

[0017] The obtaining the rendering position information of the image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information, comprises:

[0018] When it is detected that a display window of the image data is created or changed, acquiring position information of the display window to obtain rendering position information;

[0019] Switching the input source of the display window to the media interface link;

[0020] The rendering position information is sent to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data in the display window through the media interface link according to the rendering position information.

[0021] Optionally, the media data includes image data, and when the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module includes:

[0022] In the current processing process, if the image data needs to be rendered and displayed, and the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module;

[0023] And / or, when the image data does not need to be rendered and displayed during this processing, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data.

[0024] Optionally, in the current processing process, if the image data does not need to be rendered and displayed, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, after the step of sending the media data to the second processing module, the method further includes:

[0025] Acquire the encoded image data sent by the second processing module, wherein the encoded image data is obtained by: the second processing module decoding the image data using a decoder established by the second operating system, and encoding the decoded image data using a preset encoding algorithm, the amount of hardware resources required by the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; and decode the encoded image data using a decoder established by the first operating system based on CPU resources;

[0026] Or, obtain the decoded image data sent by the second processing module, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system.

[0027] Optionally, the method further includes:

[0028] In this processing, if the image data does not need to be rendered and displayed, and the GPU resources of the first processing module meet the third preset resource condition, the image data is processed using a decoder established based on the GPU resources by the first operating system.

[0029] According to a second aspect of an embodiment of the present application, a media data processing method is provided, which is applied to a second processing module in an electronic device, wherein the electronic device further includes a first processing module, a first operating system is running in the first processing module, and a second operating system is running in the second processing module. The method includes:

[0030] receiving media data sent by the first processing module, wherein the media data is sent by the first processing module to the second processing module when the hardware resources of the second processing module meet a first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data;

[0031] The media data is decoded using a decoder established by the second operating system.

[0032] Optionally, the media data includes image data; the first processing module is connected to the second processing module via a network port link and a media interface link; and the method further includes:

[0033] In the case where the image data needs to be rendered and displayed during this processing, obtaining rendering position information sent by the first processing module through the network port link;

[0034] In response to the rendering position information, the decoded image data output by the decoder is rendered and displayed in the display window through the media interface link.

[0035] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link; and the method further includes:

[0036] In the case where the image data does not need to be rendered and displayed during this processing, and the bit rate of the decoded image data is not higher than the bandwidth of the network port link, the decoded image data is sent to the first processing module;

[0037] Or, when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, the decoded image data is encoded using a preset encoding algorithm to obtain encoded image data, and the encoded image data is sent to the first processing module, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data.

[0038] Optionally, the method further includes:

[0039] After receiving the media data sent by the first processing module, establishing a decoder using the second operating system;

[0040] After the decoding process of the media data is completed, the corresponding decoder is deleted and the corresponding hardware resources in the second processing module are released.

[0041] According to a third aspect of an embodiment of the present application, a media data processing device is provided, which is applied to a first processing module in an electronic device, wherein the electronic device further includes a second processing module, wherein a first operating system runs in the first processing module, and a second operating system runs in the second processing module, wherein the device includes:

[0042] An information acquisition module, used to acquire the media data to be processed and the hardware resource information of the second processing module;

[0043] A data sending module is used to send the media data to the second processing module when the hardware resource information of the second processing module meets the first preset resource condition, so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

[0044] Optionally, the device further comprises:

[0045] a data decoding module, configured to decode the media data using a decoder established by the first operating system when the hardware resources of the second processing module do not meet the first preset resource condition;

[0046] Optionally, the data decoding module includes:

[0047] a resource determination submodule, configured to determine whether the GPU resources of the first processing module meet a second preset resource condition when the hardware resources of the second processing module do not meet the first preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data, wherein the media data includes image data;

[0048] A first processing submodule, configured to process the image data by using a decoder established by the first operating system based on GPU resources if the second preset resource condition is met;

[0049] A second processing submodule, configured to process the image data using a decoder established by the first operating system based on CPU resources if the second preset resource condition is not met;

[0050] Optionally, the media data includes image data; and the device further includes:

[0051] An image acquisition module, used for acquiring rendering position information of the image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information;

[0052] Optionally, the first processing module and the second processing module are connected via a network port link and a media interface link;

[0053] The image acquisition module comprises:

[0054] A position detection submodule, for acquiring position information of the display window and obtaining rendering position information when detecting that the display window of the image data is created or changed;

[0055] An input switching submodule, used for switching the input source of the display window to the media interface link;

[0056] A window rendering submodule, configured to send the rendering position information to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data into the display window through the media interface link according to the rendering position information;

[0057] Optionally, the media data includes image data, and the data sending module includes:

[0058] A first sending submodule, configured to send the media data to the second processing module if the image data needs to be rendered and displayed during this processing and the hardware resource information of the second processing module meets a first preset resource condition;

[0059] The second sending submodule is used to send the media data to the second processing module when the image data does not need to be rendered and displayed during this processing, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data.

[0060] Optionally, the device further comprises:

[0061] a coded data acquisition module, used for acquiring coded image data sent by the second processing module, wherein the coded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system and encoding the decoded image data using a preset encoding algorithm, wherein the amount of hardware resources required by the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; an image decoding submodule, used for decoding the coded image data using a decoder established by the first operating system based on CPU resources;

[0062] Or, a decoded data acquisition module is used to acquire decoded image data sent by the second processing module, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system.

[0063] Optionally, the device further comprises:

[0064] An image processing module is used to process the image data using a decoder established based on the GPU resources through the first operating system when the image data does not need to be rendered and displayed during this processing and the GPU resources of the first processing module meet the third preset resource conditions.

[0065] According to a fourth aspect of an embodiment of the present application, a media data processing device is provided, which is applied to a second processing module in an electronic device, wherein the electronic device further includes a first processing module, a first operating system is running in the first processing module, and a second operating system is running in the second processing module, wherein the device includes:

[0066] a media data receiving module, configured to receive media data sent by the first processing module, wherein the media data is sent by the first processing module to the second processing module when the hardware resources of the second processing module meet a first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data;

[0067] The media data decoding module is used to decode the media data using a decoder established by the second operating system.

[0068] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link and a media interface link; the device further includes:

[0069] A rendering position acquisition module, used for acquiring rendering position information sent by the first processing module through the network port link when the image data needs to be rendered and displayed during the current processing;

[0070] A rendering position display module, configured to render and display the decoded image data output by the decoder into the display window via the media interface link in response to the rendering position information;

[0071] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link; and the device includes:

[0072] a coded image acquisition module, used for, when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, encoding the decoded image data using a preset encoding algorithm to obtain the coded image data, and sending the coded image data to the first processing module through the network port link, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data;

[0073] Or, a decoded image acquisition module is used to send the decoded image data to the first processing module through the network port link when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is not higher than the bandwidth of the network port link.

[0074] Optionally, the device further comprises:

[0075] A decoder establishment module, configured to establish a decoder using the second operating system after receiving the media data sent by the first processing module;

[0076] The resource release module is used to delete the corresponding decoder and release the corresponding hardware resources in the second processing module after completing the decoding process of the media data.

[0077] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, any of the above-mentioned media data processing methods applied to the first processing module is implemented.

[0078] In a sixth aspect of an embodiment of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned media data processing methods applied to the second processing module is implemented.

[0079] According to a seventh aspect of the embodiments of the present application, a chip is provided, including a processing unit and a storage unit, wherein the storage unit is used to store a computer program;

[0080] The processing unit is used to implement any of the above-mentioned media data processing methods applied to the first processing module when executing the program stored in the storage unit.

[0081] According to an eighth aspect of the embodiments of the present application, a chip is provided, including a processing unit and a storage unit, wherein the storage unit is used to store a computer program;

[0082] The processing unit is used to implement any of the above-mentioned media data processing methods applied to the second processing module when executing the program stored in the storage unit.

[0083] According to a ninth aspect of the embodiments of the present application, an electronic device is provided, including:

[0084] A first processing module and a second processing module, wherein a first operating system runs in the first processing module and a second operating system runs in the second processing module;

[0085] The first processing module is used to implement any of the above media data processing methods applied to the first processing module during operation;

[0086] The second processing module is used to implement any of the above-mentioned media data processing methods applied to the second processing module during runtime.

[0087] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute and implement any of the above-mentioned media data processing methods applied to the first processing module.

[0088] The embodiment of the present application also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute and implement any of the above-mentioned media data processing methods applied to the second processing module.

[0089] Beneficial effects of the embodiments of the present application:

[0090] The embodiment of the present application provides a media data processing method, device, electronic device and storage medium, which are applied to a first processing module in an electronic device, wherein the electronic device also includes a second processing module, wherein a first operating system is running in the first processing module, and a second operating system is running in the second processing module, wherein the method includes: obtaining the media data to be processed and the hardware resource information of the second processing module; when the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data. Through the scheme of the embodiment of the present application, when the remaining hardware resources of the second processing module are sufficient to process the media data, the media data can be sent to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data. Since the software running in the system runs through the first processing module, and the media data is decoded through the second processing module, the decoding efficiency of the media data can be improved.

[0091] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0093] Figure 1 A schematic diagram of a flow chart of a media data processing method provided in an embodiment of the present application;

[0094] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0095] Figure 3 Another schematic diagram of a flow chart of a media data processing method provided in an embodiment of the present application;

[0096] Figure 4 A schematic diagram of another process of the media data processing method provided in the embodiment of the present application;

[0097] Figure 5 A schematic diagram of a process for displaying decoded image data provided by an embodiment of the present application;

[0098] Figure 6 A schematic diagram of an example of a media data processing flow provided in an embodiment of the present application;

[0099] Figure 7 A schematic diagram of a process for link rendering and display through a media interface provided in an embodiment of the present application;

[0100] Figure 8 An example diagram of a media data processing method provided in an embodiment of the present application;

[0101] Fig. 9 A schematic diagram of an example of an electronic device provided in an embodiment of the present application;

[0102] Fig.10 A flowchart of another media data processing method provided in an embodiment of the present application;

[0103] Fig.11 A schematic diagram of the structure of a media data processing device provided in an embodiment of the present application;

[0104] Fig.12 Another structural schematic diagram of a media data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0105] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0106] First, the professional terms that may be used in the embodiments of the present application are explained:

[0107] CPU: The central processing unit (CPU) is the computing and control core of the computer system and the final execution unit for information processing and program running.

[0108] GPU: Graphics processing unit (GPU), also known as display core, visual processor, display chip, is a microprocessor that specializes in performing image and graphics related calculations on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.).

[0109] MediaCodec: A class provided by Android for encoding and decoding audio and video. It implements the encoding and decoding functions by accessing the underlying codec. It is part of the Android media infrastructure framework and is usually used together with MediaExtractor, MediaSync, MediaMuxer, MediaCrypto, MediaDrm, Image, Surface, and AudioTrack.

[0110] MediaPlayer: can be used to play audio and video files, or audio streams. Developers can use it to play local audio or online audio. MediaPlayer belongs to the android.media package.

[0111] FFmpeg: A set of open source computer programs that can be used to record, convert digital audio and video, and convert them into streams. It uses LGPL or GPL licenses. It provides a complete solution for recording, converting and streaming audio and video. It contains a very advanced audio / video codec library libavcodec. In order to ensure high portability and codec quality, many codes in libavcodec are developed from scratch.

[0112] In a first aspect of an embodiment of the present application, a media data processing method is first provided, which is applied to a first processing module in an electronic device, wherein the electronic device further includes a second processing module, wherein a first operating system is running in the first processing module, and a second operating system is running in the second processing module, see Figure 1 , the above method comprises:

[0113] Step S11, obtaining the media data to be processed and the hardware resource information of the second processing module;

[0114] Step S12, when the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

[0115] The electronic device in the embodiment of the present application may be a tablet computer, a mobile phone, etc. For details, see Figure 2The electronic device may be installed with a first processing module and a second processing module, the hardware resources of the first processing module are different from the hardware resources of the second processing module. In one example, the first processing module and the second processing module may be different processing cores on the same chip. In another example, the first processing module and the second processing module may be different chips. A first operating system runs in the first processing module and a second operating system runs in the second processing module. The first operating system and the second operating system may be different operating systems. For example, the first operating system and the second operating system may be an Android system and a Linux system, respectively, or they may be the same operating system, all of which are within the protection scope of the present application.

[0116] The media data in the embodiment of the present application includes video data, and may also include various types of data such as audio data. Specifically, the first operating system running in the first processing module may receive the media data through the network or read the file. For example, the media data may be encoded data, such as H264 / H265, etc., which has the characteristics of high compression rate and cannot be directly rendered and displayed. There is an interaction of hardware resource information between the first processing module and the second processing module. In one example, the second processing module may actively send its own hardware resource information to the first processing module. Specifically, the second processing module may actively send its own hardware resource information to the first processing module when the usage of its own hardware resources changes; the second processing module may also periodically send its own hardware resource information to the first processing module. In addition, the second processing module may also respond to the query message of the first processing module and send its own hardware resource information to the first processing module. Specifically, the first processing module may send a query message to the second processing module after obtaining the media data to be processed, and the second processing module responds to the query message and sends its own hardware resource information to the first processing module. The hardware resource information of the second processing module is used to determine whether the remaining hardware resources of the second processing module are sufficient to process the media data. In an example, the hardware resource information of the second processing module can be the hardware resource utilization rate of the second processing module, or directly the judgment result of whether the remaining hardware resources sent by the second processing module are sufficient to process the media data.

[0117] Since the first processing module is the main control module of the electronic device, a variety of software programs are running in the first processing module, resulting in fewer resources that can be allocated to media data decoding. If the media data is decoded through the first processing module, the processing pressure of the first processing module will increase, resulting in low decoding efficiency of the media data, and even causing the electronic device to run stuck. Through the method of the embodiment of the present application, the second processing module is a processing module different from the first processing module. When the remaining hardware resources of the second processing module are sufficient to process the media data, the media data can be sent to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data. Decoding the media data through the second processing module will not cause the first processing module to freeze, and can provide more hardware resources for the media data, thereby improving the decoding efficiency of the media data.

[0118] When the hardware resources of the second processing module are sufficient, the second processing module can be used to process the media data. When the hardware resources of the second processing module are insufficient, the first processing module can be used to decode the media data. Figure 3 , the above method further comprises:

[0119] Step S31: When the hardware resources of the second processing module do not meet the first preset resource condition, the media data is decoded using a decoder established by the first operating system.

[0120] The hardware resources of the second processing module do not meet the first preset resource condition, indicating that the hardware resources of the second processing module are insufficient to process the media data. In this case, the processing of the media data can be completed by the hardware resources of the first processing module itself. In one example, the first processing module can use a decoder established by the first operating system to decode the media data. Specifically, it can be determined whether the hardware resources of the second processing module meet the first preset resource condition. If the hardware resources of the second processing module do not meet the first preset resource condition, the media data can be decoded using a decoder established by the first operating system.

[0121] In some scenarios, the first processing module includes multiple types of hardware resources, such as GPU resources and CPU resources. Figure 4 , when the hardware resources of the second processing module do not meet the first preset resource condition, decoding the media data using a decoder established by the first operating system, including:

[0122] Step S311, when the hardware resources of the second processing module do not meet the first preset resource condition, determine whether the GPU resources of the first processing module meet the second preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data;

[0123] Step S312, if the second preset resource condition is met, the image data is processed using a decoder established by the first operating system based on the GPU resources;

[0124] Step S313: If the second preset resource condition is not met, the image data is processed using a decoder established by the first operating system based on CPU resources.

[0125] Among them, in actual use, decoding of media data by GPU is also called hard decoding, and decoding of media data by CPU is also called soft decoding. Since the number of cores of GPU is much higher than the number of cores of CPU, the parallel computing capability of GPU is higher than that of CPU, and the decoding process for media data is a parallel computing process (multiple coding blocks in the same video frame are decoded in parallel), so the efficiency of hard decoding by GPU for decoding media data is higher than the efficiency of soft decoding by CPU. Since GPU is better at processing the decoding operation of media data than CPU, it can be judged whether the GPU resources of the second processing module meet the second preset resource condition first. If the GPU resources of the second processing module meet the second preset resource condition, the image data is processed by the decoder established by the first operating system based on the GPU resources; if the GPU resources of the second processing module do not meet the second preset resource condition, the image data is processed by the decoder established by the first operating system based on the CPU resources. Through this strategy, the resources such as GPU and CPU of the first processing module can be combined with the resources of the second processing module to increase the decoding performance.

[0126] The decoders in the first processing module and the second processing module can be pre-created or established in real time after receiving the media data each time. In the case of establishing the decoder in real time, when the decoding task of the media data is completed, the decoder can also be destroyed to release the hardware resources occupied by it. In the actual use process, the first operating system and the second operating system are respectively Android system and Linux system as examples for explanation. The decoding of image data can be achieved by rewriting the standard encoding and decoding method of the Android system. Since Android implements the system encoding and decoding functions through Mediacodec and MediaPlay, taking Mediacodec as an example, the decoding function can be achieved by rewriting the MediaCodec code located in the C++Framework layer, wherein the MediaCodec in the JAVA layer also implements decoding by calling the MediaCodec in the C++ layer. The rewriting method is as follows: 1. Create resources: When creating a decoder, such as using a series of methods such as AMediaCodec_createDecoderByType, the GPU hard decoding of the first processing module, the CPU soft decoding and the decoding resources of the first processing module are all combined in the creation process of the decoder. According to the above strategy, it is selected whether to collect resources such as Android system GPU, CPU, etc., or to use the decoding resources of the Linux system to maximize the increase in encoding and decoding performance. 2. Decoding data: After the decoder is created, when the user calls AMediaCodec_queueInputBuffer (queue input buffer of the media data decoder), etc., the first processing module confirms the type of decoder created. If it is a decoder of the second operating system, the media data is sent to the second processing module through the network port link; if it is a decoder of the GPU of the first operating system, it is decoded through the GPU of the Android system module; if it is a decoder of the CPU of the first operating system, the media data is sent to a dedicated decoder such as FFMPEG based on CPU resources; when calling AMediaCodec_getOutputBuffer, etc. to obtain the decoded data, the processing method is similar. For example, if it is a decoder of the second operating system, try to obtain the decoded data from the second processing module. 3. Destroy the decoder. When the media data decoding is completed, the decoder can be deleted using AMediaCodec_delete, etc. For example, if it is a decoder of the second operating system, the decoder is deleted in the second processing module. If it is a decoder of the GPU of the first operating system, the decoder based on GPU resources is deleted in the first processing module; if it is a decoder of the CPU of the first operating system, the decoder based on CPU resources is deleted in the first processing module.

[0127] For the case of local decoding in the first processing module, the decoded image data can be directly used for rendering and display. In one example, the surface information can be sent by the user, and the decoded data can be directly rendered at the specified position through the surface. Among them, in the embodiments of the present application, the control used is often not a surface, but a view control that encapsulates a surface, such as surfaceview, textview, etc. When creating and using these views, the system will call the surface method, and the view also provides a method for obtaining surface information.

[0128] In actual use, if the image data in the media data needs to be rendered during this processing, after the second processing module completes the decoding of the image data, the second processing module can directly render and display it. In an example, the media data includes image data; if the image data in the media data needs to be rendered during this processing, refer to Figure 5 , the above method further comprises:

[0129] Step S51, obtaining rendering position information of the image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information.

[0130] Among them, the first processing module has control over the display screen and can control the rendering position of the image data on the display screen. Therefore, the first processing module needs to send the rendering position information of the image data to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information. Among them, the rendering position information can refer to the coordinate position information on the screen. For example, in actual use, the electronic device in the embodiment of the present application can include a screen. When the first processing module obtains the rendering position information, the coordinate position of a certain position or a certain area on the screen can be obtained, and the coordinate position is sent to the second processing module as the rendering position information.

[0131] In one example, the first processing module can write the rendering position information into a preset field of the display window information (surface information), and send the display window information to the decoder in the second processing module through the network port link. The rendering position information can be added to the surface information structure, and a preset field is added to the surface information for writing the rendering position information, such as the coordinates of the rendering position. The surface information is sent to the second processing module through the first processing module to realize the sending of the rendering position information. In one example, a variable space for saving the position coordinates can be added to the surface information to store the rendering position information. In addition to the rendering position information, the surface information also includes other display window control information. For details, please refer to the relevant surface information structure, which will not be repeated here.

[0132] In actual use, the amount of decoded image data is very large. If the first processing module retrieves the data and displays it on the upper screen, it will cause a shortage of communication resources between the first processing module and the second processing module. In an embodiment of the present application, the second processing module directly renders the decoded image data on the screen, which can save communication resources between the first processing module and the second processing module.

[0133] The first processing module and the second processing module may include multiple communication links. In one example, the first processing module and the second processing module are connected via a network port link and a media interface link. Figure 6In the figure, FAPP and GAPP represent different APPs (Applications). In the MediaCodec (media code) of the Java Framework (Java framework) System server (system service), and then through the Zygote Android Runtime in JNI (Java NativeInterface), under C++AppNDK, FAPP and GAPP use different media codes JMediaCodec and AmediaCodec. Finally, in C++Framework (C++ framework), it is determined whether the A system resources are sufficient. If so, they are processed through 3559API and rendered using the A system DSP Codec. If not, it is determined whether the GPU resources are sufficient. If so, they are processed through the Media Server (media service) and rendered through the USE GPU Codec. If the GPU resources are not sufficient, it is determined whether the CPU and memory resources are sufficient. If so, they are processed through FFMPEGetc. and rendered through the USE CPUCodec. The first operating system and the second operating system in the embodiment of the present application can be an Android system and a Linux system respectively. There are two main connections used. One is a bidirectional path through the network, that is, a network port link, which is mainly responsible for controlling the transmission of signaling and media data; the other path is a unidirectional path of HDMI, which is responsible for rendering the decoded data on the screen of the Android system. The Android system obtains the data to be decoded through 3559API (a chip), and sends the data to be decoded to AppPro in the Linux system through the bidirectional channel Network, and then decodes it through DspCore in the Linux system to obtain the decoded data, and finally renders the decoded data to the screen of the Android system through the unidirectional path HDMI.

[0134] Optionally, in the above step S51, the rendering position information of the image data is obtained, and the rendering position information is sent to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information, see Figure 7 ,include:

[0135] Step S511, when it is detected that a display window for image data is created or changed, the position information of the display window is acquired to obtain rendering position information.

[0136] For example, in the surface information structure of the first processing module, a preset field is added to save the acquired rendering position information, which can be a position coordinate variable field; when creating and changing the surface, the rendering position coordinates are obtained and added to the preset field of the surface information. When the user creates a view and changes the position of the view, the system will call the surfaceCreated creation method and the surfaceChanged position change method to obtain the position information from the surfaceFlinger, and save the acquired position coordinates to the newly added position coordinate variable. When the first processing module sends the specified position information to the second processing module, the position coordinate information can be transmitted to the second processing module. After the user sends the surface information to the decoder, the position coordinate information saved by the surface will be paid attention to at all times during the decoding process. When the coordinate information changes, the new coordinate information will be transmitted to the second operating system through the network to update the rendering display position. Specifically, when the first processing module sends the specified position information to the second processing module, a new variable space is added to save the position coordinates. Then when the user creates a view and changes the position of the view, the system will call the surfaceCreated creation method and the surfaceChanged position change method to obtain the position information from surfaceFlinger and transmit it to the second operating system.

[0137] Step S512: switch the input source of the display window to the media interface link.

[0138] Step S513: Send the rendering position information to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data in the display window through the media interface link according to the rendering position information.

[0139] In the embodiment of the present application, the execution order of step S512 and step S513 is not limited. Step S512 may be executed first and then step S513, or step S513 may be executed and then step S513 may be executed, or step S512 and step S513 may be executed in parallel. In one example, the switching of the input source of the display window may be completed first, and then the rendering position information may be sent, so that the second processing module may directly use the media interface link to render and display the video data after receiving the rendering position information. If the rendering position information is sent first, and then the input source of the display window is switched to the media interface link, the first processing module also needs to send additional signaling after the switching is completed to notify the second processing module that the input source of the display window has been switched, so that the rendering display can begin.

[0140] When the first processing module detects that a display window for image data is created or changed, it obtains the position information of the display window and obtains the rendering position information; switches the input source of the display window to the media interface link; sends the rendering position information to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data in the display window through the media interface link according to the rendering position information. The media interface link here can be a media interface such as HDMI (High Definition Multimedia Interface, high-definition multimedia interface), which is not specifically limited in this application. After the first processing module switches the input source of the display window to the media interface link, the second processing module can directly realize the rendering of the video data on the screen through the media interface link according to the rendering position information. For example, the second processing module displays the image at the surface coordinate position through HDMI, and Dspcore decodes the data and displays it to the specified position of the Android screen through HDMI. In this process, the first processing module does not need to perform additional operations, and only needs to notify the second processing module of the changed rendering position information when changing the display window of the image data.

[0141] The media interface link is used for the second processing module to display the decoded video data on the screen, and the network port link is used for data and signaling interaction between the first processing module and the second processing module. After the second processing module completes the decoding of the image data, if the first processing module is used to render and display it, the first processing module needs to retrieve the decoded image data from the second processing module through the network port link. However, the amount of decoded image data is very large. Retrieving the decoded image data by the first processing module through the network port link will occupy a large amount of network port link bandwidth between the first processing module and the second processing module. Not only is the retrieval time of the decoded image data long, but it will also affect the normal communication process between the first processing module and the second processing module. In the embodiment of the present application, the first processing module switches the input source of the display window to the media interface link. When the second processing module obtains the decoded image data, the rendering and display of the decoded image data can be directly realized through the media interface link, which greatly reduces the communication pressure of the network port link between the first processing module and the second processing module, can save the communication resources of the first processing module and the second processing module, speed up the display of image data, and reduce data processing delay.

[0142] In addition, in the embodiment of the present application, the first processing module and the second processing module each run an independent operating system, and the first processing module and the second processing module communicate through a network port link, that is, the first processing module and the second processing module are two relatively independent processing systems, and the two implement decoding of media data based on their own decoders, and the two implement data interaction between the two operating systems through a network port link. Compared with the first processing module calling other hardware resources through an API interface (Application Programming Interface), there is no need to develop an API interface separately, thereby reducing the workload of secondary development.

[0143] When the media data includes image data, optionally, when the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module includes:

[0144] In this processing process, when the image data needs to be rendered and displayed, and the hardware resource information of the second processing module meets the first preset resource condition, the first processing module sends the media data to the second processing module.

[0145] And / or, when the image data does not need to be rendered and displayed during this processing, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data.

[0146] Among them, whether the image data needs to be rendered and displayed during this processing refers to whether the processing task for the media data received this time includes the rendering and display task of the image data; it does not include consideration of whether the decoded image data is rendered and displayed in the subsequent process. This is because the first processing module cannot predict the subsequent processing process and can only make judgments based on the currently issued tasks.

[0147] In one example, in the current processing process, if the image data does not need to be rendered and displayed, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, after the step of sending the media data to the second processing module, the method further includes:

[0148] Obtain the encoded image data sent by the second processing module, wherein the encoded image data is obtained by: the second processing module decoding the image data using a decoder established by the second operating system, and encoding the decoded image data using a preset encoding algorithm, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; and decode the encoded image data using a decoder established based on CPU resources by the first operating system.

[0149] Or, the decoded image data sent by the second processing module is obtained, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system.

[0150] Optionally, the above method also includes: when the image data does not need to be rendered and displayed during the current processing and the GPU resources of the first processing module meet the second preset resource conditions, the image data is processed using a decoder established based on the GPU resources by the first operating system.

[0151] The first processing module and the second processing module are connected via a network port link and a media interface link, wherein the media interface link is used for the second processing module to display the decoded video data on the screen, and the network port link is used for data and signaling interaction between the first processing module and the second processing module. After the second processing module completes the decoding of the image data, if the image data is not displayed on the screen, but the first processing module needs to retrieve it for other operations, then it is necessary to complete the retrieval of the decoded image data through the network port link. However, the amount of decoded image data is very large, and the retrieval of the decoded image data by the first processing module through the network port link will occupy a large amount of network port link bandwidth between the first processing module and the second processing module, which not only takes a long time to retrieve the decoded image data, but also affects the normal communication process between the first processing module and the second processing module. Therefore, when the image data does not need to be decoded, in order to reduce the communication pressure of the network port link between the first processing module and the second processing module, it is prioritized to determine whether the GPU resources in the first processing module are sufficient to decode the media data. If sufficient, the first processing module directly uses its own GPU resources to process the media data, and there is no need to send the media data to the second processing module through the network port link. After that, the decoded data is retrieved through the network port link, which can save the communication resources between the first processing module and the second processing module and reduce the data processing delay.

[0152] Among them, in actual use, decoding of media data through GPU is also called hard decoding, and decoding of media data through CPU is also called soft decoding. Since the parallel computing capability of GPU is much higher than that of CPU, and the decoding of media data requires multi-channel parallel computing, the efficiency of hard decoding of media data through GPU is often higher. Therefore, in order to improve the efficiency of decoding, if the GPU resources in the first processing module are not sufficient to decode the media data, the media data is sent to the second processing module through the network port link.

[0153] The second processing module, after acquiring the media data, uses its own hardware resources to complete the decoding of the media data and obtains the decoded image data. The second processing module determines whether the bit rate of the decoded image data is higher than the actual bandwidth of the network port link. If the bit rate of the decoded image data is not higher than the actual bandwidth of the network port link, the decoded image data can be directly transmitted back through the network port link. If the bit rate of the decoded image data is higher than the actual bandwidth of the network port link, the second processing module encodes the decoded image data using a preset encoding algorithm, and then sends the encoded image data to the first processing module through the network port link. In the case where the bit rate of the decoded image data is higher than the actual bandwidth of the network port link, the network port link cannot realize the real-time transmission of the decoded image data, resulting in a long data transmission time and a high data processing delay. Therefore, the decoded image data is encoded by a preset encoding algorithm so that the bit rate of the decoded image data is not higher than the actual bandwidth of the network port link. The bit rate of the decoded image data can be calculated by a related bit rate algorithm, for example, the bit rate of the decoded image data = the size of a single-frame video frame image × frame rate.

[0154] The preset encoding algorithm focuses on the simplification of the encoding algorithm and does not pursue a high compression rate. Specifically, the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for directly decoding the image data, so that the first processing module can use fewer hardware resources to complete the decoding of the encoded image data. Specifically, the decoder established by the second processing module decodes the media data, and the decoded image data obtained can be data in the format of YUV420, NV12, etc.

[0155] In one example, after the first module receives the encoded image data, it can first determine whether its own GPU resources are sufficient to decode the encoded image data. When the GPU resources are insufficient to decode the encoded image data, the encoded image data is decoded by its own CPU resources. In one example, considering that the first processing module only hands over the decoding task to the second processing module when the GPU resources are insufficient, in order to simplify the judgment logic of the first processing module and simplify unnecessary computing resource overhead, after the first processing module receives the encoded image data, it can directly use the decoder established based on the CPU resources to decode the encoded image data.

[0156] In one possible implementation, see Figure 8 The above process can be implemented through the following steps:

[0157] Step A, in response to receiving a processing task of media data, determining whether image data in the media data in this processing process needs to be rendered and displayed. If rendering and display are required, step B is executed; if rendering and display are not required, step E is executed.

[0158] Step B: if the image data needs to be rendered and displayed during this processing, determine whether the hardware resource information of the second processing module meets the first preset resource condition. If the first preset resource condition is met, execute step C; if not, execute step D.

[0159] Step C, when the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, the rendering position information of the image data is obtained, and the rendering position information is sent to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information.

[0160] Step D: When the hardware resources of the second processing module do not meet the first preset resource condition, the media data is decoded by using a decoder established by the first operating system, and the decoded image data is rendered and displayed.

[0161] Step E: If the image data does not need to be rendered and displayed during this processing, determine whether the GPU resources of the first processing module meet the second preset resource condition. If the second preset resource condition is met, execute step F; if the second preset resource condition is not met, execute step G.

[0162] Step F: When the GPU resources of the first processing module meet the second preset resource condition, the image data is processed using a decoder established based on the GPU resources by the first operating system.

[0163] Step G, when the GPU resources of the first processing module do not meet the second preset resource condition, determine whether the hardware resource information of the second processing module meets the first preset resource condition. If the first preset resource condition is met, execute step H; if the first preset resource condition is not met, execute step J.

[0164] Step H: sending the media data to the second processing module.

[0165] After receiving the media data, the second processing module sends the decoded image data to the first processing module through the network port link if the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is not higher than the bandwidth of the network port link; and after decoding the image data using a decoder established by the second operating system, encoding the decoded image data using a preset encoding algorithm to obtain encoded image data, and sending the encoded image data to the first processing module through the network port link if the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link.

[0166] Step I, obtaining the encoded image data sent by the second processing module, and decoding the encoded image data using a decoder established by the first operating system based on CPU resources; or, obtaining the decoded image data sent by the second processing module.

[0167] Step J: Generate an alarm message indicating that the current hardware resources are insufficient.

[0168] The electronic device according to the embodiment of the present application can be developed through a universal method, that is, system standard API development, rather than development based on the API provided by a specified SDK, which is more universal.

[0169] In order to illustrate the method of the embodiment of the present application, the following is described in conjunction with specific embodiments:

[0170] Step 1. Establish the connection between the Android system module (corresponding to the first processing module) and the dedicated chip (using the Linux operating system internally) system module (corresponding to the second processing module, hereinafter referred to as the L system). There are two main connections used. One is a bidirectional path through the network port link, which is mainly responsible for controlling the transmission of signaling and media data; the other path is a unidirectional path of HDMI (corresponding to the media interface link), which is responsible for rendering the decoded data on the screen of the Android system, see Fig. 9, where data is received through the 3559API interface in Android, then sent to the Linux System through Network 1000Mbps network communication, rendered and decoded by AppPro and DspCore, and rendered to the Screen through the HDMI 4k 60fps interface.

[0171] Step 2.1. Create resources. When creating a decoder, if you use AMediaCodec_createDecoderByType and other methods, the GPU hardware decoding and CPU soft decoding of the Android system module and the decoding resources of the L system module are all combined into the creation process of the decoder. According to a certain strategy (use the L system when the L system resources are sufficient, use the Android GPU when the L system resources are insufficient and the native Android GPU resources are sufficient, and use the Android CPU when the above are insufficient and the Android CPU / memory resources are not high), combine the Android system GPU and CPU and other resources with other L system resources to maximize the increase in encoding and decoding performance.

[0172] Step 2.2. Encode and decode data. After creating the decoder, when the user calls AMediaCodec_queueInputBuffer and other methods to return the cache, the type of the created decoder is confirmed internally. If it is the decoder of the L system module, it will be sent to the L system module through the network port link. If it is the GPU of the Android system module, it will be decoded through the GPU of the Android system module. If it is the CPU resource of the Android system module, the image data will be sent to a dedicated decoder such as FFMPEG. When the user calls AMediaCodec_getOutputBuffer and other methods to obtain the cache, the processing method is similar to the former. If it is the L system, it will try to obtain the decoded data from the L system.

[0173] Step 2.3. Destroy the decoder. When the user calls AMediaCodec_delete to delete the decoder, the decoder is eliminated and hardware resources are released.

[0174] Step 3.1. Add a new space for saving position coordinate variables. Add a preset field to the original surface information structure to save the obtained option position information, which can be a position coordinate variable field.

[0175] Step 3.2. Get the position coordinates when creating and changing the surface. When the user creates a view and changes the position of the view, the system will call the surfaceCreated creation method and the surfaceChanged position change method. At this time, the rendering position information is obtained from surfaceFlinger and the obtained position coordinates are saved to the position coordinate variables added in Step 3.1.

[0176] Step 3.3. Transmit the rendering position information to the L system module. When the user sends the surface information to the decoder, the decoder will always pay attention to the position coordinate information saved by the surface. When the rendering position information changes, the new rendering position information will be transmitted to the L system module through the network to update the rendering display position.

[0177] Step 3.4. Hand over the display right of the surface location to the L system module. After confirming that the decoder uses the L system module, modify the surface control input properties through the TvManager management module, and switch the surface input source from the Android system default channel to the HDMI channel. That is, from the perspective of the Android system, the surface location will no longer display content. If the L system does not render and display, the surface location will be black and will not display any content.

[0178] Step 3.5. The L system module renders and displays. After obtaining the rendering position information, it renders and displays according to the coordinates of the rendering position information. The L system module displays the image at the surface coordinate position through the HDMI link. Dspcore decodes the data and displays it to the specified position of the Android screen through HDMI.

[0179] Step 4. If the surface rendering is not required, the decoded data needs to be retrieved. The content introduced in Step 3 does not need to be implemented, but it is necessary to consider retrieving the decoded data from the L system. Since the decoded data is very large (the data volume of 1 channel 1080P25fps reaches 590Mbps), even a gigabit network card cannot support it, so the following processing is required: if there is a decoding task (not displayed through the surface), first determine whether the GPU resources of the Android system module are sufficient to process the media data. If the GPU resources are sufficient to process the media data, directly use the GPU resources for decoding; if the GPU resources are not sufficient to process the media data, determine whether the hardware resources of the L system module are sufficient to process the media data; if the hardware resources of the L system module are sufficient to process the media data, send the media data to the L system module. Unlike Step 2, the L system module needs to encode the decoded data. At this time, the compression rate does not need to be considered. Only an efficient encoding algorithm method, such as JPEG (1080P 25fps, the data volume is about 40Mbps, which can meet 20 channels of 1080P), is required. Then it is transmitted back to the Android system module through the network port link, and the Android system module performs simple decoding through the CPU. If the hardware resources of the L system module are insufficient to process the media data, an alarm message of insufficient processing resources is generated.

[0180] In a second aspect of an embodiment of the present application, a media data processing method is provided, which is applied to a second processing module in an electronic device, wherein the electronic device further comprises a first processing module, wherein a first operating system is running in the first processing module, and a second operating system is running in the second processing module, see Fig.10 , the above method comprises:

[0181] Step S101, receiving media data sent by a first processing module, wherein the media data is sent by the first processing module to a second processing module when the hardware resources of the second processing module meet a first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data;

[0182] Step S102: Decoding the media data using a decoder established by the second operating system.

[0183] Optionally, the media data includes image data; the first processing module is connected to the second processing module via a network port link and a media interface link; the method further includes:

[0184] In the case where the image data needs to be rendered and displayed during this processing, obtaining rendering position information sent by the first processing module through the network port link;

[0185] In response to the rendering position information, the decoded image data output by the decoder is rendered and displayed in the display window through the media interface link.

[0186] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link; and the method further includes:

[0187] In the case that the image data does not need to be rendered and displayed during this processing, and the bit rate of the decoded image data is not higher than the bandwidth of the network port link, the decoded image data is sent to the first processing module through the network port link;

[0188] Or, when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, the decoded image data is encoded using a preset encoding algorithm to obtain encoded image data, and the encoded image data is sent to the first processing module through the network port link, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data.

[0189] Optionally, the above method further includes:

[0190] After receiving the media data sent by the first processing module, establishing a decoder using the second operating system;

[0191] After the decoding process of the media data is completed, the corresponding decoder is deleted and the corresponding hardware resources in the second processing module are released.

[0192] It can be seen that through the method of the embodiment of the present application, the remaining hardware resources in the second processing module are sufficient to process the media data, and the media data can be sent to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data. Since the software running in the system is run through the first processing module, and the media data is decoded through the second processing module, the decoding efficiency of the media data can be improved.

[0193] In a third aspect of the embodiments of the present application, a media data processing device is provided, which is applied to a first processing module in an electronic device, wherein the electronic device further includes a second processing module, wherein a first operating system is running in the first processing module, and a second operating system is running in the second processing module, see Fig.11 , the above-mentioned device comprises:

[0194] The information acquisition module 1101 is used to acquire the media data to be processed and the hardware resource information of the second processing module;

[0195] The data sending module 1102 is used to send the media data to the second processing module when the hardware resource information of the second processing module meets the first preset resource condition, so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

[0196] Optionally, the above device further comprises:

[0197] A data decoding module, configured to decode the media data using a decoder established by the first operating system when the hardware resources of the second processing module do not meet the first preset resource condition;

[0198] Optional, data decoding module, including:

[0199] A resource determination submodule, configured to determine whether the GPU resources of the first processing module meet a second preset resource condition when the hardware resources of the second processing module do not meet the first preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data;

[0200] A first processing submodule, configured to process the image data using a decoder established by the first operating system based on GPU resources if a second preset resource condition is met;

[0201] A second processing submodule, configured to process the image data using a decoder established by the first operating system based on CPU resources if the second preset resource condition is not met;

[0202] Optionally, the media data includes image data; and the above-mentioned device further includes:

[0203] An image acquisition module, used for acquiring rendering position information of image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information;

[0204] Optionally, the first processing module is connected to the second processing module via a network port link and a media interface link;

[0205] Image acquisition module, including:

[0206] The position detection submodule is used to obtain the position information of the display window and the rendering position information when detecting that the display window of the image data is created or changed;

[0207] An input switching submodule, used to switch the input source of the display window to a media interface link;

[0208] The window rendering submodule is used to send the rendering position information to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data into the display window through the media interface link according to the rendering position information;

[0209] Optionally, the media data includes image data, and the data sending module includes:

[0210] A first sending submodule, configured to send the media data to the second processing module if the image data needs to be rendered and displayed during the current processing and the hardware resource information of the second processing module meets the first preset resource condition;

[0211] The second sending submodule is used to send the media data to the second processing module when the image data does not need to be rendered and displayed during the current processing, the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data.

[0212] Optionally, the above device further comprises:

[0213] a coded data acquisition module, used to acquire coded image data sent by the second processing module, wherein the coded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system, and encoding the decoded image data using a preset encoding algorithm, wherein the amount of hardware resources required by the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; and an image decoding submodule, used to decode the coded image data using a decoder established by the first operating system based on CPU resources;

[0214] Or, a decoded data acquisition module is used to acquire decoded image data sent by the second processing module, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system.

[0215] Optionally, the above device further comprises:

[0216] The image processing module is used to process the image data using a decoder established based on the GPU resources through the first operating system when the image data does not need to be rendered and displayed during the current processing and the GPU resources of the first processing module meet the third preset resource condition.

[0217] It can be seen that through the device of the embodiment of the present application, when the remaining hardware resources of the second processing module are sufficient to process the media data, the media data can be sent to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data. Since the software running in the system is run through the first processing module, and the media data is decoded through the second processing module, the decoding efficiency of the media data can be improved.

[0218] In a fourth aspect of the embodiments of the present application, a media data processing device is provided, which is applied to a second processing module in an electronic device, wherein the electronic device further comprises a first processing module, wherein a first operating system is running in the first processing module, and a second operating system is running in the second processing module, see Fig.12 , the above-mentioned device comprises:

[0219] The media data receiving module 1201 is used to receive the media data sent by the first processing module, wherein the media data is sent by the first processing module to the second processing module when the hardware resources of the second processing module meet the first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data;

[0220] The media data decoding module 1202 is used to decode the media data using a decoder established by the second operating system.

[0221] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link and a media interface link; the above device also includes:

[0222] A rendering position acquisition module, used to acquire rendering position information sent by the first processing module through the network port link when the image data needs to be rendered and displayed during the current processing;

[0223] A rendering position display module, for rendering and displaying decoded image data output by the decoder into a display window via a media interface link in response to the rendering position information;

[0224] Optionally, the media data includes image data; the first processing module and the second processing module are connected via a network port link; the above device also includes:

[0225] A coded image acquisition module is used to encode the decoded image data using a preset coding algorithm to obtain the coded image data when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, and send the coded image data to the first processing module through the network port link, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset coding algorithm is less than the amount of hardware resources required for decoding the image data;

[0226] Or, a decoded image acquisition module is used to send the decoded image data to the first processing module through the network port link when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is not higher than the bandwidth of the network port link.

[0227] Optionally, the above device further comprises:

[0228] A decoder establishment module, configured to establish a decoder using a second operating system after receiving the media data sent by the first processing module;

[0229] The resource release module is used to delete the corresponding decoder and release the corresponding hardware resources in the second processing module after completing the decoding process of the media data.

[0230] It can be seen that through the device of the embodiment of the present application, when the remaining hardware resources of the second processing module are sufficient to process the media data, the media data can be sent to the second processing module, so that the second processing module uses the decoder established by the second operating system to decode the media data. Since the software running in the system is run through the first processing module, and the media data is decoded through the second processing module, the decoding efficiency of the media data can be improved.

[0231] The embodiment of the present application also provides a chip, including at least one processing unit and a storage unit;

[0232] A storage unit for storing computer programs;

[0233] The processing unit is used to implement any of the above-mentioned media data processing methods applied to the first processing module when executing the program stored in the storage unit.

[0234] In one example, the processing unit of the chip includes a GPU and a CPU, and the storage unit of the chip can be a separate memory or an embedded storage core (Die).

[0235] The embodiment of the present application also provides a chip, including at least one processing unit and a storage unit;

[0236] A storage unit for storing computer programs;

[0237] The processing unit is used to implement any of the above-mentioned media data processing methods applied to the second processing module when executing the program stored in the storage unit.

[0238] In one example, the processing unit of the chip includes a GPU, and the storage unit of the chip can be a separate memory or an embedded storage core (Die).

[0239] The memory mentioned above may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0240] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0241] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, any of the above-mentioned media data processing methods applied to the first processing module is implemented.

[0242] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned media data processing methods applied to the second processing module is implemented.

[0243] According to a ninth aspect of the embodiments of the present application, an electronic device is provided, including:

[0244] A first processing module and a second processing module, wherein a first operating system runs in the first processing module and a second operating system runs in the second processing module;

[0245] A first processing module, used to implement any of the above media data processing methods applied to the first processing module during operation;

[0246] The second processing module is used to implement any of the above media data processing methods applied to the second processing module during runtime.

[0247] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any media data processing method applied to the first processing module in the above embodiments.

[0248] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any media data processing method applied to the second processing module in the above embodiments.

[0249] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0250] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0251] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0252] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A media data processing method, characterized in that: A first processing module is applied to an electronic device, the electronic device further comprising a second processing module, a first operating system is run in the first processing module, and a second operating system is run in the second processing module, the method comprising: Acquire the media data to be processed and the hardware resource information of the second processing module; When the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

2. The method according to claim 1, characterized in that The method further comprises: When the hardware resources of the second processing module do not meet the first preset resource condition, the media data is decoded using a decoder established by the first operating system.

3. The method according to claim 2, characterized in that The method of decoding the media data by using a decoder established by the first operating system when the hardware resources of the second processing module do not meet the first preset resource condition includes: In the case where the hardware resources of the second processing module do not meet the first preset resource condition, determining whether the GPU resources of the first processing module meet the second preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data, and the media data includes image data; If the second preset resource condition is met, processing the image data using a decoder established by the first operating system based on GPU resources; If the second preset resource condition is not met, the image data is processed using a decoder established by the first operating system based on CPU resources.

4. The method according to claim 1, characterized in that: The media data includes image data; the method further includes: The rendering position information of the image data is acquired, and the rendering position information is sent to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information.

5. The method according to claim 4, characterized in that The first processing module and the second processing module are connected via a network port link and a media interface link; The obtaining the rendering position information of the image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information, comprises: When it is detected that a display window of the image data is created or changed, acquiring position information of the display window to obtain rendering position information; Switching the input source of the display window to the media interface link; The rendering position information is sent to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data in the display window through the media interface link according to the rendering position information.

6. The method according to claim 1, characterized in that The media data includes image data, and when the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module includes: In the current processing process, if the image data needs to be rendered and displayed, and the hardware resource information of the second processing module meets the first preset resource condition, sending the media data to the second processing module; And / or, when the image data does not need to be rendered and displayed during this processing, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, the media data is sent to the second processing module, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data.

7. The method according to claim 6, characterized in that In the current processing process, if the image data does not need to be rendered and displayed, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, after the step of sending the media data to the second processing module, the method further includes: Acquire the encoded image data sent by the second processing module, wherein the encoded image data is obtained by: the second processing module decoding the image data using a decoder established by the second operating system, and encoding the decoded image data using a preset encoding algorithm, the amount of hardware resources required by the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; and decode the encoded image data using a decoder established by the first operating system based on CPU resources; or, obtaining decoded image data sent by the second processing module, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system; The method further comprises: In this processing, if the image data does not need to be rendered and displayed, and the GPU resources of the first processing module meet the third preset resource condition, the image data is processed using a decoder established based on the GPU resources by the first operating system.

8. A media data processing method, characterized in that: A second processing module is applied to an electronic device, the electronic device further comprising a first processing module, a first operating system is running in the first processing module, and a second operating system is running in the second processing module, the method comprising: receiving media data sent by the first processing module, wherein the media data is sent by the first processing module to the second processing module when the hardware resources of the second processing module meet a first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data; The media data is decoded using a decoder established by the second operating system.

9. The method according to claim 8, characterized in that The media data includes image data; the first processing module and the second processing module are connected via a network port link and a media interface link; the method further includes: In the case where the image data needs to be rendered and displayed during this processing, obtaining rendering position information sent by the first processing module through the network port link; In response to the rendering position information, the decoded image data output by the decoder is rendered and displayed in a display window through the media interface link.

10. The method according to claim 8, characterized in that The media data includes image data; the first processing module and the second processing module are connected via a network port link; the method further includes: In the case that the image data does not need to be rendered and displayed during this processing, and the bit rate of the decoded image data is not higher than the bandwidth of the network port link, the decoded image data is sent to the first processing module through the network port link; Or, when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, the decoded image data is encoded using a preset encoding algorithm to obtain encoded image data, and the encoded image data is sent to the first processing module through the network port link, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data.

11. The method according to claim 8, characterized in that The method further comprises: After receiving the media data sent by the first processing module, establishing a decoder using the second operating system; After the decoding process of the media data is completed, the corresponding decoder is deleted and the corresponding hardware resources in the second processing module are released.

12. A media data processing device, characterized in that: A first processing module applied to an electronic device, the electronic device further comprising a second processing module, a first operating system running in the first processing module, a second operating system running in the second processing module, the device comprising: An information acquisition module, used to acquire the media data to be processed and the hardware resource information of the second processing module; A data sending module is used to send the media data to the second processing module when the hardware resource information of the second processing module meets the first preset resource condition, so that the second processing module uses a decoder established by the second operating system to decode the media data; wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data.

13. The device according to claim 12, characterized in that The device also includes: a data decoding module, configured to decode the media data using a decoder established by the first operating system when the hardware resources of the second processing module do not meet the first preset resource condition; The data decoding module comprises: a resource determination submodule, configured to determine whether the GPU resources of the first processing module meet a second preset resource condition when the hardware resources of the second processing module do not meet the first preset resource condition; wherein the second preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data, wherein the media data includes image data; A first processing submodule, configured to process the image data by using a decoder established by the first operating system based on GPU resources if the second preset resource condition is met; A second processing submodule, configured to process the image data using a decoder established by the first operating system based on CPU resources if the second preset resource condition is not met; The media data includes image data; the device also includes: An image acquisition module, used for acquiring rendering position information of the image data, and sending the rendering position information to the second processing module, so that the second processing module renders and displays the decoded image data according to the rendering position information; The first processing module and the second processing module are connected via a network port link and a media interface link; The image acquisition module comprises: A position detection submodule, for acquiring position information of the display window and obtaining rendering position information when detecting that the display window of the image data is created or changed; An input switching submodule, used for switching the input source of the display window to the media interface link; A window rendering submodule, configured to send the rendering position information to the second processing module through the network port link, so that the second processing module renders and displays the decoded image data into the display window through the media interface link according to the rendering position information; The media data includes image data, and the data sending module includes: A first sending submodule, configured to send the media data to the second processing module if the image data needs to be rendered and displayed during this processing and the hardware resource information of the second processing module meets a first preset resource condition; a second sending submodule, configured to send the media data to the second processing module when the image data does not need to be rendered and displayed during the current processing, and the GPU resources of the first processing module do not meet the third preset resource condition, and the hardware resource information of the second processing module meets the first preset resource condition, wherein the third preset resource condition indicates that the remaining hardware resources of the GPU of the first processing module are sufficient to process the media data; The device also includes: a coded data acquisition module, used for acquiring coded image data sent by the second processing module, wherein the coded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system and encoding the decoded image data using a preset encoding algorithm, wherein the amount of hardware resources required by the decoding algorithm corresponding to the preset encoding algorithm is less than the amount of hardware resources required for decoding the image data; an image decoding submodule, used for decoding the coded image data using a decoder established by the first operating system based on CPU resources; or, a decoded data acquisition module, used for acquiring decoded image data sent by the second processing module, wherein the decoded image data is obtained by the second processing module decoding the image data using a decoder established by the second operating system; The device also includes: An image processing module is used to process the image data using a decoder established based on the GPU resources through the first operating system when the image data does not need to be rendered and displayed during this processing and the GPU resources of the first processing module meet the third preset resource conditions.

14. A media data processing device, characterized in that: A second processing module applied to an electronic device, the electronic device further comprising a first processing module, a first operating system running in the first processing module, a second operating system running in the second processing module, the device comprising: a media data receiving module, configured to receive media data sent by the first processing module, wherein the media data is sent by the first processing module to the second processing module when the hardware resources of the second processing module meet a first preset resource condition, wherein the first preset resource condition indicates that the remaining hardware resources of the second processing module are sufficient to process the media data; The media data decoding module is used to decode the media data using a decoder established by the second operating system.

15. The device according to claim 14, characterized in that The media data includes image data; the first processing module and the second processing module are connected via a network port link and a media interface link; the device also includes: A rendering position acquisition module, used for acquiring rendering position information sent by the first processing module through the network port link when the image data needs to be rendered and displayed during the current processing; A rendering position display module, configured to render and display the decoded image data output by the decoder into a display window via the media interface link in response to the rendering position information; The device also includes: A decoded image acquisition module, used for sending the decoded image data to the first processing module through the network port link when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is not higher than the bandwidth of the network port link; Or, a coded image acquisition module, used for encoding the decoded image data using a preset coding algorithm to obtain coded image data when the image data does not need to be rendered and displayed during this processing and the bit rate of the decoded image data is higher than the bandwidth of the network port link, and sending the coded image data to the first processing module through the network port link, wherein the amount of hardware resources required for the decoding algorithm corresponding to the preset coding algorithm is less than the amount of hardware resources required for decoding the image data; The device also includes: A decoder establishment module, configured to establish a decoder using the second operating system after receiving the media data sent by the first processing module; The resource release module is used to delete the corresponding decoder and release the corresponding hardware resources in the second processing module after completing the decoding process of the media data.

16. An electronic device, characterized in that: include: A first processing module and a second processing module, wherein a first operating system runs in the first processing module and a second operating system runs in the second processing module; The first processing module is used to implement the method according to any one of claims 1 to 7 during operation; The second processing module is used to implement the method described in any one of claims 8-11 during runtime.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 11 are implemented.

18. A chip, characterized in that: comprising at least one processing unit and a storage unit; A storage unit for storing computer programs; A processing unit, for implementing the method steps described in any one of claims 1-7 or 8-11 when executing a program stored in a storage unit.

Images