Multimedia data coding and decoding method and device based on multimedia framework

Abstract

The embodiment of the invention discloses a multimedia data encoding and decoding method and device based on a multimedia framework. The multimedia framework comprises middleware and a drive adaptation layer, the middleware provides an OpenMAX interface, the drive adaptation layer provides a common interface irrelevant to an operating system, and the common interface is packaged with drive program interfaces in different operating systems; receiving an address and control information of to-be-processed multimedia data sent by an application program through the OpenMAX interface, and transmitting the address and control information to the drive adaptation layer; and calling a driving program through the public interface to trigger the driving program to obtain the multimedia data to be processed according to the address, and driving a codec corresponding to the control information to perform coding and decoding processing on the multimedia data to be processed. According to the method and the device, various types of multimedia data can be flexibly processed, the method and the device are not limited to specific types of input sources, the method and the device can be operated on various operating systems, and the cost and difficulty of cross-platform development are reduced.

Description

Technical Field

[0001] The present application relates to the field of navigation technology, and in particular to a multimedia data encoding and decoding method and device based on a multimedia framework. Background Art

[0002] In the QNX (QNX Neutrino Real-Time Operating System) operating system, the existing multimedia interfaces are mainly divided into two categories: client API (Application Programming Interface) provided by the service process based on the media server (such as mm-stream and mm-renderer), and QNX-OpenMAX interface similar to the standard OpenMAX. Although these interfaces provide certain convenience for users to realize multimedia functions, there are many limitations in practical applications, which makes it difficult to meet the growing demand for multimedia application development. First, although the client API using the media server service process is simple to call, its internal implementation and interface have been solidified, and only support network protocols (such as RTP, RTSP) and local files as input sources, which cannot meet the needs of users to process other types of input sources, and seriously limit its application scope and scenarios. Secondly, the QNX-OpenMAX interface is incompatible with the standard OpenMAX, which requires users to invest a lot of time and energy to familiarize themselves with its specific interface functions, increasing the learning cost. In addition, the platform relevance of the interface makes it difficult to port to other operating systems, greatly increasing the cost and difficulty of cross-platform development.

[0003] With the increasing number of multimedia applications, users' demand for efficient development and cross-platform porting is growing. The existing QNX multimedia interface can no longer meet the needs of users due to the above problems, which has brought great obstacles to the development and promotion of multimedia functions. Therefore, it is particularly urgent to develop a more effective multimedia framework to solve these problems. Summary of the invention

[0004] The present application provides a multimedia data encoding and decoding method and device based on a multimedia framework.

[0005] This application provides the following solutions:

[0006] According to a first aspect, a multimedia data encoding and decoding method based on a multimedia framework is provided, wherein the multimedia framework includes a middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a public interface independent of an operating system, wherein the public interface encapsulates driver interfaces in different operating systems; the method includes:

[0007] Receive the address and control information of the multimedia data to be processed sent by the application through the OpenMAX interface of the middleware, and transfer them to the driver adaptation layer;

[0008] Call the driver program through the common interface of the driver adaptation layer to trigger the driver program to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to perform codec processing on the multimedia data to be processed.

[0009] Optionally, the multimedia framework further includes an APP docking layer, and the APP docking layer provides an API interface;

[0010] The receiving the address and control information of the multimedia data to be processed sent by the application through the OpenMAX interface of the middleware includes:

[0011] Receive the address and the control information through the API interface of the APP docking layer;

[0012] The APP docking layer converts the address and control information into a format adapted to the OpenMAX interface, and transfers the converted address and control information to the middleware through the OpenMAX interface.

[0013] Optionally, the API interface includes at least one of the following:

[0014] An initialization interface for creating a multimedia session instance;

[0015] A session configuration interface for configuring the session parameters of the multimedia session instance;

[0016] A data input interface for inputting the address and the control information into the middleware;

[0017] A result acquisition interface for acquiring the processing result obtained after the codec performs codec processing on the multimedia data to be processed;

[0018] A resource release interface for releasing the resources occupied by the processing result;

[0019] A session destruction interface for releasing the multimedia session instance.

[0020] Optionally, the call sequence of the API interface is: the initialization interface, the session configuration interface, the data input interface, the result acquisition interface, the resource release interface, and the session destruction interface.

[0021] Optionally, the transferring to the driver adaptation layer includes:

[0022] The address and the control information are converted into a format adapted to the public interface and passed to the driver adaptation layer.

[0023] Optionally, the OpenMAX interface is an interface of an OpenMAX integration layer.

[0024] Optionally, the public interface of the driver adaptation layer controls the driver program by calling an ioctl control function.

[0025] According to a second aspect, a multimedia data encoding and decoding device based on a multimedia framework is provided, wherein the multimedia framework includes a middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a common interface independent of the operating system, and the common interface encapsulates a driver program interface in different operating systems; the device includes a middleware module and a driver adaptation layer module, wherein:

[0026] The middleware module is configured to receive the address and control information of the to-be-processed multimedia data sent by the application program through the OpenMAX interface, and pass the information to the driver adaptation layer module;

[0027] The driver adaptation layer module is configured to call the driver program through the public interface to trigger the driver program to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to perform encoding and decoding processing on the multimedia data to be processed.

[0028] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

[0029] According to a fourth aspect, there is provided an electronic device, comprising:

[0030] one or more processors; and

[0031] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the first aspects above.

[0032] According to a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of any one of the methods described in the first aspect.

[0033] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0034] This application is applied in a multimedia framework composed of a middleware and a driver adaptation layer. The address and control information of the multimedia data to be processed sent by the application are received through the OpenMAX interface of the middleware, and passed to the driver adaptation layer, and then the driver is called through the public interface of the driver adaptation layer to trigger the driver to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to encode and decode the multimedia data to be processed. This architecture decouples the application from the underlying hardware, allowing the multimedia framework to flexibly process various types of multimedia data and is no longer limited to a specific type of input source. In addition, the driver adaptation layer provides a public interface that is independent of the operating system, encapsulates the driver interface in different operating systems, and enables the multimedia framework to run on multiple operating systems, reducing the cost and difficulty of cross-platform development. At the same time, the driver is called by the driver adaptation layer, and the hardware codec is directly used for encoding and decoding, which improves processing efficiency and meets the needs of efficient development.

[0035] Of course, any product implementing the present application does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 4 is a flowchart of a multimedia data encoding and decoding method based on a multimedia framework provided in an embodiment of the present application;

[0038] Figure 2 A schematic diagram of the structure of a multimedia framework provided in an embodiment of the present application;

[0039] Figure 3 A flowchart for implementing the data input interface provided in the embodiment of the present application;

[0040] Figure 4 A flowchart for implementing the result acquisition interface provided in the embodiment of the present application;

[0041] Figure 5 A schematic block diagram of a multimedia data encoding and decoding device based on a multimedia framework provided in an embodiment of the present application;

[0042] Figure 6 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0044] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0045] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0046] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0047] As mentioned above, the existing QNX multimedia interface can only support fixed types of input sources and has poor flexibility. On the other hand, it cannot realize cross-operating system applications. For different operating systems, codes need to be written and maintained separately, which obviously wastes development costs and has low development efficiency.

[0048] In view of this, the present application provides a new idea. First, a new multimedia framework is provided, which consists of a middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a public interface that is independent of the operating system, and the public interface encapsulates the driver interface in different operating systems. The multimedia framework can be installed in different operating systems, such as the QNX operating system, the Linux operating system, etc.

[0049] Based on the above multimedia framework, an embodiment of the present application provides a multimedia data encoding and decoding method based on the multimedia framework. Figure 1The flowchart of the multimedia data encoding and decoding method based on the multimedia framework provided in the embodiment of the present application is applicable to a computer device with an operating system installed, and can be set on a server side or a computer terminal. The method may include the following steps:

[0050] Step 101: receiving the address and control information of the multimedia data to be processed sent by the application through the OpenMAX interface of the middleware, and passing it to the driver adaptation layer;

[0051] Step 102: calling the driver program through the public interface of the driver adaptation layer to trigger the driver program to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to perform encoding and decoding processing on the multimedia data to be processed.

[0052] It can be seen that the embodiment of the present application decouples the application from the underlying hardware, so that the multimedia framework can flexibly process various types of multimedia data and is no longer limited to a specific type of input source. In addition, the driver adaptation layer provides a public interface that is independent of the operating system and encapsulates the driver interface in different operating systems, so that the multimedia framework can run on multiple operating systems, reducing the cost and difficulty of cross-platform development. At the same time, by calling the driver through the driver adaptation layer, the hardware codec is directly used for encoding and decoding processing, which improves processing efficiency and meets the needs of efficient development.

[0053] The steps in the above process and the effects that can be further produced are described in detail below in conjunction with embodiments.

[0054] The above step 101, namely "receiving the address and control information of the multimedia data to be processed sent by the application through the OpenMAX interface of the middleware, and passing it to the driver adaptation layer" is described in detail below in conjunction with the embodiments.

[0055] An application is software running on a user device and used to perform multimedia data processing tasks, such as video playback, audio recording, image editing, etc. In the embodiment of the present application, the application may be, but is not limited to, a video player for playing video files stored locally or on a network.

[0056] The address of the multimedia data to be processed refers to the physical or logical location where the multimedia data is stored. In the embodiment of the present application, the address usually refers to a memory address.

[0057] If the multimedia data to be processed is stored in a file system, the application needs to read the file and load the data into a memory, thereby obtaining the memory address of the multimedia data to be processed.

[0058] If the multimedia data to be processed comes from the network, the application needs to download the data through a network protocol (such as HTTP) and load the data into the memory, thereby obtaining the memory address of the multimedia data to be processed.

[0059] Control information refers to instructions sent by the application to the multimedia framework to control the processing method of the multimedia data to be processed. Control information can include but is not limited to the following:

[0060] Codec parameters: such as video resolution, frame rate, etc.

[0061] Processing instructions: such as start decoding, stop decoding, etc.

[0062] In an embodiment of the present application, the application can directly call the OpenMAX interface of the middleware to send the address and control information of the multimedia data to be processed to the middleware. The OpenMAX interface can flexibly process various types of input sources, thereby greatly improving the flexibility of multimedia data processing.

[0063] Optionally, the OpenMAX interface in the embodiment of the present application is an interface of the OpenMAX Integration Layer (IL). The IL layer provides a standardized media component interface, enabling developers and platform providers to integrate and communicate with multimedia codecs implemented in hardware or software.

[0064] In addition, when the middleware passes the received address and control information to the driver adaptation layer, the address and control information may be first converted into a format adapted to a public interface in the driver adaptation layer, and then passed to the driver adaptation layer.

[0065] It should be noted that since the OpenMAX interface can process multiple types of multimedia data, if the application calls these interfaces directly, it needs to process a large number of parameters and call sequences, which will increase the complexity of development. For example, the OpenMAX integration layer defines multiple component roles and port types, and each component and port has a specific configuration and calling method, which requires developers to have a deep understanding of the OpenMAX architecture and interface details, increasing the difficulty of development and the cost of learning.

[0066] Therefore, in order to further reduce the difficulty of development, the multimedia framework in the embodiment of the present application may also include an APP docking layer, which provides an API interface.

[0067] In this case, step 101 may specifically include:

[0068] First, the address and control information of the multimedia data to be processed are received through the API interface of the APP docking layer; then the APP docking layer converts the address and control information into a format adapted to the OpenMAX interface, and passes the converted address and control information to the middleware.

[0069] The API interface may include but is not limited to:

[0070] Initialization interface, used to create a multimedia session instance;

[0071] The session configuration interface is used to configure the session parameters of the multimedia session instance;

[0072] A data input interface, used to input address and control information into the middleware;

[0073] The result acquisition interface is used to obtain the processing result obtained by the codec after encoding and decoding the multimedia data to be processed;

[0074] Resource release interface, used to release the resources occupied by the processing results;

[0075] The session destruction interface is used to release the multimedia session instance.

[0076] The above API interface is for application calls. The interface is simple and easy to connect, and can meet most usage scenarios.

[0077] Furthermore, the calling sequence of the above API interface is: initialization interface, session configuration interface, data input interface, result acquisition interface, resource release interface and session destruction interface.

[0078] The calling sequence of the API interface refers to calling these interfaces in a certain order in the application program to implement a complete multimedia data encoding and decoding process.

[0079] The following is an example to illustrate the calling sequence in the embodiment of the present application.

[0080] 1) Initialization Interface

[0081] Corresponding function: void*decode_session_create(dec_param_t*param);

[0082] Function: Create a decoding session instance, which is the starting point of the entire process.

[0083] When to call: Before starting to process any multimedia data.

[0084] Input parameters: param is the decoding parameters, such as video compression format, resolution, etc.

[0085] Return value: pointer to the decoded session for subsequent function calls.

[0086] Description: This interface is used to initialize the session and prepare necessary resources for subsequent operations.

[0087] 2) Session Configuration Interface

[0088] Corresponding function 1: int session_set_event_callback(void*session,void*obj,void*priv_obj,event_cb_ptr cb);

[0089] Function: Set the callback function of the decoding event, which is used for interactive communication between the framework and the application layer (i.e., application layer) code.

[0090] Calling time: After creating the decoding session and before starting to input data.

[0091] Input parameters:

[0092] session: Decode the session pointer.

[0093] obj and priv_obj: parameters and private parameters of the callback function.

[0094] cb: callback function pointer.

[0095] Return value: BST_OK indicates success, other values ​​indicate failure.

[0096] Description: This interface is used to configure the event handling mechanism of the session to ensure that the application can receive notifications from the framework.

[0097] Corresponding function 2: int decode_set_output_bufs(void*dec,int cnt,int*fd);

[0098] Function: Set the fd (file descriptor) and number of decoded output dma-buf, and support the input of buffer allocated in the application layer.

[0099] Calling time: After creating the decoding session and before starting to input data.

[0100] Input parameters:

[0101] dec: decode session pointer.

[0102] cnt: Enter the number of dma-bufs.

[0103] fd: fd array of input dma-buf.

[0104] Return value: BST_OK indicates success, other values ​​indicate failure.

[0105] Description: This interface is used to configure the decoder's output buffer to ensure that the application can receive the decoded data.

[0106] 3) Data Input Interface

[0107] Corresponding function: int decode_write_input_data(void*dec,stream_buf_t*buf);

[0108] Function: Input code stream to decoder.

[0109] Calling time: After completing session configuration and before starting to process data.

[0110] Input parameters:

[0111] dec: decode session pointer.

[0112] buf: compressed video stream.

[0113] Return value:

[0114] BST_OK: Success.

[0115] BST_ERROR_RETRY: Indicates that the buffer in the framework is full and no free buffer is available. Try calling this function again later.

[0116] Other values: Failed.

[0117] Description: This interface is used to input the multimedia data to be processed into the decoder and start the encoding and decoding process.

[0118] 4) Result Retrieval Interface

[0119] Corresponding function 1: int decode_get_output_bufs(void*dec,int*cnt,int*fd);

[0120] Function: Get the fd of all decoded data dma-buf, and use dma_buf to transfer the decoded data between the application layer and the multimedia framework.

[0121] Calling time: After input data, after the decoder completes processing.

[0122] Input parameters: dec: decoding session pointer.

[0123] Output parameters:

[0124] cnt: Output dma-buf number.

[0125] fd: output dma-buf fd array.

[0126] Return value: BST_OK indicates success, other values ​​indicate failure.

[0127] Description: This interface is used to obtain the decoded data buffer for further processing by the application.

[0128] Corresponding function 2: int decode_dequeue_output_buf(void*dec,yuv_buf_t*buf);

[0129] Function: Get the buf of the decoded data. The application layer can further process the data after obtaining it.

[0130] Calling time: After input data, after the decoder completes processing.

[0131] Input parameters: dec: decoding session pointer.

[0132] Output parameter: buf: outputs the buf of the decoded data.

[0133] Return value:

[0134] BST_OK: Success.

[0135] BST_ERROR_RETRY: Indicates that no decoded data is available in the frame and you need to try calling this function again later.

[0136] Other values: Failed.

[0137] Description: This interface is used to obtain decoded data for further processing by the application, such as display or storage.

[0138] 5) Resource Release Interface

[0139] Corresponding function: int decode_release_output_buf(void*dec,int fd,int index);

[0140] Function: After the application layer processes the decoded data, it calls this function to release buf and return the ownership of buf to the framework layer.

[0141] Calling time: After obtaining the decoded data and completing the processing.

[0142] Input parameters:

[0143] dec: decode session pointer.

[0144] fd: dma-buf fd to be released.

[0145] index: dma-buf index to be released.

[0146] Return value: BST_OK indicates success, other values ​​indicate failure.

[0147] Description: This interface is used to release buffers that are no longer in use to ensure that resources can be effectively managed by the framework layer.

[0148] 6)Session Destruction Interface

[0149] Corresponding function: int session_destory(void*session);

[0150] Function: Destroy the decoding session and release all related resources.

[0151] Calling time: After all data processing is completed.

[0152] Input parameters: session: decoding session pointer previously created by decode_session_create.

[0153] Return value: BST_OK indicates success, other values ​​indicate failure.

[0154] Description: This interface is used to end the session and release all resources related to the decoding session.

[0155] Through this calling sequence, the application can efficiently interact with the multimedia framework to complete the encoding and decoding processing of multimedia data.

[0156] The above step 102, i.e., "calling the driver program through the public interface of the driver adaptation layer to trigger the driver program to obtain the multimedia data to be processed according to the address, and driving the codec corresponding to the control information to encode and decode the multimedia data to be processed" is described in detail below in conjunction with the embodiments.

[0157] In the embodiment of the present application, the common interface of the driver adaptation layer is a set of interfaces that are independent of the operating system and are used to encapsulate the driver interfaces in different operating systems. These interfaces provide a unified method so that the middleware can interact with the underlying hardware without having to care about the specific operating system details.

[0158] Through the public interface, the middleware can call the driver's functions without directly interacting with the hardware. This process is independent of the operating system, making the multimedia framework portable.

[0159] In the embodiment of the present application, the driver obtains the multimedia data to be processed according to the address (such as memory address, file path or network address) provided by the application. These addresses indicate the storage location of the data, and the driver accesses the multimedia data to be processed through these addresses, and selects and drives the corresponding codec to encode and decode the multimedia data to be processed according to the control information.

[0160] Among them, the codec is a hardware or software module that performs specific encoding and decoding operations and is controlled by a driver.

[0161] The codec in the embodiment of the present application performs codec processing on the multimedia data to be processed according to the instruction of the driver program. The codec processing can be decoding the compressed multimedia data into the original format (decoding), or encoding the original format multimedia data into the compressed format (encoding).

[0162] Finally, the data processed by the codec is returned to the application through the public interface. The application can obtain the processing results by calling the corresponding interface and perform further processing or display.

[0163] Furthermore, the public interface of the driver adaptation layer can control the driver by calling the ioctl control function.

[0164] In the embodiment of the present application, the encoding and decoding of multimedia data can be realized by calling the driver program through the public interface of the driver adaptation layer. The public interface provides a calling method that is independent of the operating system, so that the application program and the middleware can flexibly interact with the underlying hardware. The driver program obtains the multimedia data to be processed according to the address, and drives the corresponding codec to perform encoding and decoding according to the control information. This design improves the flexibility and portability of the system and simplifies the development of application programs.

[0165] like Figure 2 , which is a schematic diagram of a multimedia framework provided in an embodiment of the present application, including an APP docking layer (bstplayer), a middleware (OpenMAX) and a driver adaptation layer (driver wrapper).

[0166] Based on the above multimedia framework, assuming that the application needs to decode a piece of video data, the specific steps may include:

[0167] Step 1: Create a decoding session

[0168] The application calls the API interface of the APP docking layer to create a decoding session.

[0169] The APP docking layer passes the request to the middleware (OpenMAX), and the middleware passes its OpenMAX

[0170] IL interface creates a decoding session instance.

[0171] The middleware calls the public interface of the driver adaptation layer, and the driver adaptation layer calls the driver program through the ioctl control function to create a decoding session instance.

[0172] Step 2: Configure the decoding session

[0173] The application sets the parameters of the decoding session, such as video format, resolution, frame rate, etc., through the API interface of the APP docking layer.

[0174] The APP docking layer converts these parameters into a format that adapts to the OpenMAX IL interface and passes it to the middleware.

[0175] The middleware passes these parameters to the driver adaptation layer through its OpenMAX IL interface, and the driver adaptation layer calls the driver through the ioctl control function to configure the decoding session.

[0176] Step 3: Input video data

[0177] The application sends the address and control information of the video data to be decoded to the middleware through the API interface of the APP docking layer.

[0178] The APP docking layer converts the address and control information into a format that adapts to the OpenMAX IL interface and passes it to the middleware.

[0179] The middleware passes the address and control information to the driver adaptation layer through its OpenMAX IL interface. The driver adaptation layer calls the driver program through the ioctl control function to obtain the video data to be decoded.

[0180] Step 4: Decode the video data

[0181] The driver obtains the video data to be decoded according to the address, and drives the corresponding codec to perform decoding processing according to the control information.

[0182] The driver saves the decoded data to the specified buffer and notifies the middleware through the ioctl control function.

[0183] The middleware passes the decoded data to the APP docking layer through its OpenMAX IL interface, and the APP docking layer passes the data to the application.

[0184] Step 5: Get the decoding result

[0185] The application obtains the decoded video data through the API interface of the APP docking layer.

[0186] The APP docking layer passes the decoded data from the middleware to the application.

[0187] The application further processes the decoded data, such as displaying and storing it.

[0188] Step 6: Release resources

[0189] After the application completes the decoding process, it releases the resources occupied by the decoding session through the API interface of the APP docking layer.

[0190] The APP docking layer passes the release request to the middleware, and the middleware notifies the driver adaptation layer through its OpenMAX IL interface.

[0191] The driver adaptation layer calls the driver through the ioctl control function to release the resources occupied by the decoding session.

[0192] Step 7: Destroy the decoding session

[0193] The application destroys the decoding session through the API interface of the APP docking layer.

[0194] The APP docking layer passes the destruction request to the middleware, and the middleware notifies the driver adaptation layer through its OpenMAX IL interface.

[0195] The driver adaptation layer calls the driver through the ioctl control function to destroy the decoding session instance.

[0196] Through the above steps, the application can utilize the multimedia framework provided by this application to perform efficient decoding processing on the video data.

[0197] Further, Figure 2The APP docking layer can be composed of two parts, namely the interface definition layer (bstplayerAPI layer) and the control and data processing layer (bstplayer implementation layer). The interface definition layer is used to define a series of API interfaces, such as initialization interface, session configuration interface, data input interface, result acquisition interface, resource release interface and session destruction interface, etc., which will not be repeated here.

[0198] The control and data processing layer is used to complete the specific implementation of the API interface. For example, the address and control information passed down by the interface definition layer need to be converted into the format required by the OpenMAX IL interface. At the same time, OpenMAX also defines a set of complex data processing APIs and strict calling processes, and calls these interfaces to complete data processing and information interaction. In the implementation process, combined with the common usage scenarios and requirements of the application layer, the processing processes that users do not care about in the OpenMAX IL interface are hidden, and the parts related to customer functions are retained and connected to the interface definition layer.

[0199] The following introduces the implementation process of "passing data to be processed" and "obtaining decoded data".

[0200] like Figure 3 The figure shows the implementation process of "transferring data to be processed".

[0201] The implementation process specifically includes:

[0202] The application writes the multimedia data to be processed into the buffer through the data access interface (such as write_input_data interface) of the bstplayer API layer.

[0203] After receiving the data, the bstplayer implementation layer calls the free buffer interface of the OpenMAX layer (such as the EmptyThisBuffer interface) and passes the buffer to the OpenMAX layer.

[0204] The OpenMAX layer queries the driver layer whether there is an idle buffer available. If so, the buffer is passed to the driver layer and the EmptyThisBuffer interface of the driver layer is called.

[0205] After receiving the buffer, the driver layer passes the buffer to the hardware codec and calls the EmptyThisBuffer interface of the hardware codec.

[0206] After the hardware codec processes the data, it fills the processed data into the buffer and sends an interrupt signal to the driver layer.

[0207] The driver layer notifies the OpenMAX layer through an idle buffer to indicate that a message (such as the EmptyBufferDone message) has been processed, meaning the buffer has been processed completely.

[0208] The OpenMAX layer notifies the bstplayer implementation layer through the callback function of the EmptyBufferDone message, indicating that the buffer has been processed completely.

[0209] The bstplayer implementation layer notifies the bstplayer API layer through the callback function of the EmptyBufferDone message, indicating that the buffer has been processed completely.

[0210] As Figure 4 shown, it is the implementation process of "obtaining decoded data".

[0211] Its implementation process specifically includes:

[0212] The application requests the decoded data through the result acquisition interface (such as the dequeue_output_buf interface) of the bstplayer API layer.

[0213] After receiving the request, the bstplayer implementation layer calls the buffer filling interface (such as the FillThisBuffer interface) of the OpenMAX layer to request the decoded data.

[0214] The OpenMAX layer queries the driver layer to check if there is available data. At this time, the driver layer communicates with the hardware decoder to query whether the data in the hardware decoder has been decoded and stored in the buffer. If so, it calls the FillThisBuffer interface of the hardware decoder to request the decoded data.

[0215] The hardware decoder fills the decoded data into the buffer according to the request of the driver layer and sends an interrupt signal to the driver layer simultaneously;

[0216] The driver layer notifies the OpenMAX layer through the buffer filling completion message (such as the FillBufferDone message), indicating that the data is ready.

[0217] The OpenMAX layer returns the decoded data to the bstplayer implementation layer and notifies the bstplayer implementation layer through the callback function of the FillBufferDone message, indicating that the data is ready.

[0218] The bstplayer implementation layer returns the decoded data to the bstplayer API layer, and the bstplayer API layer returns the data to the application.

[0219] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] According to another embodiment, a multimedia data encoding and decoding device based on a multimedia framework is provided. Figure 5 A schematic block diagram of a map updating device according to an embodiment is shown. Figure 5 As shown, the multimedia framework includes middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a common interface that is independent of the operating system, and the common interface encapsulates driver interfaces in different operating systems. The device 500 includes a middleware module 501 and a driver adaptation layer module 502, wherein:

[0221] The middleware module 501 is configured to receive the address and control information of the multimedia data to be processed sent by the application through the OpenMAX interface, and pass it to the driver adaptation layer module 502;

[0222] The driver adaptation layer module 502 is configured to call the driver through the public interface to trigger the driver to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to encode and decode the multimedia data to be processed.

[0223] The multimedia framework further includes an APP docking layer, which provides an API interface;

[0224] The apparatus 500 may further include:

[0225] The APP docking layer module 503 is configured to receive the address and the control information through the API interface; convert the address and the control information into a format adapted to the OpenMAX interface, and pass the converted address and control information to the middleware module 501.

[0226] Optionally, the API interface includes at least one of the following:

[0227] Initialization interface, used to create a multimedia session instance;

[0228] A session configuration interface, used to configure session parameters of the multimedia session instance;

[0229] A data input interface, used to input the address and the control information into the middleware;

[0230] A result acquisition interface, used to obtain a processing result obtained after the codec performs encoding and decoding processing on the multimedia data to be processed;

[0231] Resource release interface, used to release the resources occupied by the processing result;

[0232] The session destruction interface is used to release the multimedia session instance.

[0233] Optionally, the calling sequence of the API interface is: the initialization interface, the session configuration interface, the data input interface, the result acquisition interface, the resource release interface and the session destruction interface.

[0234] Optionally, the middleware module 501 is configured as follows:

[0235] The address and the control information are converted into a format adapted to the public interface and passed to the driver adaptation layer.

[0236] Optionally, the OpenMAX interface is an interface of the OpenMAX integration layer.

[0237] Optionally, the public interface of the driver adaptation layer module 502 controls the driver program by calling an ioctl control function.

[0238] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without creative work.

[0239] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0240] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0241] And an electronic device, comprising:

[0242] one or more processors; and

[0243] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0244] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0245] in, Figure 6 The architecture of the electronic device is shown as an example, which may include a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, and a memory 620. The processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620 may be communicatively connected via a communication bus 630.

[0246] Among them, the processor 610 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0247] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store an operating system 621 for controlling the operation of the electronic device 600, and a basic input and output system (BIOS) 622 for controlling the low-level operation of the electronic device 600. In addition, a web browser 623, a data storage management system 624, a multimedia data encoding and decoding device 625 based on a multimedia framework, etc. can also be stored. The above-mentioned multimedia data encoding and decoding device 625 based on a multimedia framework can be an application program that specifically implements the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided by the present application is implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0248] The input / output interface 613 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0249] The network interface 614 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0250] The bus 630 comprises a pathway for transmitting information between the various components of the device (eg, the processor 610, the video display adapter 611, the disk drive 612, the input / output interface 613, the network interface 614, and the memory 620).

[0251] It should be noted that, although the above device only shows a processor 610, a video display adapter 611, a disk drive 612, an input / output interface 613, a network interface 614, a memory 620, a bus 630, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0252] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0253] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A multimedia data encoding and decoding method based on a multimedia framework, characterized in that: The multimedia framework includes a middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a public interface that is independent of the operating system, and the public interface encapsulates driver interfaces in different operating systems; the method includes: Receive the address and control information of the multimedia data to be processed sent by the application program through the OpenMAX interface of the middleware, and pass it to the driver adaptation layer; The driver is called through the public interface of the driver adaptation layer to trigger the driver to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to perform codec processing on the multimedia data to be processed.

2. The method according to claim 1, characterized in that The multimedia framework also includes an APP docking layer, which provides an API interface; The receiving, through the OpenMAX interface of the middleware, the address and control information of the to-be-processed multimedia data sent by the application program comprises: Receiving the address and the control information through the API interface of the APP docking layer; The APP docking layer converts the address and control information into a format adapted to the OpenMAX interface, and transmits the converted address and control information to the middleware through the OpenMAX interface.

3. The method according to claim 2, characterized in that The API interface includes at least one of the following: Initialization interface, used to create a multimedia session instance; A session configuration interface, used to configure session parameters of the multimedia session instance; A data input interface, used to input the address and the control information into the middleware; A result acquisition interface, used to obtain a processing result obtained after the codec performs encoding and decoding processing on the multimedia data to be processed; Resource release interface, used to release the resources occupied by the processing result; The session destruction interface is used to release the multimedia session instance.

4. The method according to claim 3, characterized in that The calling sequence of the API interface is: the initialization interface, the session configuration interface, the data input interface, the result acquisition interface, the resource release interface and the session destruction interface.

5. The method according to claim 1, characterized in that The transmission to the driver adaptation layer includes: The address and the control information are converted into a format adapted to the public interface and passed to the driver adaptation layer.

6. The method according to any one of claims 1 to 5, characterized in that: The OpenMAX interface is an interface of the OpenMAX integration layer.

7. According to the method according to any one of claims 1 to 5, the public interface of the driver adaptation layer controls the driver program by calling an ioctl control function.

8. A multimedia data encoding and decoding device based on a multimedia framework, characterized in that: The multimedia framework includes a middleware and a driver adaptation layer, wherein the middleware provides an OpenMAX interface, and the driver adaptation layer provides a public interface that is independent of the operating system, and the public interface encapsulates driver interfaces in different operating systems; the device includes a middleware module and a driver adaptation layer module, wherein: The middleware module is configured to receive the address and control information of the to-be-processed multimedia data sent by the application program through the OpenMAX interface, and pass the information to the driver adaptation layer module; The driver adaptation layer module is configured to call the driver program through the public interface to trigger the driver program to obtain the multimedia data to be processed according to the address, and drive the codec corresponding to the control information to perform encoding and decoding processing on the multimedia data to be processed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

10. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images