Multimedia data processing method and device

Abstract

The invention provides a multimedia data processing method, a multimedia data processing device and a computer storage medium / computer program product. The method comprises the steps that a first processing process writes a shared memory address into a first processing chain table and configures a corresponding trigger event; the first processing process transmits the multimedia data to first processing hardware so as to process the multimedia data through the first processing hardware, and a processing result is written into a shared memory corresponding to the shared memory address; after the first processing hardware writes the processing result into the shared memory, the write-in information is transmitted to the first processing process to activate the trigger event; and after monitoring that the trigger event is activated through a second processing process, the second processing module writes the shared memory address into a second processing chain table, so that the shared memory address is obtained from the second processing chain table through second processing hardware, and a processing result in the shared memory is extracted according to the shared memory address for post-processing. Through the method, the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer data processing technology, and in particular to a multimedia data processing method, a multimedia data processing device, and a computer storage medium / computer program product. Background Technology

[0002] Multimedia infotainment systems, often referred to as IVI or In-Vehicle Infotainment Systems, are a core component of the modern car cabin. They have evolved from simple radio / CD players into highly integrated digital platforms designed to provide drivers and passengers with information, entertainment, connectivity, and convenience features.

[0003] However, in current Android in-vehicle infotainment systems, data is sent to the display module only after decoding is complete, which is a serial process. The processing time between decoding and sending the data to the display module is very long, resulting in a long time from when the frame is sent from the source to when it is displayed on the screen, leading to high display latency and a reduced user experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this application proposes a multimedia data processing method, a multimedia data processing device, and a computer storage medium / computer program product.

[0005] To address the aforementioned technical problems, this application proposes a multimedia data processing method, which includes: a first processing module transmitting multimedia data and a shared memory address to a first processing process through a first processing interface process;

[0006] The first processing process writes the shared memory address into the first processing linked list and configures the corresponding triggering event;

[0007] The first processing process passes the shared memory address to the second processing module;

[0008] The first processing process transmits the multimedia data to the first processing hardware so that the multimedia data can be processed by the first processing hardware, and writes the processing result into the shared memory corresponding to the shared memory address;

[0009] After the first processing hardware writes the processing result into the shared memory, it transmits the write information to the first processing process to activate the triggering event.

[0010] After the second processing module detects the activation of the trigger event through the second processing process, it writes the shared memory address into the second processing linked list, so as to obtain the shared memory address from the second processing linked list through the second processing hardware, and extract the processing result in the shared memory according to the shared memory address for post-processing.

[0011] The multimedia data processing method further includes:

[0012] The first processing module configures processing parameters based on the multimedia data and transmits the processing parameters to the second processing module through the handle of the second processing module.

[0013] The multimedia data refers to audio stream data or video stream data.

[0014] Wherein, the first processing module is a decoding module, and the second processing module is a display module;

[0015] Alternatively, the first processing module may be an encoding module, and the second processing module may be a storage module.

[0016] The first processing process is a decoding process;

[0017] The first processing process writes the shared memory address into the first processing linked list and configures the corresponding triggering event, including:

[0018] The first processing process writes the shared memory address into the first processing linked list;

[0019] When the decoding process confirms that the decoding order and display order are consistent according to the syntax of the multimedia data, it creates a handle to a single timeline, configures trigger events for all shared memory addresses on the single timeline, and writes them into the first processing linked list.

[0020] The first processing process is a decoding process;

[0021] The first processing process writes the shared memory address into the first processing linked list and configures the corresponding triggering event, including:

[0022] The first processing process writes the shared memory address into the first processing linked list;

[0023] When the decoding process determines that the decoding order and display order are inconsistent based on the syntax of the multimedia data, it assigns a timeline to each shared memory and configures the triggering event of the shared memory address on the timeline corresponding to each shared memory, and writes it into the first processing linked list.

[0024] Specifically, defining a time axis for each shared memory segment includes:

[0025] The decoding process queries whether an associated timeline exists based on the shared memory address;

[0026] If so, the decoding process determines the associated timeline as the timeline of the shared memory;

[0027] If not, the decoding process creates a new timeline as the timeline for the shared memory.

[0028] To address the aforementioned technical problems, this application also proposes a multimedia data processing apparatus, which includes:

[0029] The first processing module is used to transfer multimedia data and shared memory addresses to the first processing process through the first processing interface process.

[0030] The first processing process is used to write the shared memory address into the first processing linked list and configure the corresponding triggering event;

[0031] The first processing process is used to pass the shared memory address to the second processing module;

[0032] The first processing process is used to transmit the multimedia data to the first processing hardware so that the multimedia data can be processed by the first processing hardware and the processing result can be written into the shared memory corresponding to the shared memory address.

[0033] The first processing hardware is configured to write the processing result into the shared memory and then transmit the write information to the first processing process to activate the triggering event.

[0034] The second processing module is used to listen to the activation of the trigger event through the second processing process, write the shared memory address into the second processing linked list, obtain the shared memory address from the second processing linked list through the second processing hardware, and extract the processing result in the shared memory according to the shared memory address for post-processing.

[0035] To address the aforementioned technical problems, this application also proposes another multimedia data processing apparatus, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the multimedia data processing method described above.

[0036] To address the aforementioned technical problems, this application also proposes a computer storage medium / computer program product, wherein the computer storage medium is used to store a computer program, and the computer program product includes a computer program, which, when executed by a computer, is used to implement the multimedia data processing method described above.

[0037] Compared with the prior art, the beneficial effects of this application are: after the first processing hardware finishes processing the current multimedia data, it activates a trigger event to notify the second processing hardware to extract the processing results from the shared memory for post-processing; through the synchronization mechanism of the trigger event, the data transmission time between the first processing hardware and the second processing hardware can be effectively reduced, thereby improving data processing efficiency. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0039] in:

[0040] Figure 1 This is a schematic diagram of the framework of the in-vehicle multimedia data processing system provided in this application;

[0041] Figure 2 This is a flowchart illustrating an embodiment of the multimedia data processing method provided in this application;

[0042] Figure 3 yes Figure 2 The flowchart of an embodiment of the multimedia data processing method step S12 is shown.

[0043] Figure 4 yes Figure 2 A flowchart illustrating another embodiment of the multimedia data processing method step S12 shown.

[0044] Figure 5 This is a schematic diagram of the structure of an embodiment of the multimedia data processing apparatus provided in this application;

[0045] Figure 6 This is a schematic diagram of another embodiment of the multimedia data processing apparatus provided in this application;

[0046] Figure 7 This is a schematic diagram of the structure of an embodiment of the computer storage medium / computer program product provided in this application. Detailed Implementation

[0047] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0048] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0049] This application is based on Figure 1 The multimedia data processing system shown in the vehicle-mounted system provides a specific multimedia data processing method to solve the problems of current multimedia data processing methods.

[0050] It should be noted that the application scenarios of the multimedia data processing method of this application include, but are not limited to, audio encoding scenarios, audio decoding scenarios, video encoding scenarios, and video decoding scenarios. The following section uses the video decoding scenario as an example to compare and introduce the current multimedia data processing methods and the multimedia data processing method of this application.

[0051] The current decoding and display solutions for Android in-vehicle infotainment systems are as follows:

[0052] After receiving the bitstream data, the application (APP) begins configuring the decoding module. The configuration parameters include the video width, video height, H.264 sequence parameter set, H.264 image parameter set, and the display module handle. The decoding module then sets the width and height information, color format, default color gamut, and default color matrix to the display module via the display module handle.

[0053] The decoding module allocates shared memory from the display module via a handle. This shared memory is used by the decoder to store the decoded YUV data, and the display module displays the YUV data from this shared memory onto the vehicle's screen. There are multiple blocks of shared memory, each with its own handle, and each block can only store the YUV data of one image. The decoding module obtains the handle to each shared memory block, configures it for the decoding hardware, and performs memory mapping. After mapping, the shared memory is returned to the display module.

[0054] After the decoding module is configured, it begins sending the bitstream data to the decoding interface process. The decoding interface process first obtains a handle to a shared memory block from the display module, then passes this handle to the decoding process, which in turn sends it to the decoding hardware. The bitstream data is then transmitted to the decoding process via inter-process communication (IPC). The decoding process sends the data to the hardware for decoding. Once decoding is complete, the decoding hardware stores the decoded YUV data in the transmitted shared memory. The decoding hardware notifies the decoding process that the current input data decoding is complete. At this point, the decoding module uses IPC to notify the decoding interface process that the input data has been decoded. Upon receiving the data and completing its decoding, the app sends the next frame of data for decoding.

[0055] When the hardware detects a data frame that can be displayed, it passes the shared memory handle corresponding to the data frame to the decoding process. The decoding process continuously passes this handle to the application, and the application decides when to display the frame. Typically, the application immediately passes the shared memory handle corresponding to the data frame to the display module via inter-process communication mechanisms after receiving the data frame.

[0056] Currently, the platform's hardware takes 6ms to decode a frame of H.264 data with a resolution of 1920x1080 and 10ms to decode a frame of data with a resolution of 3840x2160. Therefore, when a frame of bitstream data is sent to the decoding hardware, it takes at least 6ms (1920x1080) or 10ms (3840x2160) before the decoded data can be sent to the decoding process, the decoding interface process, the APP, and the display module.

[0057] Once the decoding hardware parses the color gamut and color matrix information, it notifies the decoding process. The decoding process then compares this information with historical color gamut and color matrix data. If any changes are found, the message is passed to the decoding interface process. The decoding interface process then needs to release all shared memory, reset the color gamut and color matrix information for the display module, allocate shared memory from the display module, and pass the shared memory handle to the decoding hardware.

[0058] Because Android's display module uses a double-buffering strategy, data frames sent to the display module are not immediately displayed on the screen. Instead, a certain amount of time is required. Our platform's screen refresh rate is 60Hz, and the screen refresh time for one frame is 16.6ms. After receiving a data frame, the display module waits (0ms to 16.6ms) before sending the data to the display hardware. The exact waiting time depends on the timing of the data frame's arrival. This is because the display module sends received data frames to the display hardware at a fixed rhythm, every 16.6ms. The display hardware needs 16.6ms to complete hardware configuration, color conversion, etc., before it can display the data on the screen.

[0059] In summary, current multimedia data processing methods suffer from the following problems, resulting in low data processing efficiency:

[0060] 1. In different application scenarios, the color gamut of the bitstream data received by the vehicle's infotainment system is different. Currently, the Android vehicle infotainment system uses a narrow color gamut (Limit Range) for display by default. If the color gamut of the data is a wide color gamut (Full Range), the display module needs to be reconfigured, which takes some time.

[0061] 2. Currently, in Android vehicle infotainment systems, data is sent to the display module only after decoding is complete. This is a serial process, and the processing between decoding and delivery to the display module is very long, resulting in a long time from when the frame is sent from the source to when it is displayed on the screen.

[0062] To address this, this application provides a specific multimedia data processing method that effectively solves the aforementioned problems. Figure 1 Taking the video decoding scenario shown as an example, please combine it with... Figure 1 Continue reading Figure 2 , Figure 2 This is a flowchart illustrating an embodiment of the multimedia data processing method provided in this application.

[0063] In the video decoding scenario, the first processing module is the decoding module, the second processing module is the display module, the first processing interface process is the decoding interface process, the first processing process is the decoding process, the second processing process is the display process, the first processing hardware is the decoding hardware, and the second processing hardware is the display hardware. The decoding module and the display module are functional modules within the vehicle-mounted app.

[0064] like Figure 2 As shown, the specific steps are as follows:

[0065] Step S11: The first processing module transmits the multimedia data and the shared memory address to the first processing process through the first processing interface process.

[0066] In the embodiments of this application, in Figure 1 During the initialization phase of the multimedia data processing system, the vehicle-mounted app receives the video stream data (multimedia data) and begins configuring the decoding module. The configuration parameters include the video width, video height, decoder parameter set, and the display module handle. The decoding module sets the width, height, and color format information to the display module via the display module handle. It parses the decoder parameter set (H264 / H265 video sequence parameter set) to obtain the color gamut and color matrix information in the video stream, and checks if the decoding and display orders are consistent. The color gamut and color matrix information are then set to the decoding process via inter-process communication, and the decoding process saves these information. Finally, the color gamut and color matrix information are configured to the display module via the display module handle.

[0067] The decoding module allocates shared memory from the display module via a handle. This shared memory is used by the decoder to store the decoded YUV data, and the display module displays the YUV data from the shared memory on the vehicle's screen. There are multiple blocks of shared memory, each with its own handle, and each block can only store the YUV data of one image. The decoding module obtains the handle of each shared memory block, configures it for the decoding hardware, and performs memory mapping. After mapping, the shared memory is returned to the display module.

[0068] This application solves the problem of the display module needing to be reconfigured by parsing the color gamut information of the bitstream data before initializing the display module on the vehicle-mounted system, and then configuring the color gamut information to the display module during initialization.

[0069] Step S12: The first processing process writes the shared memory address into the first processing linked list and configures the corresponding triggering event.

[0070] In this embodiment, the decoding process writes the shared memory address into a linked list and configures a corresponding trigger event for that shared memory address. This trigger event indicates whether bitstream data has been written into the shared memory corresponding to the shared memory address. When the decoding hardware writes bitstream data into the shared memory, the trigger event is activated so that the display module can obtain this information in a timely manner.

[0071] Regarding the configuration logic for triggering events, this application provides a single timeline with multiple fences for situations where the video decoding order and display order are consistent. Please refer to [link / reference] for details. Figure 3 , Figure 3 yes Figure 2 The flowchart shown is a schematic diagram of an embodiment of the multimedia data processing method step S12.

[0072] like Figure 3As shown, the specific steps are as follows:

[0073] Step S21: The first processing process writes the shared memory address into the first processing linked list.

[0074] In this embodiment, the decoding module obtains a handle to a shared memory block from the display module and then passes the handle to the decoding process.

[0075] Step S22: When the decoding process confirms that the decoding order and display order are consistent according to the syntax of the multimedia data, it creates a handle to a single timeline, configures trigger events for all shared memory addresses on the single timeline, and writes them into the first processing list.

[0076] In this embodiment, when the decoding process determines that the decoding order and display order of the bitstream are consistent based on the syntax of the bitstream data, it uses the handle of the timeline created when the decoder receives the configuration that the decoding order and display order are consistent, creates a fence on that timeline, and saves it to the structure of this shared memory. Then, the frame rate information of the video is saved to the structure of the shared memory.

[0077] This application saves the created fence to a linked list of fence and shared memory IDs maintained by the decoding process. The decoding process first passes the shared memory to the decoding hardware, then passes the structure information containing the shared memory information to the decoding interface process, and finally passes it to the display module.

[0078] After receiving the shared memory, the display module creates a new thread to wait for the corresponding fence. If the fence is not activated, the thread will wait indefinitely until the decoding process activates it. Once the fence is activated, the display module adds the shared memory to the display linked list, closes the fence, and the thread exits.

[0079] Then, the display module calculates a time interval t based on the frame rate, and retrieves a frame of data from the display list every t time interval and sends it to the display hardware for display.

[0080] Since the color gamut information and color matrix information of the bitstream decoding are parsed and configured in advance, these information will not change after the first frame is decoded, and the first frame data can be decoded directly.

[0081] After decoding is complete, the decoding hardware stores the decoded YUV data in the shared memory. The decoding hardware then notifies the decoding process that the current bitstream decoding is complete. The decoding process finds the corresponding fence from the linked list containing the fence and the shared memory ID, activates the fence, and thus completes the task of transmitting information to the display module.

[0082] Regarding the configuration logic for triggering events, this application provides a multi-timeline, multi-fence synchronization scheme for situations where the video decoding order and display order are inconsistent. Please refer to [link / reference needed] for details. Figure 4 , Figure 4 yes Figure 2 The flowchart of another embodiment of the multimedia data processing method step S12 is shown.

[0083] like Figure 4 As shown, the specific steps are as follows:

[0084] Step S31: The first processing process writes the shared memory address into the first processing linked list.

[0085] In this embodiment, the decoding module obtains a handle to a shared memory block from the display module and then passes the handle to the decoding process.

[0086] Step S32: When the decoding process confirms that the decoding order and display order are inconsistent according to the syntax of the multimedia data, a timeline is determined for each shared memory, and the triggering event of the shared memory address is configured on the timeline corresponding to each shared memory and written into the first processing linked list.

[0087] In this embodiment of the application, when the decoding process receives shared memory, it uses the ID contained in the shared memory to uniquely identify the shared memory and queries a linked list of shared memory IDs and timeline handle mappings maintained by the decoding process to see if there is an associated timeline handle for the current shared memory ID.

[0088] If no timeline handle exists, a timeline handle is created, and the shared memory ID and timeline handle are saved to a linked list. The timeline handle is then returned. If a relationship already exists in the linked list, the corresponding timeline handle is found based on the shared memory ID.

[0089] The decoding process uses the timeline handle to create a fence on that timeline and saves it to a structure in this shared memory. Then, the video frame rate information is saved to the shared memory structure.

[0090] This application saves the created fence to a linked list of fence and shared memory IDs maintained by the decoding process. The decoding process first passes the shared memory to the decoding hardware, then passes the structure information containing the shared memory information to the decoding interface process, and finally passes it to the display module.

[0091] After receiving the shared memory, the display module creates a new thread that waits for the corresponding fence. If the fence is not activated, the thread will wait indefinitely until the decoding process activates it. Once the fence is activated, the shared memory is added to the display linked list, the fence is closed, and the thread exits.

[0092] Then, the display module calculates a time interval t based on the frame rate, and retrieves a frame of data from the display list every t time interval and sends it to the display hardware for display.

[0093] Since the color gamut information and color matrix information of the bitstream decoding are parsed and configured in advance, these information will not change after the first frame is decoded, and the first frame data can be decoded directly.

[0094] After decoding is complete, the decoding hardware stores the decoded YUV data in the shared memory. The decoding hardware then notifies the decoding process that the current bitstream decoding is complete. The decoding process finds the corresponding fence from the linked list containing the fence and the shared memory ID, activates the fence, and thus completes the task of transmitting information to the display module.

[0095] Step S13: The first processing process passes the shared memory address to the second processing module.

[0096] Step S14: The first processing process transmits multimedia data to the first processing hardware so that the multimedia data can be processed by the first processing hardware, and the processing result is written to the shared memory corresponding to the shared memory address.

[0097] In this embodiment, the decoding hardware decodes the decoded bitstream and writes the decoded image data into shared memory.

[0098] Step S15: After the first processing hardware writes the processing result to the shared memory, it transmits the write information to the first processing process to activate the trigger event.

[0099] In this embodiment, after the decoding hardware writes the image data into the shared memory, it passes the write information to the decoding process, which then activates the fence configured in step S12.

[0100] Step S16: After the second processing module detects the activation of the trigger event through the second processing process, it writes the shared memory address into the second processing linked list, so as to obtain the shared memory address from the second processing linked list through the second processing hardware, and extract the processing result in the shared memory according to the shared memory address for post-processing.

[0101] In this embodiment, when the display module detects fence activation through the display process, it writes the shared memory address corresponding to the fence into the display linked list, and periodically extracts the image data of the shared memory corresponding to the shared memory address that already exists in the display linked list and sends it to the display hardware for display.

[0102] In one specific implementation, it is assumed that the frame type of the first 5 frames of the bitstream is IPBBB.

[0103] Frame number 1 2 3 4 5 Frame type I P B B B

[0104] The order in which the decoding process sends frames to the decoding hardware corresponds to frame numbers 1, 2, 3, 4, and 5.

[0105] The correct display order is IBBBP, and the corresponding frame numbers are 1, 3, 4, 5, 2.

[0106] Frame number 1 3 4 5 2 Frame type I B B B P

[0107] The order in which the APP sends the frames to the decoding hardware corresponds to frame numbers 1, 2, 3, 4, and 5. Simultaneously, the order in which the decoding process sends the frames to the display process also corresponds to frame numbers 1, 2, 3, 4, and 5. Each frame creates a fence, designated fence1, fence2, fence3, fence4, and fence5. The display process waits in five threads for fence1, fence2, fence3, fence4, and fence5 to become active.

[0108] After the decoding hardware finishes decoding the first frame, since the frame type is a keyframe, it will immediately push the frame forward. The decoding process will receive a message from the decoding hardware that a frame is ready to be displayed, at which point fence1 will be activated. The thread waiting for fence1 in the display process will be awakened, and the frame with frame number 1 will be placed in the display queue.

[0109] After the decoding hardware finishes decoding the second frame, since the frame type is a forward reference value, subsequent frames will refer to it. After all hardware decoding is completed, there will be no displayable frame messages.

[0110] After the decoding hardware finishes decoding the third frame, according to the decoding frame push logic, it can be pushed out for display. Therefore, the decoding process will receive a message from the decoding hardware that a frame is ready to be displayed. At this time, by querying the linked list, fence3 will be activated. The thread waiting for fence3 in the display process will be awakened, and the frame with frame number 3 will be placed in the display queue.

[0111] After the decoding hardware finishes decoding the fourth frame, according to the decoding frame push logic, it can be pushed out for display. Therefore, the decoding process will receive a message from the decoding hardware that a frame is ready to be displayed. At this time, by querying the linked list, fence4 will be activated. The thread waiting for fence4 in the display process will be awakened, and the frame with frame number 4 will be placed in the display queue.

[0112] After the decoding hardware finishes decoding the fourth frame, according to the decoding frame push logic, it can be pushed out for display. Therefore, the decoding process will receive a message from the decoding hardware that a frame is ready to be displayed. At this time, by querying the linked list, fence5 will be activated. The thread waiting for fence5 in the display process will be awakened, and the frame with frame number 5 will be placed in the display queue.

[0113] According to the decoder's frame-pushing logic, frame number 2 is no longer referenced by other frames and can be pushed out for display. Therefore, the decoding process receives a message from the decoding hardware that a frame is ready to be displayed. At this point, a lookup in the linked list activates fence2. The thread waiting for fence2 in the display process is awakened, and frame number 2 is placed into the display queue.

[0114] In this way, the frame numbers of the display module in the queue are 1, 3, 4, 5, 2, which is completely consistent with the display order.

[0115] It should be noted that the activation event mechanism used in this application is not limited to fence; the socket mechanism can also be used for synchronization.

[0116] Specifically, the process obtains a handle to a shared memory block from the display module and passes it to the decoding process. Upon receiving the shared memory, the decoding process uses its unique ID to check a linked list maintaining a mapping between shared memory IDs and socket handles. If no matching socket handle exists for the current shared memory ID, a directory named "dev / unix / codec / shared memory ID" is created. A socket handle is then created using this directory, and the shared memory ID, socket handle, and socket directory are saved to the linked list. The socket handle and directory are then returned. If a matching relationship already exists in the linked list, the corresponding socket handle and directory are located based on the shared memory ID. The socket directory is saved to the shared memory's structure. Finally, the video frame rate information is saved to the shared memory's structure.

[0117] The socket handle is saved to a linked list of socket handles and shared memory IDs maintained by the decoding process. The decoding process first passes the shared memory to the decoding hardware, then passes the structure information containing the shared memory information to the decoding interface process, and finally passes it to the display module.

[0118] After receiving the shared memory, the display module first retrieves the socket handle corresponding to the socket directory from the environment variables based on the socket's directory name. If not found, it creates the socket handle based on the socket directory and adds both the socket directory and the socket handle to the environment variables. It then adds the socket handle to epoll / select to begin listening. When an event occurs on the monitored socket handle, it reads the message. If the message is "decode done," the shared memory is added to the display list. If the message is "decode flush," the shared memory is added to the free list and no longer sent for display; it can later be used by the decoding process to refill data.

[0119] The display module calculates a time interval t based on the frame rate, and retrieves a frame of data from the display list every t time interval and sends it to the display hardware for display.

[0120] Since the color gamut information and color matrix information of the bitstream decoding are parsed and configured in advance, these information will not change after the first frame is decoded, and the first frame data can be decoded directly.

[0121] After decoding is complete, the decoding hardware stores the decoded YUV data in the shared memory that was passed down. The decoding hardware then notifies the decoding process that the current bitstream decoding is complete. The decoding process finds the corresponding socket from the linked list containing socket handles and shared memory IDs, and uses that socket to send a "decode done" message.

[0122] Since the cross-process transmission time of the decoded data plus the processing time of the display module is greater than the hardware decoding time, the decoding time, the data transmission time to the display module, and the processing time of the display module can be parallelized, effectively speeding up the display of the first frame.

[0123] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0124] To implement the above-mentioned multimedia data processing method, this application also proposes a multimedia data processing apparatus 500, for details please refer to [link / reference needed]. Figure 5 , Figure 5 This is a schematic diagram of an embodiment of the multimedia data processing apparatus provided in this application.

[0125] The multimedia data processing device 500 of this embodiment includes: a first processing module 51, a first processing process 52, a first processing hardware 53, a second processing module 54, a second processing process 55, and a second processing hardware 56.

[0126] The first processing module 51 is used to pass multimedia data and shared memory addresses to the first processing process 52 through the first processing interface process.

[0127] The first processing process 52 is used to write the shared memory address into the first processing linked list and configure the corresponding triggering event.

[0128] The first processing process 52 is used to pass the shared memory address to the second processing module 54.

[0129] The first processing process 52 is used to transmit the multimedia data to the first processing hardware 53, so that the multimedia data can be processed by the first processing hardware 53, and the processing result can be written into the shared memory corresponding to the shared memory address.

[0130] The first processing hardware 53 is used to write the processing result into the shared memory and then transmit the write information to the first processing process 52 to activate the triggering event.

[0131] The second processing module 54 is used to write the shared memory address into the second processing linked list after the trigger event is activated by the second processing process 55, so as to obtain the shared memory address from the second processing linked list through the second processing hardware 56, and extract the processing result in the shared memory according to the shared memory address for post-processing.

[0132] In some specific implementations, the first processing module configures processing parameters based on the multimedia data and transmits the processing parameters to the second processing module through the handle of the second processing module.

[0133] In some specific implementations, the multimedia data is audio stream data or video stream data.

[0134] In some specific embodiments, the first processing module is a decoding module and the second processing module is a display module; or the first processing module is an encoding module and the second processing module is a storage module.

[0135] In some specific implementations, the first processing process is a decoding process; the first processing process writes the shared memory address into a first processing linked list and configures the corresponding triggering event, including: the first processing process writes the shared memory address into the first processing linked list; when the decoding process confirms that the decoding order and display order are consistent according to the syntax of the multimedia data, it creates a handle to a single timeline, configures the triggering events of all shared memory addresses on the single timeline, and writes them into the first processing linked list.

[0136] In some specific implementations, the first processing process is a decoding process; the first processing process writes the shared memory address into a first processing linked list and configures the corresponding trigger event, including: the first processing process writes the shared memory address into the first processing linked list; when the decoding process determines that the decoding order and display order are inconsistent according to the syntax of the multimedia data, it determines a timeline for each shared memory and configures the trigger event of the shared memory address on the timeline corresponding to each shared memory, and writes it into the first processing linked list.

[0137] In some specific implementations, determining a timeline for each shared memory includes: the decoding process querying whether an associated timeline exists based on the shared memory address; if so, the decoding process determines the associated timeline as the timeline of the shared memory; if not, the decoding process creates a new timeline as the timeline of the shared memory.

[0138] To implement the above-described multimedia data processing method, this application also proposes another multimedia data processing apparatus, for details please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of another embodiment of the multimedia data processing apparatus provided in this application.

[0139] The multimedia data processing device 400 of this embodiment includes a processor 41, a memory 42, an input / output device 43, and a bus 44.

[0140] The processor 41, memory 42, and input / output device 43 are respectively connected to the bus 44. The memory 42 stores program data, and the processor 41 is used to execute the program data to implement the multimedia data processing method described in the above embodiments.

[0141] In this embodiment, processor 41 can also be referred to as a CPU (Central Processing Unit). Processor 41 may be an integrated circuit chip with signal processing capabilities. Processor 41 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor, or processor 41 can be any conventional processor.

[0142] This application also provides a computer storage medium / computer program product; please refer to the following: Figure 7 , Figure 7 This is a schematic diagram of an embodiment of the computer storage medium / computer program product provided in this application. The computer storage medium 600 is used to store a computer program 61. The computer program product 600 includes the computer program 61. When the computer program 61 is executed by a computer, it is used to implement the multimedia data processing method of the above embodiment.

[0143] When the embodiments of this application are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimedia data processing method, characterized by, The multimedia data processing method comprises: The first processing module transmits the multimedia data and the shared memory address to the first processing process through a first processing interface process; The first processing process writes the shared memory address into a first processing chain table and configures a corresponding trigger event; The first processing process transmits the shared memory address to the second processing module; The first processing process transmits the multimedia data to the first processing hardware to process the multimedia data through the first processing hardware and writes the processing result into the shared memory corresponding to the shared memory address; After the first processing hardware writes the processing result into the shared memory, the first processing hardware transmits the writing information to the first processing process to activate the trigger event; After the second processing module listens to the activation of the trigger event through a second processing process, the second processing module writes the shared memory address into a second processing chain table to obtain the shared memory address from the second processing chain table through second processing hardware and post-processes the processing result in the shared memory according to the shared memory address.

2. The multimedia data processing method of claim 1, wherein the multimedia data processing method further comprises: The first processing module configures a processing parameter according to the multimedia data and transmits the processing parameter to the second processing module through a handle of the second processing module.

3. The multimedia data processing method of claim 1, wherein the multimedia data is audio stream data or video stream data.

4. The multimedia data processing method of claim 3, wherein the first processing module is a decoding module and the second processing module is a display module; or the first processing module is an encoding module and the second processing module is a storage module.

5. The multimedia data processing method of claim 1, wherein the first processing process is a decoding process; and the first processing process writes the shared memory address into the first processing chain table and configures a corresponding trigger event, comprising: The first processing process writes the shared memory address into the first processing chain table; and The decoding process creates a single time axis handle when the decoding order and the display order of the syntax of the multimedia data are consistent and configures the trigger event of all shared memory addresses on the single time axis and writes the trigger event into the first processing chain table.

6. The multimedia data processing method of claim 1, wherein the first processing process is a decoding process; and the first processing process writes the shared memory address into the first processing chain table and configures a corresponding trigger event, comprising: The first processing process writes the shared memory address into the first processing chain table; and The decoding process determines a time axis for each shared memory when the decoding order and the display order of the syntax of the multimedia data are inconsistent and configures the trigger event of the shared memory address on the time axis corresponding to each shared memory and writes the trigger event into the first processing chain table. ​ ​ ​ ​ ​ ​ ​ ​ 7. The multimedia data processing method of claim 6, wherein determining a timeline for each shared memory comprises: inquiring, by the decoding process, whether there is an associated timeline according to the shared memory address; if yes, determining, by the decoding process, the associated timeline as the timeline of the shared memory; if no, creating, by the decoding process, a new timeline as the timeline of the shared memory. The multimedia data processing apparatus comprises:

8. A multimedia data processing apparatus, characterized by comprising: a first processing module configured to pass multimedia data and a shared memory address to a first processing process via a first processing interface process; the first processing process configured to write the shared memory address into a first processing linked list and configure a corresponding trigger event; the first processing process configured to pass the shared memory address to a second processing module; the first processing process configured to pass the multimedia data to a first processing hardware to process the multimedia data via the first processing hardware and write the processing result into a shared memory corresponding to the shared memory address; the first processing hardware configured to pass write information to the first processing process after writing the processing result into the shared memory to activate the trigger event; the second processing module configured to write the shared memory address into a second processing linked list after the second processing process listens to the activation of the trigger event to acquire the shared memory address from the second processing linked list via a second processing hardware and post-process the processing result in the shared memory according to the shared memory address. The multimedia data processing apparatus comprises a memory and a processor coupled with the memory; 9. A multimedia data processing apparatus, characterized by comprising: wherein the memory is configured to store program data, and the processor is configured to execute the program data to implement the multimedia data processing method of any one of claims 1 to 7. The computer storage medium is configured to store a computer program, and the computer program product comprises a computer program, which, when executed by a computer, is configured to implement the multimedia data processing method of any one of claims 1 to 7.

10. A computer storage medium / computer program product, characterized in that, ​

Images