Media data transmission method, media chip and storage medium

Abstract

The application discloses a media data transmission method, a media chip and a storage medium. The method comprises the following steps: a first publisher module writes first media data into a first buffer, wherein the first publisher module is any one of a plurality of publisher modules; the first publisher module sends a message publishing request to a central coordinator module, wherein the message publishing request comprises a first topic and a file descriptor of the first buffer; the central coordinator module creates a first file descriptor copy corresponding to the file descriptor for a first subscriber module, wherein the first subscriber module is a subscriber module subscribing to the first topic in a plurality of subscriber modules; the central coordinator module pushes the first file descriptor copy to the first subscriber module; and the first subscriber module reads the first media data from the first buffer through the first file descriptor copy. The application is beneficial to reducing the CPU occupancy rate of the media chip.

Description

Technical Field

[0001] This application relates to the field of data transmission technology, specifically to a media data transmission method, a media chip, and a storage medium. Background Technology

[0002] In embedded multimedia systems, multiple media processing pipelines of a media chip (e.g., a system-on-chip (SoC)) are also known as multiple media modules, such as acquisition modules, encoding modules, Real-time Transport Protocol (RTP) streaming modules, MJPEG preview modules, MP4 recording modules, etc. These modules share the same media data.

[0003] Multiple media modules of a media chip can be understood as user mode. Due to the access restrictions of kernel mode, when a media module needs to use this media data to transmit the media data through a socket or pipe, the central processing unit (CPU) of the media chip needs to perform a double copy from user mode to kernel mode and then back to user mode. Therefore, for a single piece of media data, the CPU needs to perform multiple memory copies, resulting in high CPU utilization and high system load when using the media chip to access media data. Summary of the Invention

[0004] This application provides a media data transmission method, a media chip, and a storage medium, which reduces the CPU utilization of the media chip through a publish-subscribe model.

[0005] In a first aspect, embodiments of this application provide a media data transmission method, the method being applied to a media chip, the media chip including multiple publisher modules, multiple subscriber modules, and a central coordinator module; The first publisher module writes the first media data into the first buffer, wherein the first publisher module is any one of the plurality of publisher modules; The first publisher module sends a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor for the first buffer; The central coordinator module creates a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules; The central coordinator module pushes a copy of the first file descriptor to the first subscriber module. The first subscriber module reads the first media data from the first buffer through the first file descriptor copy.

[0006] Secondly, embodiments of this application provide a media chip, including: multiple publisher modules, multiple subscriber modules, and a central coordinator module; The first publisher module is used to write the first media data into the first buffer, wherein the first publisher module is any one of the plurality of publisher modules; The first publisher module is configured to send a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor for the first buffer; The central coordinator module is used to create a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules; The central coordinator module is used to push the copy of the first file descriptor to the first subscriber module; The first subscriber module is configured to read the first media data from the first buffer via the first file descriptor copy.

[0007] Thirdly, embodiments of this application provide a media chip, including: a processor and a memory, the processor being connected to the memory, the memory being used to store a computer program, and the processor being used to execute the computer program stored in the memory, so that the media chip performs the method as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the method described in the first aspect.

[0010] Implementing the embodiments of this application has the following beneficial effects: As can be seen in this embodiment, when modules (i.e. threads) in the media chip need to share media data, the modules autonomously write the media data into a buffer, and then other modules read the media data from the buffer for use. This eliminates the need for the media chip's CPU to copy the media data, thus reducing CPU usage. Furthermore, in implementing media data transmission, this application does not transmit the media data itself to other modules, but rather the file descriptor of the buffer. Since the file descriptor's memory (typically 4 bytes) is very small, the amount of data transmitted is reduced. Compared to directly transmitting media data, modules in the media chip can quickly obtain the file descriptor and use it to quickly retrieve the media data from the buffer, improving the efficiency of media data transmission between modules or threads in the media chip. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A schematic diagram illustrating media data transmission as provided in an embodiment of this application; Figure 2 A schematic diagram of a media chip provided in an embodiment of this application; Figure 3 A schematic flowchart illustrating a media data transmission method provided in an embodiment of this application; Figure 4 A schematic diagram illustrating the processing of a file descriptor copy provided in this application embodiment; Figure 5 A schematic diagram of a streaming method provided in an embodiment of this application; Figure 6 A schematic diagram of a transmission media data provided in an embodiment of this application; Figure 7 A schematic diagram of another media chip provided in an embodiment of this application; Figure 8 This is a schematic diagram of another media chip provided in an embodiment of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0014] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0015] In this document, the term "embodiment" means that a particular feature, result, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0016] See Figure 1 , Figure 1 This is a schematic diagram illustrating a media data transmission method provided in an embodiment of this application.

[0017] For example, such as Figure 1 Media chips have two operating permissions: kernel mode and user mode. Modules with user mode operating permissions include, but are not limited to: acquisition module, encoding module, preview module, etc.

[0018] First, the media chip acquires raw media data, for example, through a High Definition Multimedia Interface (HDMI). Then, the raw media data is cached in a kernel-mode buffer. Next, the media chip's CPU copies (i.e., copies) the raw media data from the kernel-mode buffer to the acquisition module's buffer, completing the raw media data acquisition process.

[0019] Furthermore, since multiple modules in user space share this media data, when other modules in user space (i.e., modules other than the acquisition module) need to use this media data, it is necessary to transfer this media data across modules in user space. Because different modules in user space do not have direct communication permissions, the CPU first copies the original media data from the acquisition module's buffer to the kernel-space buffer (i.e., user space to kernel space), and then copies the media data from the kernel-space buffer to the corresponding user-space buffer (i.e., kernel space to user space). For example, ... Figure 1 As shown, when the encoding module needs to use raw media data, the CPU copies the raw media data from the acquisition module's buffer to the kernel-mode buffer, and then copies the media data from the kernel-mode buffer to the encoding module's buffer. Similarly, when the preview module needs to use raw media data, the CPU copies the raw media data from the acquisition module's buffer to the kernel-mode buffer, and then copies the media data from the kernel-mode buffer to the preview module's buffer.

[0020] The buffers corresponding to the kernel mode and the buffers corresponding to each module in the user mode are allocated from the memory of the media chip.

[0021] It can be seen that when media data needs to be transferred between different modules of the media chip, the media data will undergo a double copying process: first from user mode to kernel mode, and then from kernel mode to user mode. This results in a large amount of CPU resources being occupied just by copying the media data, leading to high CPU utilization and high system load.

[0022] See Figure 2 , Figure 2 This is a schematic diagram of a media chip provided in an embodiment of this application. Figure 2 As shown, the media chip includes multiple publisher modules, multiple subscriber modules, and a central coordinator module. The central coordinator module in this application functions similarly to a Broker in a message queue; therefore, the central coordinator module in this application can also be referred to as a Broker.

[0023] Optionally, such as Figure 2 As shown, the aforementioned publisher modules include publisher module A, publisher module B, ..., etc., and the aforementioned subscriber modules include subscriber module A, subscriber module B, ..., etc.

[0024] The modules mentioned in this application are software modules abstracted from the media chip, that is, software code with corresponding processing functions is abstracted into corresponding modules, not specific hardware modules in the media chip. Therefore, the modules involved in this application can also be understood as threads. Thus, the publisher module involved in this application can also be understood as a publisher thread, and the subscriber module can also be understood as a subscriber thread. In addition, the publisher module and publisher thread involved in this application can also be simply referred to as publisher; the subscriber thread and subscriber module can also be simply referred to as subscriber. These descriptions are essentially consistent and do not need to be distinguished.

[0025] Optionally, the media data types in this application include raw media data and encoded media data. Each type of media data includes one or a combination of images, video, and audio. For ease of understanding, this application primarily uses images as an example of media data. There are various formats for caching images, such as NV12, NV21, MJPEG, etc. For ease of understanding, this application primarily uses the NV12 format as the format for caching and encoding images, but does not limit the image format. Since video is also transmitted frame by frame, the processing method for video is similar to that for images and will not be described again.

[0026] Furthermore, Broker pre-creates topics for publishing various types of media data. For example, it creates a topic hdmi-in-0_nv12 for the original image, a topic hdmi-in-0_pcm for the compressed audio, and a topic hdmi-in-0_h264 for the encoded image.

[0027] For example, adopting Figure 2 The media chip shown, when there is media data to be transmitted (or shared) in the aforementioned publisher module, for example, when the first publisher module (i.e. Figure 2 When the publisher module A shown has first media data to transmit, the first publisher module will first write the first media data into the first buffer (e.g., ...). Figure 2 Buffer 2 shown), wherein the first buffer is generated by the central coordinator module Broker from multiple pre-requested buffers (such as... Figure 2 The buffers shown (buffer 1, buffer 2, buffer 3, ...) are allocated to the first publisher module. Then, the first publisher module sends a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor fd of the first buffer, and the message publishing request is used to request that the file descriptor be published to the first topic.

[0028] Then, the Broker identifies the subscriber modules among the aforementioned subscriber modules that have subscribed to the first topic. For example, the first subscriber module (such as...) is identified... Figure 2 If subscriber module A) subscribes to the first topic, the Broker creates a copy of the first file descriptor (fd) corresponding to the aforementioned file descriptor (fd) for the first subscriber module and pushes the first fd copy to the first subscriber module. Finally, the first subscriber module reads the aforementioned first media data from the first buffer using the first fd copy. For example, as shown... Figure 2 As shown, the first subscriber module reads the aforementioned first media data from buffer 2 through the first fd copy.

[0029] As can be seen, when modules in the media chip of this application need to share media data, the modules autonomously write the media data into a buffer, and then other modules read the media data from the buffer for use. This eliminates the need for the media chip's CPU to copy the media data, thus reducing CPU usage. Furthermore, when implementing media data transmission, this application does not directly transmit the media data itself to other modules, but rather the file descriptor of the buffer. Since the memory of the file descriptor (typically 4 bytes) is very small, the amount of data transmitted is reduced. Compared to directly transmitting media data, modules in the media chip can quickly obtain the file descriptor and use it to quickly retrieve the media data from the buffer, improving the efficiency of media data transmission between modules or threads within the media chip.

[0030] See Figure 3 , Figure 3 This is a flowchart illustrating a media data transmission method provided in an embodiment of this application. The method is applied to the aforementioned media chip. Optionally, the media chip involved in this application can be a SoC chip; however, the type of media chip is not limited. The method includes, but is not limited to, the following steps: S301: The first publisher module writes the first media data into the first buffer.

[0031] The first publisher module can be any one of the plurality of publisher modules. Optionally, the first publisher module can be any publisher module among the plurality of publisher modules that needs to share or transmit media data.

[0032] Optionally, the aforementioned first buffer is allocated to the first publisher module by the central coordinator module (Broker).

[0033] It should be noted that this application mainly uses the example of media chips being applicable to various frameworks, but primarily focuses on their application under the Linux framework. Accordingly, under the Linux framework, the buffer mentioned in this application can be called a Direct Memory Access (DMA) buffer. These two terms are essentially the same and do not need to be distinguished.

[0034] For example, the central coordinator module pre-allocates multiple buffers from the media chip's memory via a Direct Memory Access Heap (DMA Heap) interface. Optionally, the media chip's memory in this application can be understood as the media chip's Random Access Memory (RAM). Specifically, during the media chip's initialization phase, the central coordinator module pre-allocates multiple contiguous buffers from memory via the DMA Heap interface. More specifically, for a frame of image with NV12 format, a resolution of 1920×1080, and 64-byte buffer alignment, the required buffer size is 3,133,440 bytes (1920×1088×1.5), approximately 3.1M. Therefore, when allocating buffers, each buffer needs to be at least 3,133,440 bytes in size.

[0035] Furthermore, after requesting the aforementioned multiple buffers, the central coordinator module manages and allocates these buffers uniformly, and configures them as dedicated buffers for the publisher module in the media chip to write media data. Therefore, the central coordinator module can obtain the current state of each buffer (i.e., whether it is currently idle or busy, i.e., occupied), and allocate the corresponding buffer to the publisher module based on the current state of each buffer.

[0036] For example, before the first publisher module writes the first media data into the first buffer, the first publisher module first sends a buffer allocation request to the central coordinator module. This buffer allocation request requests the central coordinator module to allocate a buffer for the first publisher module. Accordingly, in response to the buffer allocation request, the central coordinator module allocates a first buffer to the first publisher module from multiple buffers. The first buffer is a free buffer among the multiple buffers; that is, a buffer is selected from the multiple free buffers and allocated to the first publisher module. How to select a buffer from the multiple free buffers is not the focus of this application.

[0037] Understandably, after requesting multiple buffers, the central coordinator module can obtain the file descriptor (fd) for each buffer. Therefore, when allocating the first buffer to the first publisher module, the central coordinator module can also send the file descriptor of the first buffer to the first publisher module.

[0038] Optionally, the aforementioned multiple publisher modules include, but are not limited to, the acquisition module and encoding module of the media chip. The multiple subscriber modules include, but are not limited to, an encoding module, a streaming module, a preview module, and a recording module. Optionally, the inference module can be a Real-time Transport Control Protocol (RTP) streaming module; the preview module can be an MJPEG preview module; and the recording module can be an MP4 recording module.

[0039] For example, in response to the first publisher module being a collection module, since the data collected by the collection module is raw media data, the data shared by the collection module with other modules (i.e., data published to other modules) is also raw media data. Therefore, the aforementioned first media data is raw media data. Optionally, the aforementioned raw media data is media data collected by the collection module via HDMI (i.e., media data collected from the outside via HDMI) or media data received via a network (e.g., via a network card). It should be noted that the reason for receiving media data from the network is that for remote conferencing, different nodes may be distributed in multiple areas. The chip in this application is a chip in a certain node. In the case of node collaboration during the remote conferencing, this node may receive media data transmitted by other nodes, thus making the aforementioned raw media data media data received via the network. Accordingly, the encoding module and the preview module need to use the raw media data, so the encoding module and the preview module are subscribers of the collection module. Therefore, when the first publisher module is a collection module, the first subscriber module is the encoding module and / or the preview module.

[0040] For example, in response to the first publisher module being the encoding module, the data shared by the encoding module with other modules (i.e., the data transmitted to other modules) is media data encoded from the original media data. Therefore, the aforementioned first media data is media data encoded by the encoding module from the original media data. Optionally, since the streaming module and the recording module need to use encoded media data, the streaming module and the recording module are subscribers of the encoding module. Therefore, when the first publisher module is the encoding module, the aforementioned first subscriber module is the streaming module and / or the recording module.

[0041] It's important to note that some modules in the media chip can function as both publisher and subscriber modules. This means they can both transmit media data to other modules and process media data received from other modules. For example, the encoding module mentioned above can act as a subscriber, subscribing to raw media data shared by the acquisition module. It reads the raw media data from the buffer published by the acquisition module and encodes it to obtain encoded media data. Then, the encoding module can act as a publisher, writing the encoded media data to a new buffer and publishing the file descriptor of the new buffer to the corresponding subscribers, such as the streaming and recording modules. These subscribers will then use the file descriptor to read the encoded media data from the new buffer.

[0042] It should be noted that in this application, the process of the publisher module writing media data into the buffer is completed autonomously by the publisher module, and is not copied by the media chip's CPU, so it does not occupy the media chip's CPU processing resources.

[0043] S302: The first publisher module sends a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor for the first buffer.

[0044] For example, after the first publisher module writes the first media data into the first buffer, it sends a message publishing request to the central coordinator module. The message publishing request includes a first topic and a file descriptor for the first buffer. The message publishing request requests that the file descriptor be published to the first topic. It can be understood that the file descriptor is the message content that the first publisher module needs to publish, and the first topic is the subject of the message content to be published by the first publisher module. Accordingly, the central coordinator module parses the message publishing request to obtain the first topic and the file descriptor. Then, the central coordinator publishes the file descriptor to the first topic.

[0045] Understandably, since the first subscriber module and the central coordinator module are essentially different processes in the user space of the Linux system, and since different processes in the Linux system cannot directly perform inter-process communication (IPC), but instead forward communication through the kernel space, the first publisher module sends a message publishing request to the central coordinator module through the kernel space. For example, the first publisher module sends a first message publishing request to the kernel space, where the first message publishing request includes a first topic and a first file descriptor for a first buffer. This first file descriptor is assigned to the first publisher module by the Broker when allocating the first buffer. Specifically, the first publisher module sends the above first message publishing request to the kernel space through the Socket-level Control Message Rights (SCM_RIGHTS) mechanism. Because file descriptors (fds) are process-private in the Linux system framework, the same fd is completely invalid in different processes. When the kernel forces cross-process sharing of the same buffer, different processes need to use different fds to legally access the same physical memory. The kernel space and user space belong to different processes. Therefore, the kernel modifies the first file descriptor (fd) to obtain a second file descriptor for the first buffer. This second file descriptor is the file descriptor of the first buffer obtained by the central coordinator module, i.e., the file descriptor in step S302 above. Then, the kernel sends the message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and the file descriptor of the first buffer.

[0046] Optionally, in one embodiment of this application, after the central coordinator module receives the message publishing request sent by the first publisher module, it uses the file description of the first buffer to determine that the first buffer has been written with media data (i.e., first media data). Before all subscriber modules have consumed all the first media data from the first buffer, the first media data should exclusively occupy the first buffer, and other media data should not be written to the first buffer.

[0047] For example, the central coordinator module generates a count value corresponding to the first buffer based on the file descriptor and the number of subscriber modules subscribed to the first topic. Specifically, based on the file descriptor, the buffer to be locked is determined to be the first buffer of that file descriptor. Then, the central coordinator module determines the number of subscriber modules subscribed to the first topic based on its maintained topic-subscriber registry. The central coordinator module then uses the number of subscriber modules subscribed to the first topic as the count value of the first buffer. Finally, the central coordinator module locks the first buffer based on the count value corresponding to the first buffer. Locking the first buffer using this count value means marking the first buffer as occupied before the count value is cleared, so that the first media data exclusively occupies the first buffer before the count value is cleared.

[0048] In one embodiment of this application, the aforementioned message publishing request further includes metadata of the first media data. This metadata includes, but is not limited to, the resolution, frame rate, sampling frequency, and source of the media data. The source of the media data primarily refers to whether the media data was captured via HDMI or originates from a network. It should be noted that the reason the publisher module publishes the media data metadata along with the buffer file descriptor to the subscriber module when publishing a message is to allow the subscriber module to know in advance what type of media data the publisher module has written to the buffer. This facilitates the determination of whether the media data is the media data it needs, and thus the decision to read the media data from the buffer. For example, if the RTP streaming module needs to stream a 1080p bitstream, after receiving the bitstream encoded by the encoding module, it determines whether the bitstream resolution is 1080p based on the bitstream's metadata. If so, it reads the bitstream from the buffer for streaming. Therefore, publishing the media data metadata along with the subscriber module facilitates the subscriber module's correct decision-making and avoids invalid reading of media data.

[0049] S303: The central coordinator module creates a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules.

[0050] For example, the central coordinator module determines the subscriber modules that have subscribed to the first topic based on the topic-subscriber registry it maintains. There can be one or more subscriber modules that have subscribed to the first topic. This application mainly uses the example of the first subscriber module being the subscriber module that has subscribed to the first topic for illustration.

[0051] It should be noted that in the existing publish-subscribe model, when the Broker pushes messages published in a topic to each subscriber, the content pushed to each subscriber is the same. However, since the subscriber module and the publisher module are different processes in user space, under the Linux system framework, different processes need to use different file descriptors (fds) to legally access the same physical memory. Therefore, unlike the existing publish-subscribe model, in this application, before the central coordinator module pushes the file descriptor (fd) published in the first topic to the subscriber modules subscribing to the first topic, it needs to generate a corresponding fd copy for each subscriber module subscribing to the first topic based on that file descriptor (fd). The fd copies corresponding to different subscriber modules are different, but all the fd copies of different subscriber modules point to the aforementioned first buffer. Therefore, for the first subscriber, the central coordinator module creates a first file descriptor copy corresponding to the file descriptor for the first subscriber module.

[0052] S304: The central coordinator module pushes a copy of the first file descriptor to the first subscriber module.

[0053] For example, the central coordinator module pushes the first fd copy to the first subscriber module via SCM_RIGHTS.

[0054] S305: The first subscriber module reads the first media data from the first buffer through the first file descriptor copy.

[0055] For example, the first subscriber module stores a copy of the first file descriptor in a first queue, where the first queue is used to cache the file descriptor copies pushed to the first subscriber module by the central coordinator module. Then, the first subscriber module consumes the first file descriptor copy from the first queue and performs address mapping on the first file descriptor copy (e.g., using mmap() mapping) to obtain the first buffer used to cache the first media data, i.e., determining the virtual address corresponding to the first subscriber module. Finally, the first subscriber module reads the first media data from the first buffer; that is, the first subscriber module uses the aforementioned virtual address to read the first media data from the first buffer.

[0056] It should be noted that, for the same buffer, although the publisher, subscriber, and kernel in this application use different file descriptors for the buffer, these different file descriptors all point to the same buffer. That is, when address mapping is performed using these different file descriptors, the different virtual addresses mapped all point to the buffer.

[0057] As can be seen, when modules in the media chip of this application need to share media data, the modules autonomously write the media data into a buffer, and then other modules read the media data from the buffer for use. This eliminates the need for the media chip's CPU to copy the media data, thus reducing CPU usage. Furthermore, when implementing media data transmission, this application does not directly transmit the media data itself to other modules, but rather the file descriptor of the buffer. Since the memory of the file descriptor (typically 4 bytes) is very small, the amount of data transmitted is reduced. Compared to directly transmitting media data, modules in the media chip can quickly obtain the file descriptor and use it to quickly retrieve the media data from the buffer, improving the efficiency of media data transmission between modules or threads within the media chip.

[0058] Optionally, in one embodiment of this application, after the first subscriber module successfully reads the first media data from the first buffer through the first file descriptor copy, the first subscriber module sends a first feedback message to the central coordinator module, wherein the first feedback message indicates that the first subscriber module has successfully read the first media data from the first buffer. Then, in response to the first feedback message, the central coordinator module decrements the count value corresponding to the first buffer by 1. After the subscriber module successfully reads the first media data, it decrements the count value of the first buffer by 1. Thus, when all subscriber modules that have subscribed to the first topic have successfully read the first media data, that is, when all subscriber modules of the first topic have successfully completed consumption, the count value of the first buffer will be reduced to 0, at which point the first buffer can be released. Therefore, in response to the count value corresponding to the first buffer being zero, the central coordinator module unlocks the first buffer to release the first buffer, that is, marks the state of the first buffer as idle, and the first buffer can be allocated to other publisher modules for writing data.

[0059] Optionally, in one embodiment of this application, since the acquisition module may continuously acquire new media data, the subscriber module will continuously receive file descriptor copies pushed by the broker. For example, if the acquisition module continuously acquires new video frames via HDMI, it will continuously publish the newly acquired video frames. The broker will then continuously push file descriptor copies of the buffers containing these new video frames to the subscriber module. However, due to network issues or limited processing resources of the subscriber module, it may be unable to consume the file descriptor copies pushed by the broker in a timely manner. This results in a large accumulation of file descriptor copies that cannot be consumed promptly. Because these file descriptor copies cannot be consumed in time, the count values ​​of the buffers corresponding to these file descriptors cannot be cleared, and the broker will not release these buffers, leading to a large amount of buffer space being occupied and causing a memory leak.

[0060] Therefore, to prevent a large number of file descriptor copies from accumulating on the subscriber module side, causing memory leaks, the queue used to store file descriptor copies in the subscriber module is set as a bounded queue. This bounded queue forces the removal of file descriptor copies that cannot be consumed in a timely manner.

[0061] The implementation process of the bounded queue in this application is explained below using the first subscriber module.

[0062] For example, the first subscriber module receives a copy of the first file descriptor through the receiving thread of IPC. Before storing the copy of the first file descriptor into the first queue, the first subscriber module obtains the current queue length of the first queue.

[0063] Optionally, in response to the queue length being less than or equal to a first threshold, indicating that the number of file descriptor copies stored in the first queue has not reached the maximum number limit of the first queue, the first subscriber module directly stores the first file descriptor copy into the first queue.

[0064] Optionally, in response to the queue length being greater than the first threshold, it indicates that the number of file descriptor copies stored in the first queue has reached the maximum number limit of the first queue. Then, the first subscriber module determines the second file descriptor copy in the first queue, wherein the second file descriptor copy is the file descriptor copy located at the head of the first queue, that is, the second file descriptor copy is the file descriptor copy that has been stored in the first queue for the longest time. Then, the second file descriptor copy is removed from the first queue, and the first file descriptor copy is stored in the first queue.

[0065] Understandably, since the second file descriptor copy is directly removed from the first queue, the first subscriber module essentially does not consume the second file descriptor copy. In a normal publish-consume model, the first subscriber will not send feedback to the central coordinator module indicating successful consumption of the second file descriptor copy. Consequently, the central coordinator module will not receive feedback from the first subscriber module regarding successful consumption of the second file descriptor copy. Therefore, the central coordinator module will not utilize the first subscriber module's decrementing of the second buffer's count value. As a result, the count value of the second buffer will never be cleared, causing the central coordinator module to continuously lock the second buffer, preventing it from being released and resulting in a buffer leak. The second buffer is the buffer described by the second file descriptor copy.

[0066] More specifically, a publisher module writes media data into the second buffer, and the first subscriber module is a subscriber of the publisher module. The central coordinator module pushes a copy of the second file descriptor in the second buffer to the first subscriber module, and the first subscriber module stores the copy of the second file descriptor in the first queue.

[0067] Therefore, to address the memory leak issue in the buffer, this application sends a feedback message to the central coordinator module when the subscriber module removes the file descriptor copy pushed by the central coordinator module from the queue. This feedback message notifies the central coordinator module to decrement the count of the buffer described by the file descriptor copy by 1. In this way, even if the subscriber module does not consume the file descriptor copy (and has no opportunity to consume it again later), the central coordinator module will still decrement the count of the corresponding buffer, thus preventing the memory leak issue.

[0068] For example, after removing the second file descriptor copy from the first queue, the first subscriber module sends a second feedback message to the central coordinator module. Then, the central coordinator module decrements the count value of the second buffer described by the second file descriptor copy by 1 based on the second feedback message.

[0069] For example, the first feedback information and the second feedback information mentioned above can be the same information or different information, and this application does not limit this. Optionally, in one embodiment, this application considers removing the file descriptor copy from the queue as a successful consumption of the file descriptor copy by the subscriber module. In this case, the central coordinator module is notified with the same information as the first feedback information (the second feedback information). Upon receiving the second feedback information, the central coordinator module determines, based on the second feedback information, that the first subscriber module has successfully consumed the second file descriptor copy, and decrements the count value of the second buffer described by the second file descriptor copy by 1. In another embodiment, this application considers removing the file descriptor copy from the queue as a failure of the subscriber module to consume the file descriptor copy. Therefore, a second feedback information different from the first feedback information is used, i.e., information specifically designed to notify the central coordinator module to decrement the count value by 1. Upon receiving the second feedback information, the central coordinator module determines, based on the second feedback information, that the count value of the second buffer needs to be decremented by 1.

[0070] Optionally, in one embodiment of this application, when a first file descriptor copy is stored in a first queue, the time when the first file descriptor copy is enqueued is recorded. When a first file descriptor copy is consumed from the first queue, the time when the first file descriptor copy is dequeued is recorded. Finally, the processing delay of the first file descriptor is calculated based on the enqueue and dequeue times of the first file descriptor copy. For example, the time difference between the dequeue and enqueue times of the first file descriptor copy is used as the processing delay. Then, if the processing delay is greater than a second threshold, an alarm is triggered, which is mainly used to indicate that the processing delay of the file descriptor copy in the queue is greater than the second threshold.

[0071] Optionally, in one embodiment of this application, after removing the second file descriptor copy from the first queue, the count value corresponding to the first queue is incremented by 1, wherein the count value corresponding to the first queue is used to count the number of file descriptor copies removed from the first queue. Counting the number of file descriptor copies removed from the queue facilitates monitoring the system load of subscribers.

[0072] Optionally, in one embodiment of this application, retrieving a first file descriptor copy from the first queue and consuming and address mapping the first file descriptor copy are implemented by the processing thread of the first subscriber module.

[0073] The following section, with reference to the accompanying diagrams, describes in more detail the entire process of the subscriber module from receiving a copy of a file descriptor to consuming that copy.

[0074] See Figure 4 , Figure 4This is a schematic diagram illustrating the processing of a file descriptor copy, provided as an embodiment of this application.

[0075] During the above processing, the subscriber module's receiving thread is used to perform the following steps, including but not limited to: S401: Receives a copy of the file descriptor pushed by the central coordinator module.

[0076] S402: Apply a mutual exclusion lock to the queue.

[0077] Since the subscriber module's receiving thread and processing thread may operate on the queue simultaneously, the purpose of mutual exclusion locking is to allow only the receiving thread to operate on the queue at the current time, so as to prevent the receiving thread and processing thread from operating on the queue at the same time, modifying the file description copy in the queue, and causing data corruption.

[0078] S403: Determine if the length of the queue is greater than the first threshold. If yes, proceed to step S404; otherwise, proceed to step S408.

[0079] S404: Retrieve the copy of the file descriptor at the head of the queue.

[0080] S405: Release the copy of the file descriptor at the head of the queue from the queue.

[0081] In step S405, releasing the file descriptor copy at the head of the queue means that the subscriber module no longer consumes the file descriptor copy, nor reads data from the buffer described by the file descriptor copy. This is mainly used to notify the central coordinator module to decrement the count value of the buffer by 1, creating conditions for the subsequent recycling and reuse of the buffer.

[0082] S406: Remove the copy of the file descriptor at the head of the queue from the queue.

[0083] Understandably, step S406 actually removes the file descriptor copy from the queue to make room for the newly received file descriptor copy.

[0084] S407: Increment the queue count by 1 using the frame loss counter.

[0085] The queue count is mainly used to count the number of file descriptor copies removed, i.e. the number of dropped frames, which facilitates queue monitoring and troubleshooting of system bottlenecks.

[0086] S408: Push the received file descriptor copy to the end of the queue and record the current timestamp.

[0087] Pushing the file descriptor copy received by the receiving thread to the end of the queue means storing the file descriptor copy received by the receiving thread at the end of the queue.

[0088] The main purpose of recording the current timestamp is to record the time when the file descriptor copy is enqueued.

[0089] S409: Release the mutex lock on the queue and send the condition variable to the processing thread.

[0090] The purpose of releasing the mutex is mainly to allow other threads to process the queue, making it easier for processing threads to process the file descriptor copies in the queue.

[0091] Sending condition variables is mainly used to wake up processing threads that are in a waiting state.

[0092] During the above processing, the subscriber module's processing thread is used to execute the following steps, including but not limited to: S410: Condition variable for waiting for the receiving thread.

[0093] S411: When a condition variable is received, retrieve the copy of the file descriptor at the head of the queue.

[0094] S412: Map the address of the file descriptor copy at the head of the queue to obtain the buffer described by the file descriptor copy.

[0095] S413: Read media data from the buffer.

[0096] S414: Calculate the time difference between the current time and the time when the file description copy was enqueued to obtain the processing delay of the file description copy.

[0097] S415: Determine whether the processing delay is greater than the second threshold. If yes, proceed to step S416; if no, proceed to step S417.

[0098] S416: Print alarm log.

[0099] S417: Instruct the central coordinator module to decrement the count value of the buffer by 1 in order to release the buffer.

[0100] In one embodiment of this application, in response to the first publisher module being an encoding module and the first subscriber module being a streaming module, the first media data is the media data encoded by the encoding module. It can be understood that when the first subscriber module is a streaming module, the encoded media data is a bitstream, and the streaming module, upon receiving the encoded bitstream, will use the bitstream for streaming.

[0101] The following is in conjunction with the appendix Figure 5 This describes the streaming process of this application. See also... Figure 5 , Figure 5This is a schematic diagram of a streaming process provided in an embodiment of this application. The streaming process is applied to the streaming module described above. For ease of understanding, this embodiment mainly uses an H.264 encoder as an example for illustration; however, this application does not limit the type of encoder.

[0102] The above streaming process includes, but is not limited to, the following steps: S501: Scan the encoded bitstream to obtain the start code in the encoded bitstream.

[0103] The starting code is either 00 00 00 01 or 00 00 01.

[0104] S502: Extract Network Abstraction Layer Units (NALUs) from the encoded bitstream based on the start code.

[0105] S503: Determine whether the NALU's payload is greater than the third threshold. If not, proceed to step S504; if yes, proceed to step S505.

[0106] S504: Using the NALU pass-through mode, the NALU is encapsulated to obtain an RTP packet.

[0107] S505: Using the FU-A fragmentation mode, the NALU is fragmented to obtain multiple code stream slices; each code stream slice is encapsulated to obtain multiple RTP packets, wherein the multiple RTP packets correspond one-to-one with the multiple code stream slices.

[0108] Each RTP packet includes: an FU Indicator field, an FU Header field, and fragment data (i.e., payload), where the fragment data is the bitstream fragment corresponding to that RTP packet. Optionally, the FU Indicator field inherits the original NRI of the NALU, and the Type is fragmentation mode FU-A, meaning the FU Indicator field value is fixed at 28 in all RTP packets. The FU Header field includes a one-bit field indicating the first fragment (S), a one-bit field indicating the last fragment (E), a one-bit reserved field, and five fields indicating the NALU type, where the NALU type is the original NALU type. For example, for the first bitstream fragment, S=1, E=0, R=0, and the original type=0101, so the FUHeader field value of the RTP encapsulated from the first bitstream fragment is: 10000101.

[0109] Furthermore, the multiple RTP packets share the same RTP timestamp, which makes it easier for the decoding module to know that the multiple RTP time packets are obtained by fragmenting and encapsulating the same NALU during streaming, thus facilitating the decoding module to reconstruct the complete NALU.

[0110] S506: Push the encapsulated RTP packets.

[0111] As can be seen, in this application, when performing RTP streaming, the appropriate transmission mode is dynamically selected adaptively according to the size of the bitstream, thereby improving the transmission efficiency and flexibility of the bitstream during streaming.

[0112] The following describes the process of media data flow between various modules of the media chip, starting from the initial acquisition of raw media data by the media chip and the subsequent use of media data by various modules in the user space of the media chip.

[0113] See Figure 6 , Figure 6 This is a schematic diagram of a transmission media data provided in an embodiment of this application.

[0114] For example, such as Figure 6 As shown, the media chip includes an acquisition module, an encoding module, a recording module, a preview module, a streaming module, and a central coordinator module. The acquisition module acquires raw media data via HDMI and writes it to buffer 1, allocated to it by the central coordinator module. Specifically, the process of writing raw media data to buffer 1 involves the media chip's CPU copying the raw media data cached in kernel mode from the HDMI cable to buffer 1; that is, the media chip performs a data copy using the CPU during the data acquisition process. Then, the acquisition module publishes the file descriptor (fd1) of buffer 1 and the metadata of the raw media data to Topic A, where Topic A is the Topic corresponding to the raw media data, used for publishing the raw media data. Next, based on fd1, the central coordinator module creates file descriptor copies fd1_1 and fd1_2 for the encoding and recording modules, respectively, to describe buffer 1. Finally, the central coordinator module pushes fd1_1 and its metadata to the encoding module, and fd1_2 and its metadata to the preview module.

[0115] Furthermore, the preview module uses fd1_2 for address mapping, reads the original media data from buffer 1, and previews the original media data. Similarly, the encoding module uses fd1_1 for address mapping, reads the original media data from buffer 1, and encodes the original data to obtain the encoded media data.

[0116] Furthermore, after encoding the raw data, the encoding module writes the encoded media data into buffer 2, which is allocated to the encoding module by the central coordinator module. Then, the encoding module publishes the file descriptor fd2 of buffer 2 and the metadata of the media data to Topic B, where Topic B is a dedicated topic for publishing the encoded media data.

[0117] Furthermore, based on fd2, the central coordinator module creates file description copies fd2_1 and fd2_2 for describing buffer 1, respectively, for the recording module and the streaming module. Then, the central coordinator module pushes fd2_1 and its metadata to the recording module, and pushes fd2_2 and its metadata to the streaming module.

[0118] Furthermore, the recording module uses fd2_1 for address mapping, reads the encoded media data from buffer 2, and uses the encoded media data for recording. Similarly, the streaming module uses fd2_2 for address mapping, reads the encoded media data from buffer 2, and uses the encoded media data for streaming.

[0119] In existing technologies, media data flow involves the CPU performing a data copy process from kernel mode to user mode for the acquisition module. Then, when the encoding and preview modules need to use the raw media data, each module needs to perform another data copy process from kernel mode to user mode (for details, please refer to...). Figure 1 Only by showing the content can the original media data be obtained; the CPU has already performed three data copies here. Furthermore, when the recording module and the streaming module need the encoded media data, each module also needs to perform another data copy process from kernel mode to user mode (for details, please refer to...). Figure 1As shown in the diagram, the CPU needs to perform two data copying processes, resulting in a total of five data copying processes before each module can acquire the media data it needs. In contrast, this application only requires one data copy during the media data acquisition phase, allowing each module to acquire its required media data, significantly reducing the number of data copying operations and CPU usage. Furthermore, existing technologies require multiple copies, leading to multi-path distribution (multi-path copying) to various modules for efficient media data transfer, resulting in high memory bandwidth usage. This application only requires one data copy, thus reducing memory bandwidth usage. Additionally, existing technologies transmit the media data itself during inter-process data transfer (copying media data), resulting in a large amount of data transferred and a large number of IPCs. This application, however, transmits file descriptors and media data metadata between processes, resulting in a smaller data volume and thus reducing the number of IPCs. For example, for an NV12 image frame with a memory size of 3.1M, the existing technology has an IPC data volume of 3.1MB / frame when transmitting across processes. However, this application transmits file descriptors and media data metadata, and the IPC data volume is 204 bytes / frame, thus greatly compressing the IPC data volume.

[0120] See Figure 7 , Figure 7 This is a schematic diagram of another media chip provided in an embodiment of this application. The media chip 700 includes: multiple publisher modules 701, a central coordinator module 702, and multiple subscriber modules 703; The first publisher module is used to write the first media data into the first buffer, wherein the first publisher module is any one of the plurality of publisher modules 701; The first publisher module is used to send a message publishing request to the central coordinator module 702, wherein the message publishing request includes a first topic and a file descriptor of the first buffer; The central coordinator module 702 is used to create a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules 703; Central coordinator module 702 is used to push the copy of the first file descriptor to the first subscriber module; The first subscriber module is configured to read the first media data from the first buffer via a copy of the first file descriptor.

[0121] In one embodiment of this application, after the first publisher module sends a message publishing request to the central coordinator module 702, the central coordinator module 702 is further configured to: Based on the file descriptor and the number of subscriber modules subscribing to the first topic, a count value corresponding to the first buffer is generated; the first buffer is locked based on the count value corresponding to the first buffer so that the first media data exclusively occupies the first buffer until the count value corresponding to the first buffer is cleared.

[0122] In one embodiment of this application, after the first subscriber module reads the first media data from the first buffer through the first file descriptor copy, the first subscriber module is further configured to: Send a first feedback message to the central coordinator module 702, wherein the first feedback message is used to indicate that the first subscriber module has successfully read the first media data from the first buffer; Central coordinator module 702 is also used for: In response to the first feedback information, the count value corresponding to the first buffer is decremented by 1; in response to the count value corresponding to the first buffer being zero, the first buffer is unlocked to release the first buffer.

[0123] In one embodiment of this application, regarding the first subscriber module reading the first media data from the first buffer via the first file descriptor copy, the first subscriber module is specifically configured to: The first file descriptor copy is stored in a first queue, wherein the first queue is used to cache the file descriptor copies pushed to the first subscriber module by the central coordinator module 702; the first file descriptor copy is consumed from the first queue; the first file descriptor copy is address-mapped to obtain the first buffer for caching the first media data; and the first media data is read from the first buffer.

[0124] In one embodiment of this application, regarding the first subscriber module storing the copy of the first file descriptor in the first queue, the first subscriber module is specifically configured to: Obtain the current queue length of the first queue; in response to the queue length being less than or equal to a first threshold, store the first file descriptor copy in the first queue; in response to the queue length being greater than the first threshold, determine the second file descriptor copy, remove the second file descriptor copy from the first queue, and store the first file descriptor copy in the first queue, wherein the second file descriptor copy is the file descriptor copy located at the head of the first queue.

[0125] In one embodiment of this application, after removing the second file descriptor copy from the first queue, the first subscriber module is further configured to send second feedback information to the central coordinator module 702; the central coordinator module 702 is further configured to decrement the count value of the second buffer described by the second file descriptor copy by 1 based on the second feedback information.

[0126] In one embodiment of this application, the central coordinator module 702 is further configured to: Multiple buffers are pre-allocated from the memory of the media chip via the Direct Memory Access Heap (DMA) interface; a buffer allocation request is received from the first publisher module; in response to the buffer allocation request, the first buffer is allocated to the first publisher module from the multiple buffers, wherein the first buffer is a free buffer among the multiple buffers.

[0127] In one embodiment of this application, the plurality of publisher modules 701 include an acquisition module and an encoding module of the media chip; the plurality of subscriber modules 703 include the encoding module, a streaming module of the media chip, a preview module, and a recording module; in response to the first publisher module being the acquisition module, the first subscriber module is the encoding module and / or the preview module, and the first media data is raw media data, which is media data acquired by the acquisition module via HDMI or media data received via a network; in response to the first publisher module being the encoding module, the first subscriber module is the streaming module and / or the recording module, and the first media data is media data encoded by the encoding module after encoding the raw media data.

[0128] See Figure 8 , Figure 8 This is a schematic diagram of another media chip provided in an embodiment of this application.

[0129] The media chip 800 includes a memory 801, a processor 802, a communication interface 803, and a bus 804. The memory 801, processor 802, and communication interface 803 are interconnected via the bus 804.

[0130] The memory 801 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 801 may store a program; when the program stored in the memory 801 is executed by the processor 802, the processor 802 and the communication interface 803 are used to execute the various steps performed by the media chip in the media data transmission method of the embodiments of this application.

[0131] The processor 802 may be a general-purpose central processing unit (CPU), microprocessor, application specific integrated circuit (ASIC), graphics processing unit (GPU), or one or more integrated circuits, used to execute relevant programs to implement the media data transmission method in the method embodiments of this application.

[0132] The processor 802 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the media data transmission method of this application can be completed by the integrated logic circuitry in the hardware of the processor 802 or by instructions in software form. The aforementioned processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory 801, and processor 802 reads information from memory 801 to execute various steps of the media data transmission method.

[0133] The communication interface 803 can be a transceiver device such as a transceiver to enable communication between the media chip 800 and other devices or communication networks. The communication interface 803 can also be an input-output interface to enable data transmission between the media chip 800 and input-output devices, including but not limited to keyboards, mice, displays, USB flash drives, and hard drives. For example, the processor 802 can receive signals through the communication interface 803.

[0134] Bus 804 may include a pathway for transmitting information between various components of device media chip 800 (e.g., memory 801, processor 802, communication interface 803).

[0135] It should be noted that, although Figure 8 The media chip 800 shown only illustrates the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, the media chip 800 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the media chip 800 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the media chip 800 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 8 All the devices shown.

[0136] This application also provides a computer-readable storage medium storing a computer program that is executed by a processor to implement some or all of the steps of any of the media data transmission methods described in the above method embodiments.

[0137] This application also provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program operable to cause a computer to perform some or all of the steps of any of the media data transmission methods described in the above method embodiments.

[0138] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0140] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.

[0141] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0142] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software program module.

[0143] If the integrated unit is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0144] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage device, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0145] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A media data transmission method, characterized in that, The method is applied to a media chip, which includes multiple publisher modules, multiple subscriber modules, and a central coordinator module. The first publisher module writes the first media data into the first buffer, wherein the first publisher module is any one of the plurality of publisher modules; The first publisher module sends a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor for the first buffer; The central coordinator module creates a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules; The central coordinator module pushes a copy of the first file descriptor to the first subscriber module. The first subscriber module obtains the current queue length of the first queue; in response to the queue length being less than or equal to a first threshold, the first subscriber module stores the first file descriptor copy into the first queue; in response to the queue length being greater than the first threshold, the first subscriber module determines the second file descriptor copy, removes the second file descriptor copy from the first queue, and stores the first file descriptor copy into the first queue, wherein the second file descriptor copy is the file descriptor copy located at the head of the first queue; The first subscriber module consumes the first file descriptor copy from the first queue; the first subscriber module performs address mapping on the first file descriptor copy to obtain the first buffer for caching the first media data; the first subscriber module reads the first media data from the first buffer; After removing the second file descriptor copy from the first queue, the first subscriber module sends a second feedback message to the central coordinator module; the central coordinator module decrements the count value of the second buffer described by the second file descriptor copy by 1 based on the second feedback message.

2. The method according to claim 1, characterized in that, After the first publisher module sends a message publishing request to the central coordinator module, the method further includes: The central coordinator module generates a count value corresponding to the first buffer based on the file descriptor and the number of subscriber modules that subscribe to the first topic. The central coordinator module locks the first buffer based on the count value corresponding to the first buffer, so that the first media data exclusively occupies the first buffer before the count value corresponding to the first buffer is cleared to zero.

3. The method according to claim 2, characterized in that, After the first subscriber module reads the first media data from the first buffer via the first file descriptor copy, the method further includes: The first subscriber module sends a first feedback message to the central coordinator module, wherein the first feedback message is used to indicate that the first subscriber module has successfully read the first media data from the first buffer; In response to the first feedback information, the central coordinator module decrements the count value corresponding to the first buffer by 1. The central coordinator module unlocks the first buffer in response to a count value of zero, thereby releasing the first buffer.

4. The method according to claim 1, characterized in that, The method further includes: The central coordinator module pre-allocates multiple buffers from the memory of the media chip via the Direct Memory Access Heap (DMA) interface. The central coordinator module receives a buffer allocation request from the first publisher module; In response to the buffer allocation request, the central coordinator module allocates the first buffer to the first publisher module from the plurality of buffers, wherein the first buffer is an idle buffer among the plurality of buffers.

5. The method according to claim 1, characterized in that, The multiple publisher modules include the acquisition module and encoding module of the media chip; The multiple subscriber modules include the encoding module, the media chip's streaming module, the preview module, and the recording module; In response to the first publisher module being the acquisition module, the first subscriber module is the encoding module and / or the preview module, and the first media data is raw media data, which is media data acquired by the acquisition module through the high-definition multimedia interface HDMI or media data received through the network; If the first publisher module is the encoding module, then the first subscriber module is the streaming module and / or the recording module, and the first media data is media data encoded by the encoding module after encoding the original media data.

6. A media chip, characterized in that, include: Multiple publisher modules, multiple subscriber modules, and a central coordinator module; The first publisher module is used to write the first media data into the first buffer, wherein the first publisher module is any one of the plurality of publisher modules; The first publisher module is configured to send a message publishing request to the central coordinator module, wherein the message publishing request includes a first topic and a file descriptor for the first buffer; The central coordinator module is used to create a copy of the first file descriptor corresponding to the file descriptor for the first subscriber module, wherein the first subscriber module is the subscriber module that subscribes to the first topic among the plurality of subscriber modules; The central coordinator module is used to push the copy of the first file descriptor to the first subscriber module; The first subscriber module is configured to: obtain the current queue length of the first queue; in response to the queue length being less than or equal to a first threshold, store the first file descriptor copy in the first queue; in response to the queue length being greater than the first threshold, determine the second file descriptor copy, remove the second file descriptor copy from the first queue, and store the first file descriptor copy in the first queue, wherein the second file descriptor copy is the file descriptor copy located at the head of the first queue; consume the first file descriptor copy from the first queue; perform address mapping on the first file descriptor copy to obtain the first buffer for caching the first media data; read the first media data from the first buffer; and after removing the second file descriptor copy from the first queue, send second feedback information to the central coordinator module. The central coordinator module is used to decrement the count value of the second buffer described by the second file descriptor copy by 1 based on the second feedback information.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is executed by a processor to implement the method as described in any one of claims 1-5.

Images