A media stream processing method and apparatus

Abstract

This invention provides a media stream processing method and apparatus, relating to the field of data processing technology. The method includes: receiving a target media stream and encapsulating the target media stream into a target data packet with a preset data format; upon first receiving the target media stream, determining target processing types and target processing order for processing the target media stream; determining target plugins in a local plugin set that process data according to each target processing type; instantiating each target plugin to obtain plugin instances; setting data flow relationships between the plugin instances according to the target processing order to obtain target plugin chain instances; and calling the target plugin chain instances to process the target data packet. Applying the solution provided in this application to process media stream data can improve the flexibility of media gateway devices in responding to diverse media stream processing needs.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a media stream processing method and apparatus. Background Technology

[0002] Media gateway devices typically possess one or more processing functions for media streams, such as protocol access, encoding / decoding, transcoding, forwarding, recording, and intelligent analysis. Most existing media gateway devices integrate these processing functions into a single processing flow, following this flow to process the received media streams.

[0003] However, the processing requirements for media streams are becoming increasingly diverse, while existing media gateway devices can only process media streams according to their fixed processing procedures. Therefore, existing media gateway devices have limited flexibility in meeting the diverse processing needs of media streams. Summary of the Invention

[0004] The purpose of this invention is to provide a media stream processing method and apparatus to improve the flexibility of media gateway devices in responding to diverse media stream processing needs. The specific technical solution is as follows:

[0005] In a first aspect, embodiments of this application provide a media stream processing method applied to a media gateway device, the method comprising:

[0006] Receive the target media stream and encapsulate the target media stream into a target data packet with a preset data format;

[0007] Upon first receiving the target media stream, determine the target processing types and the target processing order for processing the target media stream;

[0008] In the local plugin set, each target plugin is determined to process data according to each target processing type, wherein the input data and output data of each plugin in the local plugin set are data packets in the preset data format;

[0009] Each target plugin is instantiated to obtain a plugin instance. According to the target processing order, the data flow relationship between each plugin instance is set to obtain a target plugin chain instance.

[0010] The target plugin chain instance is invoked to process the target data packet.

[0011] In one possible embodiment, if a plugin adjustment instruction is received during the process of calling the target plugin chain instance to process the target data packet, the method further includes the following steps:

[0012] The target plugin chain instance is paused from processing unprocessed target data packets, and received but unprocessed target data packets, as well as continuously received target data packets, are cached.

[0013] According to the plugin adjustment instructions, create a substitute plugin chain instance for the target plugin chain instance;

[0014] After the target plug-in chain instance finishes processing the data packets that were not processed during the pause and the substitute plug-in chain instance is successfully created, the substitute plug-in chain instance is invoked to process the cached data packets and the newly received target data packets.

[0015] In one possible embodiment, pausing the target plug-in chain instance to process unprocessed target data packets includes:

[0016] The first thread adds a thread lock to the second thread running the target plugin chain instance to prevent the second thread from processing unprocessed target data packets. Both the first thread and the second thread belong to the media stream processing process in the media gateway device.

[0017] In one possible embodiment, creating a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instruction includes:

[0018] Using a third thread, a substitute plugin chain instance is created for the target plugin chain instance according to the plugin adjustment instructions;

[0019] The first thread, the second thread, and the third thread all belong to the media stream processing process in the media gateway device. The first thread is used to add and remove thread locks for the second thread. The second thread runs the target plugin chain instance and is used to process the target data packets.

[0020] In one possible embodiment, the invocation of the substitute plug-in chain instance to process the cached data packets and the newly received target data packets includes:

[0021] The first thread releases the thread lock of the second thread. Both the first and second threads belong to the media stream processing process in the media gateway device. When the first thread pauses the target plug-in chain instance, it adds a thread lock to the second thread that is running the target plug-in chain instance to prevent the second thread from processing unprocessed target data packets.

[0022] The second thread runs the substitute plug-in chain instance to process cached data packets and newly received target data packets.

[0023] In one possible embodiment, the step of invoking the target plugin chain instance to process the target data packet includes:

[0024] Set the memory pointer to point to the target plugin chain instance;

[0025] The target data packet is processed by calling the instance pointed to by the memory pointer;

[0026] The process of calling the substitute plugin chain instance to process cached data packets and newly received target data packets includes:

[0027] Adjust the memory pointer to point to the substitute plug-in chain instance;

[0028] The instance pointed to by the memory pointer is invoked to process the cached data packets and the newly received target data packets.

[0029] In one possible embodiment, the media gateway device stores service configuration information corresponding to a preset data source, and the service configuration information records the processing type and processing order for processing the media stream of the data source;

[0030] The determination of the target processing type and target processing order for processing the target media stream includes:

[0031] Determine the target data source for pushing the target media stream;

[0032] Among the service configuration information stored in the media gateway device, determine the service configuration information corresponding to the target data source;

[0033] Obtain the target processing type and target processing order recorded in the determined business configuration information.

[0034] Secondly, embodiments of this application provide a media stream processing apparatus applied to a media gateway device, the apparatus comprising:

[0035] A media stream encapsulation unit is used to receive a target media stream and encapsulate the target media stream into a target data packet with a preset data format.

[0036] The media stream determination unit is used to determine, upon first receiving the target media stream, the target processing types and the target processing order for processing the target media stream;

[0037] The plugin determination unit is used to determine each target plugin in the local plugin set that performs data processing according to each target processing type, wherein the input data and output data of each plugin in the local plugin set are data packets in the preset data format;

[0038] The instance acquisition unit is used to instantiate each target plugin to obtain a plugin instance, and set the data flow relationship between each plugin instance according to the target processing order to obtain a target plugin chain instance.

[0039] The first processing unit is used to call the target plugin chain instance to process the target data packet.

[0040] In one possible embodiment, the device further includes:

[0041] The instruction receiving unit is used to, during the process of calling the target plugin chain instance to process the target data packet, if a plugin adjustment instruction is received, pause the processing of the unprocessed target data packet by the target plugin chain instance, and cache the received and unprocessed target data packets as well as the continuously received target data packets.

[0042] An instance creation unit is used to create a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instructions.

[0043] The second processing unit is used to call the substitute plug-in chain instance to process the cached data packets and the newly received target data packets after the target plug-in chain instance has finished processing the data packets that were not processed during the pause and the substitute plug-in chain instance has been successfully created.

[0044] In one possible embodiment, the instruction receiving unit is specifically used for:

[0045] The first thread adds a thread lock to the second thread running the target plugin chain instance to prevent the second thread from processing unprocessed target data packets. Both the first thread and the second thread belong to the media stream processing process in the media gateway device.

[0046] In one possible embodiment, the instance creation unit is specifically used for:

[0047] A substitute plugin chain instance for the target plugin chain instance is created using a third thread, according to the plugin adjustment instructions.

[0048] The first thread, the second thread, and the third thread all belong to the media stream processing process in the media gateway device. The first thread is used to add and remove thread locks for the second thread. The second thread runs the target plugin chain instance and is used to process the target data packets.

[0049] In one possible embodiment, the second processing unit is specifically used for:

[0050] The first thread releases the thread lock of the second thread. Both the first and second threads belong to the media stream processing process in the media gateway device. When the first thread pauses the target plug-in chain instance, it adds a thread lock to the second thread that is running the target plug-in chain instance to prevent the second thread from processing unprocessed target data packets.

[0051] The second thread runs the substitute plug-in chain instance to process cached data packets and newly received target data packets.

[0052] In one possible embodiment, the first processing unit is specifically used for:

[0053] Set the memory pointer to point to the target plugin chain instance;

[0054] The target data packet is processed by calling the instance pointed to by the memory pointer;

[0055] The second processing unit is specifically used for:

[0056] Adjust the memory pointer to point to the substitute plug-in chain instance;

[0057] The instance pointed to by the memory pointer is invoked to process the cached data packets and the newly received target data packets.

[0058] In one possible embodiment, the media gateway device stores service configuration information corresponding to a preset data source, and the service configuration information records the processing type and processing order for processing the media stream of the data source;

[0059] The media stream determination unit is specifically used for:

[0060] Determine the target data source for pushing the target media stream;

[0061] Among the service configuration information stored in the media gateway device, determine the service configuration information corresponding to the target data source;

[0062] Obtain the target processing type and target processing order recorded in the determined business configuration information.

[0063] Thirdly, embodiments of this application provide an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0064] Memory, used to store computer programs;

[0065] When a processor executes a program stored in memory, it implements any of the steps described in the first aspect.

[0066] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the methods described in the first aspect.

[0067] Beneficial effects of the embodiments of the present invention:

[0068] As can be seen from the above, when processing media streams using the solution provided in this application, when the media gateway device receives a new media stream, i.e., when it receives the target media stream for the first time, the media gateway device can determine the target processing type and target processing order for processing the target media stream. Based on the target processing type, it finds the target plugin, instantiates the target plugin to obtain a plugin instance, and sets the data flow relationship of each plugin instance according to the target processing order to obtain a target plugin chain instance. Thus, the media gateway device can call the target plugin chain instance to process the data packets of the target media stream. It is evident that, facing various media streams with diverse business requirements, the media gateway device can create customized plugin chain instances for the media stream upon first receiving it. That is, the media gateway device can flexibly arrange and combine plugins according to the business requirements of the media stream to form a customized processing pipeline. Therefore, applying the media stream processing solution provided in this application can improve the flexibility of the media gateway device in responding to diverse media stream processing needs. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0070] Figure 1 A schematic diagram illustrating a media stream processing method provided in an embodiment of this application;

[0071] Figure 2 A flowchart illustrating the first media stream processing method provided in this application embodiment;

[0072] Figure 3a A schematic diagram of the first method for creating a substitute plug-in chain instance provided in this application embodiment;

[0073] Figure 3b A schematic diagram illustrating the second method for creating a substitute plug-in chain instance provided in this application embodiment;

[0074] Figure 3c A flowchart illustrating the third method for creating a substitute plug-in chain instance provided in this application embodiment;

[0075] Figure 3d A schematic diagram illustrating the fourth method for creating a substitute plug-in chain instance provided in this application embodiment;

[0076] Figure 4 This application provides a schematic diagram of inter-thread interaction as an embodiment of the present application.

[0077] Figure 5 This is a schematic diagram of the structure of a media gateway device provided in an embodiment of this application;

[0078] Figure 6 A flowchart illustrating the second media stream processing method provided in this application embodiment;

[0079] Figure 7 A schematic diagram of the structure of the first media stream processing device provided in the embodiments of this application;

[0080] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on this application are within the scope of protection of the present invention.

[0082] First, the application scenarios of the media stream processing solution provided in the embodiments of this application will be described.

[0083] See Figure 1 This provides a scenario diagram illustrating a media stream processing method, from Figure 1 As can be seen, the media gateway device is used to process received media streams. These streams can be audio streams, video streams, media streams obtained via RTMP (Real-Time Messaging Protocol), media streams obtained via WebRTC (WebReal-Time Communications), and so on. The media gateway device can perform different processing on different media streams, obtaining different processing results, and then push the streams to downstream servers, for example, via RTMP, GB28181, WebRTC, etc. Furthermore, when the media gateway device receives a video stream and performs video recording processing on it, the resulting video file is the video file itself, which the media gateway device can also store.

[0084] The following is a detailed description of a media stream processing method and apparatus provided in the embodiments of this application.

[0085] In one possible embodiment, the above-described media stream processing method is applied to a media gateway device. The method includes: receiving a target media stream and encapsulating the target media stream into a target data packet with a preset data format; upon first receiving the target media stream, determining the target processing types and target processing order for processing the target media stream; determining each target plugin in a local plugin set that processes data according to each target processing type, wherein the input and output data of each plugin in the local plugin set are data packets with a preset data format; instantiating each target plugin to obtain a plugin instance, setting the data flow relationship between each plugin instance according to the target processing order to obtain a target plugin chain instance; and calling the target plugin chain instance to process the target data packet.

[0086] In this solution, when a media gateway device receives a new media stream, i.e., upon first receiving the target media stream, the media gateway device can determine the target processing type and target processing order for the target media stream. Based on the target processing type, it finds the target plugin, instantiates the target plugin to obtain a plugin instance, and sets the data flow relationship of the plugin instances according to the target processing order to obtain a target plugin chain instance. Thus, the media gateway device can call the target plugin chain instance to process the data packets of the target media stream. It is evident that, facing various media streams with diverse business requirements, the media gateway device can create customized plugin chain instances for the media stream upon first receiving it. That is, the media gateway device can flexibly arrange and combine plugins according to the business requirements of the media stream to form a customized processing pipeline. Therefore, applying the media stream processing solution provided in this application embodiment can improve the flexibility of the media gateway device in responding to diverse media stream processing needs.

[0087] From another perspective, the media gateway device in this solution possesses a programmable chained processing mechanism. This mechanism abstracts the media stream processing process into a plug-in chain, allowing data to be processed sequentially on a customizable pipeline, achieving a high degree of controllability and programmability in the processing flow. Within the plug-in chain, each plug-in has a single, independent responsibility; changes to the internal logic of any plug-in will not affect other plug-ins or the gateway core. Chained processing also enables faster and clearer problem localization when errors occur during media stream processing, and faults in individual plug-ins can be effectively isolated, thereby improving the robustness of the media gateway device's operating system.

[0088] Each step in the method mentioned in the above embodiments will be described in detail below.

[0089] In one possible embodiment, see Figure 2The above media stream processing method includes the following steps S201-S205.

[0090] Step S201: Receive the target media stream and encapsulate the target media stream into a target data packet with a preset data format.

[0091] The aforementioned preset data format is a fixed format for pre-set data packets. This preset data format specifies the format of the media stream payload (i.e., the data that actually carries business information, such as H.264 data) and metadata. The metadata includes media stream timestamps, data formats, resolutions, media stream identifiers, etc.

[0092] Specifically, a media gateway device can receive media streams from different sources and with different protocols, such as WebRTC, RTMP, RTSP (Real Time Streaming Protocol), GB / T 28181, etc., and parse the media streams to obtain the various data contained in the media streams, thereby encapsulating the various data contained in the media streams into target data packets with preset data formats.

[0093] Step S202: Upon receiving the target media stream for the first time, determine the target processing types and the target processing order for processing the target media stream.

[0094] The aforementioned processing types may include encoding / decoding, transcoding, forwarding, recording, intelligent analysis, and so on.

[0095] When encapsulating a target media stream into a target data packet, the media gateway device can detect in parallel whether it is receiving the target media stream for the first time. For example, it can compare the media stream identifier of the target media stream (called the first identifier) ​​with the media stream identifiers of the media stream being processed by the media gateway device (called the second identifiers). If the first identifier is different from each of the second identifiers, it means that the media gateway device is receiving the target media stream for the first time.

[0096] Upon receiving a target media stream for the first time, the media gateway device can determine the target processing types and the order of processing for the target media stream. Specific determination methods will be described in subsequent embodiments and will not be detailed here.

[0097] Step S203: Determine the target plugins in the local plugin set that perform data processing according to each target processing type.

[0098] The local plugin collection contains a series of independent media processing plugins, which are the units that actually perform data processing tasks. For example, the local plugin collection may include decoding plugins, watermarking plugins, intelligent analysis plugins, encoding plugins, recording plugins, and so on.

[0099] The input and output data of each plugin in the local plugin collection are data packets in a preset data format.

[0100] The plugins in the local plugin collection are pre-developed by developers. During plugin development, developers can design both input and output data packets to be in a preset data format, ensuring smooth data packet flow between various plugins. Furthermore, each plugin processes data according to a specific data processing type; for example, an encoding plugin encodes data, an intelligent analysis plugin performs intelligent data analysis, and a recording plugin records video, etc. Therefore, the plugins included in the local plugin collection actually reflect the data processing capabilities of the media gateway device.

[0101] If new functionality needs to be added to a media gateway device, developers only need to develop the corresponding plugin and add it to the local plugin collection. The media gateway device can then use this plugin from the local collection to process media streams, achieving plug-and-play functionality and significantly reducing the cost of expanding the media gateway device's capabilities, thus improving its scalability. For example, if a watermarking function is required, developers only need to develop a watermarking plugin and add it to the local plugin collection. The media gateway device can then use this plugin to add watermarks to the media streams during processing. Similarly, deleting or modifying a function only requires deleting or modifying a plugin in the local plugin collection. Therefore, whether adding, deleting, or modifying functionality, developers only need to adjust the plugins in the local plugin collection, without affecting the core of the media gateway device, thus improving its reliability, stability, and scalability.

[0102] Developers can develop plugins according to a unified standard, ensuring that each plugin's functionality is cohesive and follows a unified interface. The advantage of developing plugins according to a unified standard is that different developers can develop different functional plugins in parallel, and the modular structure also simplifies and simplifies the long-term maintenance and functional iteration of media gateway devices.

[0103] For example, it can be based on Process (packet) The UnifiedMediaPacket's error method, Setup(config string) method, and Teardown() method are used to develop various plugins in a unified manner.

[0104] Process (packet) The `UnifiedMediaPacket`'s `error` method receives media packets as input, executes the core processing logic of the plugin under development (such as modifying the payload, adding or deleting metadata), and returns the data packet after execution to be passed to the next plugin. If an error is returned, the processing of the data packet can be interrupted.

[0105] The Setup(config string) method is the plugin initialization method, used to receive the plugin's configuration parameters when initializing the plugin.

[0106] The Teardown() method is a plugin destruction method used to destroy plugins and release device resources.

[0107] Specifically, each plugin is used to process data according to a data processing type. Therefore, there is a correspondence between plugins and data processing types. The media gateway device can store this correspondence. After the media gateway device obtains the target processing type by performing the above step S202, it can determine the plugin corresponding to the target processing type according to the correspondence. This plugin is the target plugin that processes data according to the target processing type.

[0108] Step S204: Instantiate each target plugin to obtain plugin instances, and set the data flow relationship between each plugin instance according to the target processing order to obtain the target plugin chain instance.

[0109] Specifically, after identifying each target plugin, plugin instantiation technology can be used to instantiate each target plugin to obtain each plugin instance.

[0110] The above target processing order refers to the order in which each target processing type processes the target media stream. Since processing types correspond to plugins, and plugin instances are instantiated from plugins, there is also a correspondence between plugin instances and plugins. Therefore, the order of the target plugins can be determined based on the above target processing order, and thus the order of each plugin instance can be determined. This allows setting the data flow relationship between each plugin instance according to its order, i.e., setting the input / output relationship between adjacent plugin instances, resulting in the target plugin chain instance. Because the input and output data of each plugin in the local plugin set are data packets in a preset data format, data packets can be transmitted between plugin instances in this target plugin chain instance. The processing results of upstream plugins can be consumed by downstream plugins, realizing information transmission and collaboration between plugins on the chain.

[0111] For example, if the target processing types include decoding, encoding, intelligent analysis, and recording, then the target plugins can be determined to include a decoding plugin, an encoding plugin, an intelligent analysis plugin, and a recording plugin, thus defining the plugin instances as: decoding plugin instance, encoding plugin instance, intelligent analysis plugin instance, and recording plugin instance. If the target processing order is: decoding, recording, intelligent analysis, and encoding, then the order of the plugin instances can be determined as: decoding plugin instance, recording plugin instance, intelligent analysis plugin instance, and encoding plugin instance. This allows setting the output of the decoding plugin instance to be connected to the input of the recording plugin instance, the output of the recording plugin instance to be connected to the input of the intelligent analysis plugin instance, and the output of the intelligent analysis plugin instance to be connected to the input of the encoding plugin instance, thus obtaining the target plugin chain instances.

[0112] Step S205: Call the target plugin chain instance to process the target data packet.

[0113] Specifically, after obtaining the target plug-in chain instance, the media gateway device can call the target plug-in chain instance to process the target data packets. Since the target media stream is streamed data, meaning the media gateway device can continuously receive the target media stream, it can continuously generate data packets for the target media stream. For example, it can encapsulate the received target media stream into target data packets at preset intervals, or it can encapsulate the received target media stream into target data packets when the size reaches a preset size. Therefore, except for the initial creation of the target plug-in chain instance upon receiving the target media stream, the media gateway device can directly call the target plug-in chain instance to process the target data packets of the target media stream upon subsequent receipts.

[0114] During the processing of the target media stream, the business requirements for the target media stream may change. For example, if the target media stream originally required intelligent analysis and watermarking, now it may also need to be recorded, or intelligent analysis may need to be replaced by recording, or watermarking may no longer be required, etc.

[0115] In response to the above situation, in one possible embodiment, if a plugin adjustment instruction is received during the process of calling the target plugin chain instance to process the target data packet, the method further includes: pausing the target plugin chain instance to process the unprocessed target data packet, and caching the received and unprocessed target data packets as well as the continuously received target data packets; creating a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instruction; and after the target plugin chain instance has finished processing the data packets that were not processed at the pause time and the substitute plugin chain instance has been successfully created, calling the substitute plugin chain instance to process the cached data packets and the newly received target data packets.

[0116] This solution, upon receiving a plug-in adjustment command, allows the media gateway device to pause processing of unprocessed target data packets by the target plug-in chain instance. A substitute plug-in chain instance is then created according to the command, and subsequently used for data processing. The plug-in adjustment command can add, delete, or replace plug-in instances within the target plug-in chain instance. Therefore, creating a substitute plug-in chain instance based on the command can be considered as adding, deleting, or replacing plug-in instances on top of the old chain (target plug-in chain instance), resulting in a new chain (substitute plug-in chain instance). This new chain then replaces the old chain for continued data processing. Throughout the entire response process, the media gateway device can still receive media streams normally, without affecting its operation. This achieves hot-swapping of the target plug-in chain instance (i.e., enabling, disabling, and replacing plug-in instances without system downtime) and enables dynamic and smooth changes to the business requirements of the target media stream, greatly improving the flexibility and agility of the media gateway device's operating system.

[0117] The steps further included in the methods mentioned in the above embodiments will be described in detail below.

[0118] In one possible embodiment, if a plugin adjustment instruction is received during the process of calling the target plugin chain instance to process the target data packet, the above method further includes the following steps one to three.

[0119] Step 1: Pause the target plugin chain instance to process unprocessed target data packets, and cache received but unprocessed target data packets as well as continuously received target data packets.

[0120] The aforementioned plugin adjustment commands can be commands that add, delete, or replace plugin instances in the target plugin chain instance.

[0121] As explained in step S205 above, the media gateway device can continuously receive the target media stream, continuously generate the target data packets of the target media stream, and call the target plug-in chain instance to process the target data packets. The target data packets generated by the target can be represented by Arabic numerals as: target data packet 1, target data packet 2, target data packet 3, etc.

[0122] Specifically, after the media gateway device creates a target plug-in chain instance, it actually executes steps S201 and S205 in parallel. That is, on one hand, it receives a continuous stream of target media streams and encapsulates them into target data packets; on the other hand, it calls the target plug-in chain to process the encapsulated target data packets. During this process, if the media gateway device receives a plug-in adjustment instruction, it can pause the target plug-in chain instance to process any unprocessed target data packets.

[0123] Although the target plug-in chain instance is paused from processing unprocessed target data packets, the media gateway device continues to receive target media streams and encapsulate them into target data packets. To ensure that important data is not lost, the media gateway device can cache received but unprocessed target data packets as well as continuously received target data packets.

[0124] For example, when a media gateway device receives a plugin adjustment instruction, if the received but unprocessed target data packet is target data packet M, the media gateway device can pause the target plugin chain instance to process target data packet M, and cache target data packet M as well as continuously received target data packets M+1, M+2, etc.

[0125] Step 2: Following the plugin adjustment instructions, create a substitute plugin chain instance for the target plugin chain instance.

[0126] The following explains the three scenarios for plugin adjustment commands.

[0127] 1. The plugin adjustment command is the command to add a new plugin instance.

[0128] In this scenario, the plugin adjustment command can specify the type of the new plugin instance and its insertion location. The media gateway device can then determine the plugin to be added based on the command, instantiate the determined plugin, and insert the resulting plugin instance into the target plugin chain instance at the location indicated by the command, thus creating a new plugin chain instance as a replacement. Inserting a new plugin instance can be done by adding a branch to the target plugin chain instance as instructed by the command, or by inserting a new plugin instance between two adjacent plugin instances in the target plugin chain instance.

[0129] For example, see Figure 3a This illustrates the first process for creating a substitute plugin chain instance. From Figure 3a As can be seen, the plugin instances in the target plugin chain instance are, according to the data flow relationship, as follows: decoding plugin instance, intelligent analysis plugin instance, and encoding plugin instance. According to the plugin adjustment instructions, a branch including the recording plugin instance can be added on the basis of the target plugin chain instance. Specifically, the input of the recording plugin instance is set as the input of the substitute plugin chain instance, and the output of the recording plugin instance is set as the output of the substitute plugin chain instance, thus obtaining the substitute plugin chain instance.

[0130] For example, see Figure 3b This illustrates the second process for creating a substitute plugin chain instance. Figure 3b Target plugin chain instances and Figure 3aThe target plugin chain instances are the same. According to the plugin adjustment instructions, new plugin instances can be inserted between the intelligent analysis plugin instance and the coding plugin instance (e.g., Figure 3b The watermarking plugin instance is obtained by disconnecting the connection between the output of the intelligent analysis plugin instance and the input of the encoding plugin instance, setting the connection between the output of the intelligent analysis plugin instance and the input of the watermarking plugin instance, and the connection between the output of the watermarking plugin instance and the input of the encoding plugin instance.

[0131] 2. The plugin adjustment command is the command to delete the plugin instance.

[0132] In this case, the plug-in adjustment command can indicate which plug-in instance needs to be deleted in the target plug-in chain instance. In this way, the media gateway device can locate the plug-in instance to be deleted in the target plug-in chain instance according to the plug-in adjustment command, disconnect the input of the plug-in instance from the output of the previous plug-in instance, disconnect the output of the plug-in instance from the input of the next plug-in instance, and reset the output of the previous plug-in instance from the input of the next plug-in instance to obtain a replacement plug-in chain instance.

[0133] For example, see Figure 3c This illustrates the third process for creating a substitute plugin chain instance. Figure 3c The target plugin chain instance contains plugin instances that are arranged according to data flow: decoding plugin instance, intelligent analysis plugin instance, watermarking plugin instance, and encoding plugin instance. If the plugin adjustment command indicates that the watermarking plugin instance should be deleted, the connection between the input of the watermarking plugin instance and the output of the intelligent analysis plugin instance, and the connection between the output of the watermarking plugin instance and the input of the encoding plugin instance can be disconnected. The connection between the output of the intelligent analysis plugin instance and the input of the encoding plugin instance can be set to be established, thus obtaining a substitute plugin chain instance.

[0134] 3. The plugin adjustment command is the command to replace the plugin instance.

[0135] In this scenario, the plugin adjustment command can specify the first plugin instance to be replaced and the second plugin instance to be replaced in the target plugin chain instance. The media gateway device can then create the second plugin instance according to the command, locate the first plugin instance in the target plugin chain instance, and replace the first plugin instance with the second plugin instance. Specifically, this involves disconnecting the input of the first plugin instance from the output of its predecessor, and disconnecting the output of the first plugin instance from the input of its next successor. Conversely, it sets the input of the second plugin instance to connect with the output of its predecessor and the input of its next successor.

[0136] For example, see Figure 3dThis illustrates the fourth process for creating a substitute plugin chain instance. Figure 3d Target plugin chain instances and Figure 3a If the target plugin chain instance is the same, and the plugin adjustment instruction indicates that the smart analysis plugin instance should be replaced with the watermark plugin instance, the media gateway device can create a watermark plugin instance and replace the smart analysis plugin instance in the target plugin chain instance with the watermark plugin instance.

[0137] Step 3: After the target plugin chain instance finishes processing the data packets that were not processed during the pause and the substitute plugin chain instance is successfully created, the substitute plugin chain instance is called to process the cached data packets and the newly received target data packets.

[0138] The pause time is defined as the moment when the target plugin chain instance stops processing unprocessed target data packets.

[0139] When a media gateway device pauses data processing on a target plug-in chain instance, there may be target data packets that are being processed but have not yet been completed. These target data packets are called residual data packets. When the media gateway device pauses data processing, it pauses the target chain instance's processing of unprocessed target data packets. However, the media gateway device can still use the target chain instance to process the residual data packets normally.

[0140] In the example of step one above, the target data packet M is an unprocessed target data packet. At the pause time mentioned above, the media gateway device may be processing the target data packet M-1 and has not yet finished processing it. If the target plug-in chain instance includes three sequential plug-in instances, the media gateway device may be processing the target data packet M-1 based on the second plug-in instance. Therefore, the target data packet M-1 is the remaining data packet. In this case, even if the media gateway device pauses the target plug-in chain instance to process the unprocessed target data packet, it can still use the target plug-in chain instance to process the target data packet M-1 normally.

[0141] The media gateway device can monitor the invocation status of each plugin instance in the target plugin chain instance. When none of the plugin instances in the target plugin chain instance have been invoked, it can be determined that the target plugin chain instance has finished processing the remaining data packets. After the target plugin chain instance has finished processing the remaining data packets and the substitute plugin chain instance has been created, the media gateway device can invoke the substitute plugin chain instance to process the cached data packets and newly received target data packets.

[0142] In one possible embodiment, suspending the processing of unprocessed target data packets by the target plug-in chain instance includes: by using a first thread to add a thread lock to a second thread running the target plug-in chain instance to prevent the second thread from processing the unprocessed target data packets, wherein both the first thread and the second thread belong to the media stream processing process in the media gateway device.

[0143] In another possible embodiment, according to the plug-in adjustment instruction, a substitute plug-in chain instance for the target plug-in chain instance is created, including: by using a third thread other than the first thread and the second thread, the substitute plug-in chain instance for the target plug-in chain instance is created according to the plug-in adjustment instruction. The first thread, the second thread and the third thread all belong to the media stream processing process in the media gateway device. The first thread is used to add and release thread locks for the second thread. The second thread runs the target plug-in chain instance and is used to process the target data packets.

[0144] In another possible embodiment, the process of calling the substitute plug-in chain instance to process the cached data packets and newly received target data packets includes: releasing the thread lock of the second thread through the first thread, wherein both the first thread and the second thread belong to the media stream processing process in the media gateway device; when the first thread pauses the target plug-in chain instance, it adds a thread lock to the second thread running the target plug-in chain instance to prevent the second thread from processing the unprocessed target data packets; and the second thread runs the substitute plug-in chain instance to process the cached data packets and newly received target data packets.

[0145] In this solution, the first, second, and third threads all belong to the media stream processing process. Each thread can share the memory of the media stream processing process, and data can be directly transferred between threads. Compared with existing technologies that use multi-process or microservice architectures to achieve functional decoupling, which split different functions into independent services and introduce additional inter-process or network communication overhead during media stream processing, increasing the burden of data serialization / deserialization, this solution has no performance loss from cross-process communication and data copying, greatly reducing the latency of real-time media stream processing and ensuring low latency and high efficiency in real-time media stream processing. Therefore, this solution can meet the stringent latency requirements of real-time media stream processing.

[0146] The following section explains each of the aforementioned threads and the specific process by which each thread processes the target data packet.

[0147] In one possible embodiment, the media gateway device runs a media stream processing process, which includes multiple threads running in parallel, including a first thread, a second thread, and a third thread.

[0148] First, let's introduce these three types of threads.

[0149] 1. First thread

[0150] The first thread can be understood as the main thread of the media stream processing process, and the first thread can manage other threads.

[0151] 2. Second thread

[0152] After obtaining the target media stream, the process or thread responsible for encapsulating the target data packet in the media gateway device can store the target data packet in independent memory allocated by the media gateway device for the media stream processing process. A second thread can then call a plugin chain instance to process the target data packet stored in this independent memory. Furthermore, there can be multiple second threads, allowing multiple target data packets to be processed in parallel.

[0153] For example, if the media stream processing process contains three second threads, namely second threads a, b, and c, when a target data packet K is stored in the aforementioned independent memory, an idle second thread (such as second thread a) can call the plug-in chain instance to process the target data packet K. When the aforementioned independent memory stores target data packet K+1, if second thread a is still processing the target data packet, another idle second thread (such as second thread b) can also call the plug-in chain instance to process the target data packet K+1. When the aforementioned independent memory stores target data packet K+2, if second threads a and b are still processing the target data packet, the last idle second thread (such as second thread c) can also call the plug-in chain instance to process the target data packet K+2.

[0154] 3. Third thread

[0155] The third thread is used to create a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instructions.

[0156] The following is combined with Figure 4 The diagram illustrating thread interaction uses a media stream processing process that includes a second thread as an example to explain the interaction process of these three types of threads.

[0157] When the media gateway device receives a plugin adjustment instruction, the first thread can, in response to receiving the instruction, acquire a thread lock for the second thread (step S401) to prevent the second thread from processing unprocessed target data packets. After acquiring the lock, the second thread will not process unprocessed target data packets in memory. However, if the second thread is processing data packets when it acquires the lock (i.e., there are remaining data packets), it will still process those remaining data packets normally. Furthermore, the first thread can monitor the status of the second thread; that is, the first thread can determine whether the second thread is processing data packets.

[0158] The first thread can also forward the plugin adjustment instruction to the third thread (step S402). After receiving the plugin adjustment instruction, the third thread creates a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instruction, and replaces the plugin chain instance called by the second thread from the target plugin chain instance with the substitute plugin chain instance (step S403). After creation is completed, the third thread can notify the first thread that the substitute plugin chain instance has been created (step S404).

[0159] Once the first thread determines, by monitoring the status of the second thread, that the remaining data packets have been processed (in the case of multiple second threads, it must be confirmed that all second threads have processed their respective remaining data packets), and receives notification from the third thread that a substitute plugin chain instance has been created, the second thread can release its thread lock (step S405). After the second thread unlocks, it can continue to call the plugin chain instance to process the unprocessed data packets stored in memory (step S406). At this time, the plugin chain instance called (or run) by the second thread is the substitute plugin chain instance, and the unprocessed data packets stored in memory include the target data packets cached during the second thread's locking period and the newly received target data packets.

[0160] In one possible embodiment, processing the target data packet by calling the target plug-in chain instance includes: setting a memory pointer to point to the target plug-in chain instance; calling the instance pointed to by the memory pointer to process the target data packet; and processing the cached data packet and newly received target data packet by calling a substitute plug-in chain instance, including: adjusting the memory pointer to point to the substitute plug-in chain instance; and calling the instance pointed to by the memory pointer to process the cached data packet and newly received target data packet. In this scheme, the memory pointer is used to specify the plug-in chain instance called by the media gateway device. When no plug-in adjustment instruction is received, the memory pointer points to the target plug-in chain instance, therefore, the media gateway device can call the target plug-in chain instance to process the target data packet. After receiving a plug-in adjustment instruction, the memory pointer is adjusted to point to the substitute plug-in chain instance, thereby changing the plug-in chain instance called by the media gateway device to the substitute plug-in chain instance. It is evident that by controlling the direction of the memory pointer, the media gateway device can call the correct plug-in chain instance to process the target data packet, thereby improving the accuracy of processing the target data packet and thus improving the accuracy of processing the media stream.

[0161] Specifically, when processing target data packets, the media gateway device locates the plug-in chain instance to be invoked based on the memory pointer, and then invokes the found plug-in chain instance to process the target data packets. Therefore, when the target media stream is received for the first time, after the target plug-in chain instance is created, the memory pointer in the media gateway device can be assigned the address information or identifier of the target plug-in chain instance. Thus, the media gateway device can locate the target plug-in chain instance based on the memory pointer and invoke the target plug-in chain instance to process the target data packets.

[0162] In addition, after receiving the plug-in adjustment instruction, the media gateway device creates a substitute plug-in chain instance. After creation, the media gateway device can modify the value of the memory pointer to the address information or identifier of the substitute plug-in chain instance. Thus, when the media gateway device searches for a plug-in chain instance based on the memory pointer, the plug-in chain instance found becomes the substitute plug-in chain instance. Therefore, the media gateway device can call the substitute plug-in chain instance to process the target data packet.

[0163] The following describes two implementation methods for determining the target processing type and target processing order in step S202 above.

[0164] In one possible embodiment, the media gateway device stores service configuration information corresponding to a preset data source. This service configuration information records the processing type and processing order for processing media streams from the data source. Determining the target processing type and target processing order for a target media stream includes: determining the target data source for pushing the target media stream; determining the service configuration information corresponding to the target data source from the various service configuration information stored in the media gateway device; and obtaining the target processing type and target processing order recorded in the determined service configuration information. In this solution, the service configuration information stored in the media gateway device records the processing type and processing order in detail. Thus, after receiving a target media stream, the media gateway device determines the target data source for pushing the target media stream and can accurately determine the service configuration information corresponding to the target data source from the various service configuration information stored in its database. This allows it to determine the processing type and processing order recorded in that service configuration information, which are then used as the target processing type and target processing order. Subsequent steps are then executed based on the determined target processing type and target processing order, thereby improving the accuracy of media stream processing.

[0165] In another possible embodiment, when the pusher sends the target media stream to the media gateway device for the first time, it can also send a processing instruction to the media gateway device to process the target media stream. This processing instruction can carry the processing type and processing order for processing the target media stream. In this way, when the target media stream is pushed for the first time, the media gateway device can also receive the processing instruction sent by the pusher and parse it to extract the processing type and processing order for processing the target media stream, which are then used as the target processing type and target processing order, respectively.

[0166] In one possible embodiment, see Figure 5 This illustrates the structure of a media gateway device, from Figure 5 As can be seen, the media gateway device contains multiple processing modules, which work together to process the media stream.

[0167] The following section introduces the various processing modules included in a media gateway device.

[0168] 1. Media Access Module 501

[0169] The media access module 501 serves as the data entry point for the media gateway device. It is responsible for receiving media streams from different sources and protocols, parsing the media streams to obtain various data contained in the media streams, and then encapsulating the various data contained in the media streams into target data packets with a preset data format.

[0170] 2. Plugin Registration and Management Module 502

[0171] The plugin registration and management module 502 caches a plugin registry. When the media gateway device starts up, it collects the metadata of all available plugins, including the plugin name, function description, factory function, etc., to form a queryable list of plugins, i.e., the plugin registry.

[0172] In addition, the plugin registration and management module 502 also includes a plugin chain manager. The plugin chain manager is responsible for the creation, destruction, sorting and connection of plugin instances in the plugin chain instance. The plugin chain manager can create and manage an independent plugin chain instance for each media stream according to the service configuration information mentioned in the above embodiment or the processing instructions transmitted from the outside.

[0173] The plugin registration and management module 502 is also used to manage local plugin collections.

[0174] 3. Media Processing Module 503

[0175] The media processing module 503 is used to receive data packets in a preset data format output by the media access module, call the plug-in chain instance created by the plug-in chain manager to process the data packets, and output the final processed data packets.

[0176] 4. Media Output Module 504

[0177] The media output module 504 serves as the data outlet of the media gateway device, responsible for pushing the final data packets processed by the plug-in chain instance to the designated receiving end or storing them according to the specified protocol (such as RTMP or WebRTC).

[0178] The following is in conjunction with the above. Figure 5 The media stream processing method provided in the embodiments of this application will be introduced.

[0179] In one possible embodiment, see Figure 6 The flowchart of the second media stream processing method is shown, which includes the following steps S601-S605.

[0180] Step S601: Registration and initialization.

[0181] During the startup process of the media gateway device, the plugin registration and management module can scan and load all available plugins, and register the metadata of the loaded plugins into the plugin registry.

[0182] Step S602: Media stream access and plug-in chain construction.

[0183] When the media access module receives a new media stream (target media stream), it can notify the plugin chain manager. The plugin chain manager queries the registry for the metadata of the plugins needed to process the media stream based on the business configuration information corresponding to the media stream, such as JSON (JavaScript Object Notation) configuration information. The plugin manager instantiates the plugins based on the retrieved metadata, obtains plugin instances, and connects the plugin instances according to the configuration order recorded in the business configuration information to form a plugin chain instance (target plugin chain instance) for the media stream.

[0184] Step S603: Media data chain processing.

[0185] The media access module encapsulates the received target media stream into a target data packet and sends it to the media processing module. The media processing module calls the plugin chain instance for that target media stream to process the data packet and output the final data packet.

[0186] For example, the media processing module inputs the received target data packet into the first plugin instance of the plugin chain instance. After the target data packet is processed by the Process method of the first plugin instance, its return value is passed to the second plugin, and so on, until it reaches the end of the plugin chain instance.

[0187] Step S604: Media stream output

[0188] The target data packet, after being processed by the plug-in chain instance, is finally sent to the media output module, which then completes the final push or storage task.

[0189] Step S605: If the plug-in chain manager receives a plug-in adjustment instruction during the above chained processing, the plug-in chain manager performs dynamic management of the plug-in chain, specifically including the following steps S605A-S605D:

[0190] Step S605A: Lock the plugin chain instance and cache received but unprocessed data packets as well as continuously received data packets.

[0191] Locking the plug-in chain instance is as mentioned in the previous embodiments: the media gateway device suspends the processing of unprocessed target data packets by the plug-in chain instance.

[0192] Step S605B: Based on the plugin adjustment instructions, create a substitute plugin chain instance for the target plugin chain instance.

[0193] Step S605C: Replace the target plugin chain instance with the substitute plugin chain instance.

[0194] If there are remaining data packets still being processed when locking the target plugin chain instance, the plugin chain manager can wait for the remaining data packets to complete the execution of the target plugin chain instance.

[0195] After the media processing module has finished processing the remaining data packets and the substitute plugin chain instance has been created, the plugin chain manager can replace the target plugin chain instance with the substitute plugin chain instance.

[0196] Specifically, the plug-in chain manager can adjust the memory pointer to point to the substitute plug-in chain instance.

[0197] Step S605D: Unlock, return to step S603.

[0198] Unlocking here means that the media gateway device re-invokes the backup plug-in chain instance to process both cached and newly received data packets.

[0199] Corresponding to the aforementioned media stream processing method, this application also provides a media stream processing apparatus.

[0200] In one possible embodiment, see Figure 7 A schematic diagram of a media stream processing device is shown, which is applied to a media gateway device. The device includes:

[0201] The media stream encapsulation unit 701 is used to receive a target media stream and encapsulate the target media stream into a target data packet with a preset data format.

[0202] The media stream determination unit 702 is used to determine, upon first receiving the target media stream, the target processing types and the target processing order for processing the target media stream;

[0203] The plugin determination unit 703 is used to determine each target plugin that performs data processing according to each target processing type in the local plugin set, wherein the input data and output data of each plugin in the local plugin set are data packets in the preset data format;

[0204] The instance acquisition unit 704 is used to instantiate each target plugin to obtain a plugin instance, and set the data flow relationship between each plugin instance according to the target processing order to obtain a target plugin chain instance.

[0205] The first processing unit 705 is used to call the target plug-in chain instance to process the target data packet.

[0206] As can be seen from the above, when processing media streams using the solution provided in this application, when the media gateway device receives a new media stream, i.e., when it receives the target media stream for the first time, the media gateway device can determine the target processing type and target processing order for processing the target media stream. Based on the target processing type, it finds the target plugin, instantiates the target plugin to obtain a plugin instance, and sets the data flow relationship of each plugin instance according to the target processing order to obtain a target plugin chain instance. Thus, the media gateway device can call the target plugin chain instance to process the data packets of the target media stream. It is evident that, facing various media streams with diverse business requirements, the media gateway device can create customized plugin chain instances for the media stream upon first receiving it. That is, the media gateway device can flexibly arrange and combine plugins according to the business requirements of the media stream to form a customized processing pipeline. Therefore, applying the media stream processing solution provided in this application can improve the flexibility of the media gateway device in responding to diverse media stream processing needs.

[0207] In one possible embodiment, the device further includes:

[0208] The instruction receiving unit is used to, during the process of calling the target plugin chain instance to process the target data packet, if a plugin adjustment instruction is received, pause the processing of the unprocessed target data packet by the target plugin chain instance, and cache the received and unprocessed target data packets as well as the continuously received target data packets.

[0209] An instance creation unit is used to create a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instructions.

[0210] The second processing unit is used to call the substitute plug-in chain instance to process the cached data packets and the newly received target data packets after the target plug-in chain instance has finished processing the data packets that were not processed during the pause and the substitute plug-in chain instance has been successfully created.

[0211] This solution, upon receiving a plug-in adjustment command, allows the media gateway device to pause processing of unprocessed target data packets by the target plug-in chain instance. A substitute plug-in chain instance is then created according to the command, and subsequently used for data processing. The plug-in adjustment command can add, delete, or replace plug-in instances within the target plug-in chain instance. Therefore, creating a substitute plug-in chain instance based on the command can be considered as adding, deleting, or replacing plug-in instances on top of the old chain (target plug-in chain instance), resulting in a new chain (substitute plug-in chain instance). This new chain then replaces the old chain for continued data processing. Throughout the entire response process, the media gateway device can still receive media streams normally, without affecting its operation. This achieves hot-swapping of the target plug-in chain instance (i.e., enabling, disabling, and replacing plug-in instances without system downtime) and enables dynamic and smooth changes to the business requirements of the target media stream, greatly improving the flexibility and agility of the media gateway device's operating system.

[0212] In one possible embodiment, the instruction receiving unit is specifically used for:

[0213] The first thread adds a thread lock to the second thread running the target plugin chain instance to prevent the second thread from processing unprocessed target data packets. Both the first thread and the second thread belong to the media stream processing process in the media gateway device.

[0214] In one possible embodiment, the instance creation unit is specifically used for:

[0215] A substitute plugin chain instance for the target plugin chain instance is created using a third thread, according to the plugin adjustment instructions.

[0216] The first thread, the second thread, and the third thread all belong to the media stream processing process in the media gateway device. The first thread is used to add and remove thread locks for the second thread. The second thread runs the target plugin chain instance, which is used to process the target data packets.

[0217] In one possible embodiment, the second processing unit is specifically used for:

[0218] The first thread releases the thread lock of the second thread. Both the first and second threads belong to the media stream processing process in the media gateway device. When the first thread pauses the target plug-in chain instance, it adds a thread lock to the second thread that is running the target plug-in chain instance to prevent the second thread from processing unprocessed target data packets.

[0219] The second thread runs the substitute plug-in chain instance to process cached data packets and newly received target data packets.

[0220] In this solution, the first, second, and third threads all belong to the media stream processing process. Each thread can share the memory of the media stream processing process, and data can be directly transferred between threads. Compared with existing technologies that use multi-process or microservice architectures to achieve functional decoupling, which split different functions into independent services and introduce additional inter-process or network communication overhead during media stream processing, increasing the burden of data serialization / deserialization, this solution has no performance loss from cross-process communication and data copying, greatly reducing the latency of real-time media stream processing and ensuring low latency and high efficiency in real-time media stream processing. Therefore, this solution can meet the stringent latency requirements of real-time media stream processing.

[0221] In one possible embodiment, the first processing unit 705 is specifically used for:

[0222] Set the memory pointer to point to the target plugin chain instance;

[0223] The target data packet is processed by calling the instance pointed to by the memory pointer;

[0224] The second processing unit is specifically used for:

[0225] Adjust the memory pointer to point to the substitute plug-in chain instance;

[0226] The instance pointed to by the memory pointer is invoked to process the cached data packets and the newly received target data packets.

[0227] In this scheme, the memory pointer is used to specify the plug-in chain instance invoked by the media gateway device. When no plug-in adjustment instruction is received, the memory pointer points to the target plug-in chain instance. Therefore, the media gateway device can invoke the target plug-in chain instance to process the target data packet. After receiving the plug-in adjustment instruction, the memory pointer is adjusted to point to the substitute plug-in chain instance, thereby changing the plug-in chain instance invoked by the media gateway device to the substitute plug-in chain instance. It can be seen that by controlling the direction of the memory pointer, the media gateway device can invoke the correct plug-in chain instance to process the target data packet, thereby improving the accuracy of processing the target data packet and thus improving the accuracy of processing the media stream.

[0228] In one possible embodiment, the media gateway device stores service configuration information corresponding to a preset data source, and the service configuration information records the processing type and processing order for processing the media stream of the data source;

[0229] The media stream determination unit 702 is specifically used for:

[0230] Determine the target data source for pushing the target media stream;

[0231] Among the service configuration information stored in the media gateway device, determine the service configuration information corresponding to the target data source;

[0232] Obtain the target processing type and target processing order recorded in the determined business configuration information.

[0233] In this solution, the service configuration information stored in the media gateway device contains detailed records of processing types and processing sequences. When the media gateway device receives a target media stream, it determines the target data source from which the stream is pushed. It can then accurately identify the corresponding service configuration information from its stored service configuration information, thereby determining the processing type and processing sequence recorded in that service configuration information. These are then used as the target processing type and target processing sequence. Subsequent steps are then executed based on the determined target processing type and target processing sequence, improving the accuracy of media stream processing.

[0234] This invention also provides an electronic device, such as... Figure 8 As shown, it includes a processor 801, a communication interface 802, a memory 803, and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804.

[0235] Memory 803 is used to store computer programs;

[0236] The processor 801, when executing a program stored in the memory 803, implements the steps of any of the above media stream processing methods.

[0237] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0238] The communication interface is used for communication between the aforementioned electronic devices and other devices.

[0239] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0240] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0241] In another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, and when the computer program is executed by a processor, it implements the steps of any of the media stream processing methods described above.

[0242] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to perform any of the media stream processing methods described above.

[0243] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0244] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0245] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0246] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A media stream processing method, characterized by, Applied to a media gateway device, the method includes: Receive the target media stream and encapsulate the target media stream into a target data packet with a preset data format; Upon first receiving the target media stream, determine the target processing types and the target processing order for processing the target media stream; In the local plugin set, each target plugin is determined to process data according to each target processing type, wherein the input data and output data of each plugin in the local plugin set are data packets in the preset data format; Each target plugin is instantiated to obtain a plugin instance. According to the target processing order, the data flow relationship between each plugin instance is set to obtain a target plugin chain instance. The target plugin chain instance is invoked to process the target data packet.

2. The method of claim 1, wherein, During the process of calling the target plugin chain instance to process the target data packet, if a plugin adjustment instruction is received, the method further includes the following steps: The target plugin chain instance is paused from processing unprocessed target data packets, and received but unprocessed target data packets, as well as continuously received target data packets, are cached. According to the plugin adjustment instructions, create a substitute plugin chain instance for the target plugin chain instance; After the target plug-in chain instance finishes processing the data packets that were not processed during the pause and the substitute plug-in chain instance is successfully created, the substitute plug-in chain instance is invoked to process the cached data packets and the newly received target data packets.

3. The method of claim 2, wherein, The step of pausing the target plugin chain instance to process unprocessed target data packets includes: The first thread adds a thread lock to the second thread running the target plugin chain instance to prevent the second thread from processing unprocessed target data packets. Both the first thread and the second thread belong to the media stream processing process in the media gateway device.

4. The method of claim 2, wherein, The step of creating a substitute plugin chain instance for the target plugin chain instance according to the plugin adjustment instructions includes: A substitute plugin chain instance for the target plugin chain instance is created using a third thread, according to the plugin adjustment instructions. The first thread, the second thread, and the third thread all belong to the media stream processing process in the media gateway device. The first thread is used to add and remove thread locks for the second thread. The second thread runs the target plugin chain instance and is used to process the target data packets.

5. The method according to claim 2, characterized in that, The process of calling the substitute plugin chain instance to process cached data packets and newly received target data packets includes: The first thread releases the thread lock of the second thread. Both the first and second threads belong to the media stream processing process in the media gateway device. When the first thread pauses the target plug-in chain instance, it adds a thread lock to the second thread that is running the target plug-in chain instance to prevent the second thread from processing unprocessed target data packets. The second thread runs the substitute plug-in chain instance to process cached data packets and newly received target data packets.

6. The method according to claim 2, characterized in that, The step of invoking the target plugin chain instance to process the target data packet includes: Set the memory pointer to point to the target plugin chain instance; The target data packet is processed by calling the instance pointed to by the memory pointer; The process of calling the substitute plugin chain instance to process cached data packets and newly received target data packets includes: Adjust the memory pointer to point to the substitute plug-in chain instance; The instance pointed to by the memory pointer is invoked to process the cached data packets and the newly received target data packets.

7. The method according to any one of claims 1-6, characterized in that, The media gateway device stores service configuration information corresponding to a preset data source. The service configuration information records the processing type and processing order for processing the media stream of the data source. The determination of the target processing type and target processing order for processing the target media stream includes: Determine the target data source for pushing the target media stream; Among the service configuration information stored in the media gateway device, determine the service configuration information corresponding to the target data source; Obtain the target processing type and target processing order recorded in the determined business configuration information.

8. A media stream processing device, characterized in that, Applied to a media gateway device, the device includes: A media stream encapsulation unit is used to receive a target media stream and encapsulate the target media stream into a target data packet with a preset data format. The media stream determination unit is used to determine, upon first receiving the target media stream, the target processing types and the target processing order for processing the target media stream; The plugin determination unit is used to determine each target plugin in the local plugin set that performs data processing according to each target processing type, wherein the input data and output data of each plugin in the local plugin set are data packets in the preset data format; The instance acquisition unit is used to instantiate each target plugin to obtain a plugin instance, and set the data flow relationship between each plugin instance according to the target processing order to obtain a target plugin chain instance; The first processing unit is used to call the target plugin chain instance to process the target data packet.

9. An electronic device, characterized in that, It includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method described in any one of claims 1-7.

Images