Media data processing method and apparatus, computer device, and readable storage medium

Abstract

The application relates to a media data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a media data stream pushed by a collection device, processing the media data stream based on multiple data processing steps and an execution sequence; obtaining a media data processing result of a previous data processing step according to the execution sequence; traversing multiple local plugins corresponding to a current data processing step in a local plugin library, processing the media data processing result by using a current local plugin, if the multiple local plugins all fail to process, querying and loading a data processing plugin corresponding to the current data processing step from a cloud plugin library, processing the media data processing result by using the data processing plugin, obtaining a media data processing result of the current data processing step, and sending the media data processing result corresponding to a last executed data processing step to a client. The method can improve the media data processing efficiency.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a media data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of audio and video processing technology, audio and video surveillance equipment has been widely used. Due to differences in surveillance equipment manufacturers and firmware versions, the audio and video stream data formats pushed from the device to the server vary widely, and client requirements for audio and video formats also differ. Therefore, the main process on the server side needs to implement numerous compatibility measures to accommodate the differences in audio and video formats pushed by the device. However, this approach requires the redevelopment of a compatibility service program for each new device, resulting in a bloated and difficult-to-maintain server-side service program, leading to low media data processing efficiency. Summary of the Invention

[0003] Therefore, it is necessary to provide a media data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product that can improve the efficiency of media data processing in response to the above-mentioned technical problems.

[0004] Firstly, this application provides a media data processing method, including:

[0005] Receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols;

[0006] Obtain multiple preset data processing steps and the execution order of each data processing step; process the media data stream based on the multiple data processing steps and the execution order;

[0007] For each data processing step, the media data processing result of the previous data processing step is obtained according to the execution order. Multiple local plugins corresponding to the current data processing step are traversed in the local plugin library. For each local plugin encountered, the media data processing result is processed using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step. If multiple local plugins fail to process the data, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded. The data processing plugin is then used to process the media data processing result to obtain the media data processing result of the current data processing step.

[0008] The media data processing results corresponding to the last data processing step are sent to the client.

[0009] In one embodiment, querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library includes:

[0010] Obtain error messages generated by the processing of media data by multiple local plugins;

[0011] Generate error characteristic values ​​based on multiple error messages;

[0012] Based on the error characteristic value, query the cloud plugin library and load the data processing plugin corresponding to the current data processing step.

[0013] In one embodiment, based on the error characteristic value, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded, including:

[0014] Search the cloud plugin library for multiple cloud plugins corresponding to the current data processing step;

[0015] If any cloud plugin's feature value matches the error feature value, then the cloud plugin is used as the data processing plugin corresponding to the current data processing step; the cloud plugin is downloaded from the cloud plugin library and installed locally.

[0016] In one embodiment, traversing multiple local plugins corresponding to the current data processing step in the local plugin library includes:

[0017] Get the number of successful executions of each of the local plugins corresponding to the current data processing step in the local plugin library;

[0018] Sort the local plugins according to the number of successful executions, and then iterate through each local plugin according to the sorting results.

[0019] In one embodiment, the media data processing method further includes:

[0020] Get the number of times each data processing plugin has been processed and the number of times it has succeeded;

[0021] The success rate of each data processing plugin is determined based on the ratio of the number of successful attempts to the number of processing attempts.

[0022] If the success rate of any data processing plugin exceeds the preset success rate, the data processing plugin will be pre-loaded locally.

[0023] In one embodiment, the media data processing method further includes:

[0024] Obtain the device model of the target device deployed in the target environment;

[0025] If the target device's model matches the acquisition device's model, then the corresponding data processing plugins for the acquisition device will be loaded from the cloud plugin library into the target environment to process the media data stream pushed by the target device.

[0026] Secondly, this application also provides a media data processing apparatus, comprising:

[0027] The receiving module is used to receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols;

[0028] The first processing module is used to obtain multiple preset data processing steps and the execution order of each data processing step; and to process the media data stream based on the multiple data processing steps and the execution order.

[0029] The second processing module is used to obtain the media data processing result of the previous data processing step according to the execution order for each data processing step; it iterates through multiple local plugins corresponding to the current data processing step in the local plugin library, and processes the media data processing result using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step; if multiple local plugins fail to process, the module queries and loads the data processing plugin corresponding to the current data processing step from the cloud plugin library, and processes the media data processing result using the data processing plugin to obtain the media data processing result of the current data processing step.

[0030] The sending module is used to send the media data processing results corresponding to the last data processing step to the client.

[0031] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0032] Receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols;

[0033] Obtain multiple preset data processing steps and the execution order of each data processing step; process the media data stream based on the multiple data processing steps and the execution order;

[0034] For each data processing step, the media data processing result of the previous data processing step is obtained according to the execution order. Multiple local plugins corresponding to the current data processing step are traversed in the local plugin library. For each local plugin encountered, the media data processing result is processed using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step. If multiple local plugins fail to process the data, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded. The data processing plugin is then used to process the media data processing result to obtain the media data processing result of the current data processing step.

[0035] The media data processing results corresponding to the last data processing step are sent to the client.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0037] Receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols;

[0038] Obtain multiple preset data processing steps and the execution order of each data processing step; process the media data stream based on the multiple data processing steps and the execution order;

[0039] For each data processing step, the media data processing result of the previous data processing step is obtained according to the execution order. Multiple local plugins corresponding to the current data processing step are traversed in the local plugin library. For each local plugin encountered, the media data processing result is processed using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step. If multiple local plugins fail to process the data, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded. The data processing plugin is then used to process the media data processing result to obtain the media data processing result of the current data processing step.

[0040] The media data processing results corresponding to the last data processing step are sent to the client.

[0041] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0042] Receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols;

[0043] Obtain multiple preset data processing steps and the execution order of each data processing step; process the media data stream based on the multiple data processing steps and the execution order;

[0044] For each data processing step, the media data processing result of the previous data processing step is obtained according to the execution order. Multiple local plugins corresponding to the current data processing step are traversed in the local plugin library. For each local plugin encountered, the media data processing result is processed using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step. If multiple local plugins fail to process the data, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded. The data processing plugin is then used to process the media data processing result to obtain the media data processing result of the current data processing step.

[0045] The media data processing results corresponding to the last data processing step are sent to the client.

[0046] The aforementioned media data processing methods, apparatus, computer equipment, computer-readable storage media, and computer program products process media data streams through multiple data processing steps with a preset execution order. Each data processing step is encapsulated as a small-granularity plugin and stored in a local plugin library and a cloud plugin library. For a single client, the client and cloud work together, and the local plugin library does not need to load all plugins, reducing resource consumption. When faced with media data streams pushed by different acquisition devices with different encapsulation protocols, transmission protocols, and preset media formats, when it is necessary to adapt to new media data formats, compared to the traditional method of developing compatible programs, the plugin-based management method only requires loading the new plugin without modifying the existing plugins. The logic and plugins are highly versatile, easier to maintain and port, and greatly improve media data processing efficiency. They process media data streams based on multiple data processing steps and execution order. When the current data processing step is reached, the local plugins in the local plugin library are traversed first, and the local plugins are used to process the media data processing results of the previous step, which helps improve media data processing efficiency. When multiple local plugins fail, the corresponding data processing plugin can be queried and loaded from the cloud plugin library. This combination of local and cloud-based approaches ensures that each data processing step succeeds by dynamically loading cloud plugins when local plugins cannot process the media data, further improving media data processing efficiency. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is an application environment diagram of a media data processing method in one embodiment;

[0049] Figure 2 This is a flowchart illustrating a media data processing method in one embodiment;

[0050] Figure 3 This is a flowchart illustrating a media data processing method in another embodiment;

[0051] Figure 4 This is a schematic diagram of a cloud-based plugin library in one embodiment;

[0052] Figure 5 This is a schematic diagram of the overall flow of a media data processing method in one embodiment;

[0053] Figure 6 This is a structural block diagram of a media data processing device in one embodiment;

[0054] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0056] With the development of audio and video processing technology, audio and video surveillance equipment has been widely used. The video surveillance network system service based on the GB / T28181 protocol is abbreviated as GBS. For each user, a GBS server, client, and at least one audio and video surveillance device are typically deployed. The GBS server acts as the server-side, and at least one audio and video surveillance device acts as the device-side. The server-side processes the media data stream received from the device-side and sends it to the client. Due to differences in surveillance equipment manufacturers and firmware versions, the audio and video stream data formats pushed from the device-side to the server-side vary widely, and client requirements for audio and video formats also differ. Therefore, the main process on the server-side requires numerous compatibility measures to meet the differences in audio and video formats pushed by the device-side. However, this method requires a restart to update the compatibility service program after each program modification, increasing program downtime and impacting the user experience. Meanwhile, different clients often require different audio and video formats, such as FLV (Flash Video), HLS (HTTP Live Streaming), and WebRTC (Web Real-Time Communications). However, for clients working on the same project, the required formats are usually limited. Providing services for all audio and video formats on the server side would consume significant service resources. Furthermore, each new device requires a newly developed compatibility service program, or a patch is needed for each problem encountered. This makes the server-side service program bloated and difficult to maintain. Moreover, maintaining compatibility with various audio and video formats in the main workflow is problematic, as any change affects the entire system, hindering development, maintenance, and testing, and ultimately leading to low media data processing efficiency.

[0057] Based on this, this application proposes a media data processing method. The server receives media data streams pushed by acquisition devices and processes the media data streams based on multiple preset data processing steps and execution order. The server adopts a plug-in management approach, encapsulating each data processing step into small-granular plug-ins and storing them in a local plug-in library and a cloud plug-in library. For a single client, the client and cloud work together, and the local plug-in library does not need to load all plug-ins, reducing resource consumption. When dealing with media data streams pushed by different acquisition devices with different encapsulation protocols, transmission protocols, and preset media formats, when adapting to new media data formats, compared to traditional methods of developing compatible programs, the plug-in management method only requires loading new plug-ins without modifying the logic of existing plug-ins. The plug-ins have high versatility, are easier to maintain and port, and greatly improve media data processing. Efficiency: Media data streams are processed based on multiple data processing steps and execution order. When the current data processing step is executed, local plugins in the local plugin library are traversed first, and the media data processing results of the previous data processing step are processed using local plugins, which helps improve media data processing efficiency. When multiple local plugins fail to process the media data, the corresponding data processing plugin can be queried and loaded from the cloud plugin library. This combination of local and cloud methods ensures that each data processing step is successfully processed by dynamically loading cloud plugins when local plugins cannot process the media data, which further improves media data processing efficiency. The server sends the media data processing results corresponding to the last executed data processing step to the client. Since each data processing step is preset, it can be flexibly adjusted according to customer needs, thus ensuring that the media data sent to the client meets the customer's requirements.

[0058] The media data processing method provided in this application embodiment can be applied to, for example... Figure 1The application environment shown includes a server 102, a client 104, and a device 106 deployed in the client environment. The server 102 can be a GBS server, and the device 106 can include at least one data acquisition device. Server 102 receives media data streams pushed by the acquisition device. The media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols. It acquires multiple preset data processing steps and their execution order. Based on these steps and the execution order, it processes the media data streams. For each data processing step, it obtains the media data processing result of the previous step according to the execution order. It iterates through multiple local plugins corresponding to the current data processing step in the local plugin library. For each local plugin encountered, it processes the media data processing result using the current plugin. If the processing is successful, the corresponding result is used as the media data processing result of the current step. If multiple local plugins fail, it queries and loads the data processing plugin corresponding to the current step from the cloud plugin library, processes the media data processing result using the plugin, and obtains the media data processing result of the current step. Finally, it sends the media data processing result corresponding to the last executed data processing step to the client 104. Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. Client 104 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle systems, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Head-mounted devices can include virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc.

[0059] In one exemplary embodiment, such as Figure 2 As shown, a media data processing method is provided, which can be applied to... Figure 1 Taking server-side 102 as an example, the explanation includes the following steps 202 to 208. Wherein:

[0060] Step 202: Receive the media data stream pushed by the acquisition device; the media data stream is obtained by processing raw stream data of a preset media format using a preset encapsulation protocol and transmission protocol.

[0061] In this context, the acquisition device refers to an audio / video acquisition device used to acquire media data streams. After acquiring the media data streams, the acquisition device pushes them to the server. The media data streams received by the server can include audio or video data streams. Multiple acquisition devices can be deployed in the customer environment, and the media data processing method proposed in this application can be used to process the media data streams pushed by each acquisition device.

[0062] Raw stream data refers to the data directly output after being encoded by an encoder in the field of audio and video coding. Raw stream data is not encapsulated by any transmission protocol and has a specific data structure and composition. The preset media format refers to the data format of the raw stream data. For example, it can be H.264 (a video compression standard), H.265 (a video compression standard), AAC (Advanced Audio Coding), G.711 (an audio coding method developed by the International Telecommunication Union), and other data formats.

[0063] Encapsulation protocols are protocols that encapsulate raw streaming data. They define the format and structure of data packets so that they can be correctly parsed and processed during transmission and storage. For example, the PS (Program Stream) encapsulation protocol.

[0064] A transport protocol refers to the protocol used to encapsulate raw stream data for transmission. For example, it could be the RTP (Real-Time Transport Protocol).

[0065] For example, the media data stream is obtained by encapsulating raw H264 format data using the PS encapsulation protocol, and then further encapsulating the encapsulation result using the RTP protocol.

[0066] Step 204: Obtain multiple preset data processing steps and the execution order of each data processing step; process the media data stream based on the multiple data processing steps and the execution order.

[0067] Since the media data stream is obtained by processing the raw stream data of the preset media format through the preset encapsulation protocol and transmission protocol, the media data stream needs to be parsed and encapsulated into the data format specified by the client in order to be recognized by the client.

[0068] Data processing steps refer to the steps of protocol parsing and transcapsulation of media data streams. These steps are pre-defined on the server side. For example, for acquisition devices based on the GB / T28181 (China's standard for public security video surveillance system) protocol standard, a unified data processing procedure can be used for media data streams, which is beneficial for the development, maintenance, and testing of server-side programs. Exemplarily, the data processing steps include four steps: RTP packet assembly and framing, PS protocol parsing, raw stream data parsing, and transcapsulation. Each data processing step may also include at least one sub-step, and these sub-steps also have an execution order.

[0069] The execution order refers to the order in which multiple data processing steps are executed. When the server processes the media data stream, it executes each data processing step in the order of execution, which can process the media data stream into the data format required by the client.

[0070] like Figure 3 The diagram shows a flow chart of a media data processing method in another embodiment. For the media data stream pushed by the acquisition device, the server first performs RTP packet assembly and framing on the media data stream, then performs PS protocol parsing on the RTP packet assembly and framing results, followed by raw stream data parsing on the PS protocol parsing results, then encapsulates the raw stream data parsing results, and finally sends the encapsulated results to the client.

[0071] Step 206: For each data processing step, obtain the media data processing result of the previous data processing step according to the execution order; traverse multiple local plugins corresponding to the current data processing step in the local plugin library, and process the media data processing result using the current local plugin for each traversed local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step; if multiple local plugins fail to process, query and load the data processing plugin corresponding to the current data processing step from the cloud plugin library, and process the media data processing result using the data processing plugin to obtain the media data processing result of the current data processing step.

[0072] The media data processing result refers to the result obtained after each data processing step is executed. Since each data processing step has an execution order, the first data processing step is used to process the media data stream, and for subsequent data processing steps, the current data processing step is used to process the media data processing result of the previous data processing step.

[0073] A local plugin repository refers to the storage space on the server side used to store plugins. A cloud plugin repository refers to the storage space on a cloud server used to store plugins. The local plugin repository stores a subset of commonly used plugins corresponding to each of the multiple data processing steps, while the cloud server stores the full set of plugins corresponding to each of the multiple data processing steps. For example... Figure 4 The diagram illustrates a cloud-based plugin library in one embodiment. This library includes multiple plugin libraries: an RTP packet assembly plugin library, an audio / video format parsing plugin library, and a container format plugin library. Each library contains multiple plugins. Specifically, the RTP packet assembly plugin library corresponds to the RTP packet assembly and frame processing step; the audio / video format parsing plugin library corresponds to the raw stream data parsing step; and the container format plugin library corresponds to the transcoding step. The server can determine the data processing plugin for each data processing step through a query. Furthermore, the cloud-based plugin library supports both horizontal and vertical expansion. Horizontal expansion means that each plugin library can be expanded with more plugins, while vertical expansion means that the cloud-based plugin library can add more plugin libraries corresponding to data processing steps. Expanding the cloud-based plugin library does not require modifying the logic of existing plugins; new plugins are added directly, conforming to the open / closed principle.

[0074] The following explanation uses an example where the current data processing step is not the first one. The server first iterates through the local plugins corresponding to the current data processing step in the local plugin library. If the current local plugin successfully processes the media data, its result is used as the media data processing result for the current step. If processing fails, the iteration continues. If multiple local plugins fail, the server queries and loads the data processing plugin corresponding to the current step from the cloud plugin library. The result obtained after this plugin processes the media data is the media data processing result for the current step.

[0075] Step 208: Send the media data processing result corresponding to the last executed data processing step to the client.

[0076] The media data stream is processed sequentially by the various plugins retrieved from the local and cloud plugin libraries. The final media data processing result is the media data that meets the client's format requirements. The server then sends this media data processing result to the client so that the client can directly process the received media data.

[0077] In the aforementioned media data processing method, the media data stream is processed through multiple data processing steps with a pre-defined execution order. Each data processing step is encapsulated into a small-granularity plugin and stored in both a local plugin library and a cloud plugin library. For a single client, the client and cloud work together, and the local plugin library does not need to load all plugins, reducing resource consumption. When dealing with media data streams pushed by different acquisition devices with different encapsulation protocols, transmission protocols, and pre-defined media formats, and when new media data formats need to be adapted, compared to the traditional method of developing compatible programs, the plugin-based management method only requires loading the new plugin without modifying the logic of the original plugins. The plugins have high versatility. This approach is easier to maintain and port, significantly improving media data processing efficiency. It processes media data streams based on multiple data processing steps and their execution order. When the current data processing step is reached, it first iterates through local plugins in the local plugin library and uses these local plugins to process the media data processing results from the previous step, thus improving efficiency. If multiple local plugins fail, the corresponding data processing plugin can be queried and loaded from the cloud plugin library. This combination of local and cloud-based methods ensures successful processing of each data processing step by dynamically loading cloud plugins when local plugins are unable to process the media data, further enhancing media data processing efficiency.

[0078] In one exemplary embodiment, querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library includes: obtaining error information generated by multiple local plugins processing the media data processing results; generating error feature values ​​based on the multiple error information; and querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library based on the error feature values.

[0079] Error messages indicate errors in the local plugin's handling of media data. These messages typically contain unparsed or unrecognized fields, allowing the server to identify the correct data processing plugin.

[0080] Error signatures refer to the characteristic values ​​corresponding to error messages. The server stores the mapping relationship between error messages and error signatures. The relationship between error messages and error signatures can be one-to-one or many-to-one. The server determines the error signature values ​​corresponding to multiple error messages by querying this mapping relationship.

[0081] For example, the error message contains the fields "format" and "H264". The corresponding error characteristic value is 264, indicating that the media data stream is in H264 format. The server can use the parsing plugin for H264 format in the cloud plugin library as the data processing plugin for the current data processing step.

[0082] refer to Figure 4 In the cloud-based plugin library, each data processing step corresponds to multiple plugins, and each plugin has its own corresponding characteristic values. After determining the error characteristic values, the server can query and load the data processing plugin corresponding to the current data processing step from the cloud-based plugin library based on the error characteristic values ​​and the characteristic values ​​corresponding to each plugin.

[0083] In this embodiment, error feature values ​​are generated based on the error information generated when the local plugin is executed. The corresponding data processing plugin is then retrieved from the cloud plugin library based on the error feature values. Since the error information indicates the error situation of the local plugin, this method of querying data processing plugins based on error information is beneficial for retrieving accurate plugins and ensuring that the retrieved data processing plugins can successfully process the media data processing results.

[0084] In an exemplary embodiment, based on the error characteristic value, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded, including: querying multiple cloud plugins corresponding to the current data processing step in the cloud plugin library; if the characteristic value of any cloud plugin is consistent with the error characteristic value, then the cloud plugin is used as the data processing plugin corresponding to the current data processing step; downloading the cloud plugin from the cloud plugin library and installing it locally.

[0085] Here, "cloud plugin" refers to the plugin corresponding to the current data processing step in the cloud plugin library. Each data processing step corresponds to at least one plugin in the cloud plugin library. When there are multiple cloud plugins corresponding to the current data processing step, it is necessary to further identify the corresponding data processing plugin from among the multiple cloud plugins.

[0086] The terminal compares the error feature value with the feature values ​​corresponding to each of the multiple cloud plugins. If the feature value corresponding to any cloud plugin matches the error feature value, the server can use that cloud plugin as the data processing plugin for the current data processing step.

[0087] After the server finds the data processing plugin, it downloads the corresponding cloud plugin from the cloud plugin library and installs it locally on the server.

[0088] In some embodiments, the server can also save the downloaded cloud plugin to a local plugin library, allowing the local plugin library to be dynamically updated as needed. Once the cloud plugin is saved in the local plugin library, subsequent media data streams that can be successfully processed by the cloud plugin can be directly processed using plugins from the local plugin library, which helps improve media data processing efficiency.

[0089] In other embodiments, if the feature values ​​of multiple cloud plugins are inconsistent with the error feature value, it indicates that there is no cloud plugin in the cloud plugin library that can process the media data processing results. The server generates a first alarm message, which is used to remind the developer that the cloud plugin library is missing the plugin corresponding to the error feature value.

[0090] In this embodiment, the error feature value is compared with the feature values ​​of multiple cloud plugins corresponding to the current data processing step in the cloud plugin library. If they match, the corresponding cloud plugin is determined to be the data processing plugin for the current data processing step. Since the feature value of the queried cloud plugin matches the error feature value, it indicates that the feature value of the cloud plugin matches the fields contained in the error information of the local plugin. Therefore, using this cloud plugin as the data processing plugin for the current data processing step can ensure the successful processing of media data. In addition, downloading and installing the cloud plugin locally is beneficial to improving the efficiency of media data processing in subsequent processing.

[0091] In an exemplary embodiment, traversing multiple local plugins corresponding to the current data processing step in the local plugin library includes: obtaining the number of successful executions of each of the multiple local plugins corresponding to the current data processing step in the local plugin library; sorting the multiple local plugins according to the number of successful executions; and traversing each local plugin according to the sorting result.

[0092] The local plugin processes the media data, which may succeed or fail. The successful execution count refers to the number of times a local plugin has successfully executed. Each local plugin has its own successful execution count, which is updated in real time.

[0093] In some embodiments, if any local plugin processes successfully, the success count of that local plugin is increased by a preset value to achieve real-time updates of the success count. For example, the preset value can be 1.

[0094] When iterating through the local plugin library on the server side, the system can obtain the number of successful executions for each of the multiple local plugins corresponding to the current data processing step, and then sort the plugins according to their number of successful executions. For example, the sorting result refers to the result after sorting the plugins, such as sorting them from largest to smallest number of successful executions. The server can then iterate through each local plugin one by one according to the sorting result.

[0095] In this embodiment, for the multiple local plugins corresponding to the current data processing step in the local plugin library, each local plugin is sorted according to the number of successful executions of each plugin. The more successful executions, the more frequently the plugin is used. Traversing according to the sorting results helps to quickly find the local plugins that can successfully process the media data processing results, thereby improving the efficiency of media data processing.

[0096] In an exemplary embodiment, the media data processing method further includes: obtaining the number of processing attempts and the number of successful attempts for each data processing plugin; determining the processing success rate of each data processing plugin based on the ratio of the number of successful attempts to the number of processing attempts; and preloading the data processing plugin locally if the processing success rate of any data processing plugin exceeds a preset success rate.

[0097] The processing count refers to the total number of times the data processing plugin processes the media data processing results or media data stream. The success count refers to the number of times a process was successfully completed. The server updates the processing count after each data processing plugin completes its processing, and updates the success count after any data processing plugin successfully completes its processing.

[0098] The media data processing method also includes: if any data processing plugin succeeds, increasing the success count for that plugin by a preset value; if any data processing plugin fails, generating a second alarm and sending it to the cloud server. The second alarm is used to alert developers that the cloud plugin corresponding to that data processing plugin has a defect and needs to be fixed promptly.

[0099] The server uses the ratio of successful attempts to processing attempts as the success rate of the corresponding data processing plugin. For data processing plugins with a success rate exceeding the preset success rate, the plugin can be pre-loaded locally on the server.

[0100] In some embodiments, after querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library, the method further includes: obtaining an associated plugin that is related to the data processing plugin on the server side; downloading the associated plugin from the cloud plugin library and installing the associated plugin locally.

[0101] The correlation refers to the probability that subsequent data processing steps will use the corresponding associated plugins after the current data processing step has been used, exceeding a preset probability. For example, if plugin A is used in step one, the probability that subsequent data processing steps will use plugin B in step two, plugin C in step three, and plugin D in step four will exceed a preset probability. Therefore, when querying and loading plugin A from the cloud plugin library, plugins from other steps can be loaded locally to avoid frequent queries to the cloud and reduce bandwidth usage.

[0102] In this embodiment, the processing success rate is determined by the number of processing attempts and the number of successful attempts for each data processing plugin. For data processing plugins with a processing success rate exceeding the preset success rate, it indicates that the data processing plugin is frequently used. The data processing plugin is pre-loaded locally, and the local data processing plugin can be directly used in subsequent data processing, avoiding frequent queries to the cloud server, reducing bandwidth, and improving the efficiency of media data processing.

[0103] In an exemplary embodiment, the media data processing method further includes: obtaining the device model of the target device deployed in the target environment; if the device model of the target device is consistent with the device model of the acquisition device, then loading each data processing plugin corresponding to the acquisition device from the cloud plugin library to the target environment for processing the media data stream pushed by the target device.

[0104] The target environment refers to an environment different from the current customer environment. The target device is the media data acquisition device deployed in the target environment. Separate client and server components are deployed in the target environment.

[0105] Since different environments may use the same protocol standard, the data processing steps for media data streams are the same in different environments. For a customer environment that has been deployed, if the target environment has deployed the same target device, the deployment of the customer environment can be copied to the target environment, thereby improving the deployment efficiency of the target environment.

[0106] Specifically, the server obtains the device model of the target device and determines whether a data acquisition device with the same model exists in the customer's environment. If the device model of any data acquisition device matches that of the target device, the corresponding data processing plugins for that data acquisition device can be loaded from the cloud plugin library into the target environment. The data processing plugins pre-loaded into the target environment can then be used to process the media data pushed by the target device.

[0107] In this embodiment, when the device model of the target device deployed in the target environment is consistent with the device model of the acquisition device, the data processing plugins corresponding to the acquisition device are pre-loaded into the target environment from the cloud plugin library. In this way, the target environment is pre-loaded with data processing plugins that can process the media data streams acquired by the target device, which can avoid frequent queries to the cloud server for plugins, reduce bandwidth, and improve the deployment efficiency of the target environment and the processing efficiency of the target environment for media data.

[0108] To illustrate the media data processing method and its effects in this solution in detail, a specific embodiment is described below:

[0109] This explanation uses a client environment with a GBS server, at least one acquisition device, and a client as an example. The acquisition device is used to acquire media data streams and push them to the server. The media data stream is obtained by processing raw stream data in a preset media format using preset encapsulation and transport protocols. For example, the media data stream is obtained by encapsulating raw H.264 format data using the PS encapsulation protocol, and then further encapsulating the encapsulation result using the RTP protocol.

[0110] The server receives the media data stream pushed by the acquisition device and processes it based on multiple preset data processing steps and the execution order of these steps. For example... Figure 5 The diagram shown is a schematic representation of the overall flow of a media data processing method in one embodiment.

[0111] For data processing step n, the media data processing result of the previous data processing step n-1 is obtained according to the execution order. Multiple local plugins corresponding to the current data processing step are traversed in the pre-installed local plugin library. For the currently traversed local plugin, the media data processing result is processed using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step, and the success count is incremented by 1. Specifically, during the local plugin traversal, the success count of each local plugin corresponding to the current data processing step in the local plugin library is obtained. The local plugins are sorted according to the success count, and the process is then traversed according to the sorting result.

[0112] If multiple local plugins fail to process the data, the system queries and loads the data processing plugin corresponding to the current data processing step from the cloud plugin library. Specifically, based on the error information generated by each local plugin in processing the media data, an error characteristic value is generated from the multiple error information. The system then queries the cloud plugin library for the cloud plugins corresponding to the current data processing step to determine if the cloud plugin library contains such a plugin. If the characteristic value of any cloud plugin matches the error characteristic value, the cloud plugin is designated as the data processing plugin for the current data processing step, downloaded from the cloud plugin library, and installed locally to update the local plugin library. If the cloud plugin library does not contain a cloud plugin with a characteristic value matching the error characteristic value, a first alarm is generated to alert the developers that the plugin corresponding to the error characteristic value is missing from the cloud plugin library.

[0113] After loading the data processing plugin from the cloud plugin library, the data processing plugin is used to process the media data processing results. If the processing is successful, the media data processing result of the current data processing step is obtained; if the processing fails, a second alarm is generated. The second alarm is used to remind the developers that the cloud plugin corresponding to the data processing plugin has defects and needs to be corrected in time.

[0114] After data processing step n is completed, the process proceeds to data processing step n+1, until all data processing steps are completed. The media data processing result corresponding to the last executed data processing step is then sent to the client, allowing the client to directly process the received media data processing result.

[0115] In some embodiments, the server can obtain the number of processing attempts and the number of successful attempts for each data processing plugin, and determine the processing success rate of each data processing plugin based on the ratio of the number of successful attempts to the number of processing attempts. If the processing success rate of any data processing plugin exceeds the preset success rate, the data processing plugin is pre-loaded locally.

[0116] In other embodiments, the server can obtain the device model of the target device deployed in the target environment. If the device model of the target device is the same as the device model of the acquisition device, the data processing plugins corresponding to the acquisition device are loaded from the cloud plugin library to the target environment for processing the media data stream pushed by the target device.

[0117] The aforementioned media data processing method processes media data streams through multiple pre-defined data processing steps with an execution order. Each data processing step is encapsulated as a small-granular plugin and stored in both a local plugin library and a cloud plugin library. For a single client, the client and cloud work together, eliminating the need to load all plugins in the local plugin library, thus reducing resource consumption. When dealing with media data streams pushed by different acquisition devices with different encapsulation protocols, transmission protocols, and pre-defined media formats, and when new media data formats need to be adapted, compared to traditional compatibility development methods, the plugin-based management method only requires loading the new plugin without modifying the logic of existing plugins. This results in higher plugin versatility. Easier to maintain and port, significantly improving media data processing efficiency; processing media data streams based on multiple data processing steps and execution order. When executing the current data processing step, it first traverses the local plugin library and uses the local plugin to process the media data processing results of the previous step, which helps improve media data processing efficiency; when multiple local plugins fail, the corresponding data processing plugin can be queried and loaded from the cloud plugin library. This combination of local and cloud methods ensures that each data processing step succeeds by dynamically loading cloud plugins when local plugins cannot process the media data, further improving media data processing efficiency. Simultaneously, it achieves linkage between the server-side and cloud servers. The server-side can dynamically load and unload plugins according to actual conditions, reducing resource consumption. Furthermore, pre-loading plugins based on their processing success rate and inter-plugin relationships reduces frequent cloud queries, thus saving bandwidth.

[0118] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0119] Based on the same inventive concept, this application also provides a media data processing apparatus for implementing the media data processing method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more media data processing apparatus embodiments provided below can be found in the limitations of the media data processing method described above, and will not be repeated here.

[0120] In one exemplary embodiment, such as Figure 6 As shown, a media data processing apparatus 100 is provided, including: a receiving module 120, a first processing module 140, a second processing module 160, and a transmitting module 180, wherein:

[0121] The receiving module 120 is used to receive the media data stream pushed by the acquisition device; the media data stream is obtained by processing raw stream data of a preset media format using a preset encapsulation protocol and transmission protocol;

[0122] The first processing module 140 is used to acquire multiple preset data processing steps and the execution order of each data processing step; and to process the media data stream based on the multiple data processing steps and the execution order.

[0123] The second processing module 160 is used to obtain the media data processing result of the previous data processing step according to the execution order for each data processing step; iterates through multiple local plugins corresponding to the current data processing step in the local plugin library, and processes the media data processing result using the current local plugin for the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step; if multiple local plugins fail to process, the module queries and loads the data processing plugin corresponding to the current data processing step from the cloud plugin library, and processes the media data processing result using the data processing plugin to obtain the media data processing result of the current data processing step.

[0124] The sending module 180 is used to send the media data processing result corresponding to the last executed data processing step to the client.

[0125] The aforementioned media data processing device processes media data streams through multiple pre-defined data processing steps with an execution order. Each data processing step is encapsulated into small-granular plugins and stored in both a local plugin library and a cloud plugin library. For a single client, the client and cloud work together, eliminating the need to load all plugins in the local plugin library, thus reducing resource consumption. When dealing with media data streams pushed by different acquisition devices, which have different encapsulation protocols, transmission protocols, and pre-defined media formats, and when new media data formats need to be adapted, compared to traditional methods of developing compatible programs, the plugin-based management method only requires loading the new plugin without modifying the logic of existing plugins. This results in higher plugin versatility. Easier to maintain and port, significantly improving media data processing efficiency; processing media data streams based on multiple data processing steps and execution order, when executing the current data processing step, first traversing the local plugins in the local plugin library, and using the local plugins to process the media data processing results of the previous data processing step, which helps improve media data processing efficiency; when multiple local plugins fail to process, the corresponding data processing plugin can be queried and loaded from the cloud plugin library. This combination of local and cloud methods can ensure that each data processing step is successfully processed by dynamically loading cloud plugins when local plugins cannot process media data, which helps to further improve media data processing efficiency.

[0126] In one embodiment, the second processing module 160 is further configured to: query and load the data processing plugin corresponding to the current data processing step from the cloud plugin library; obtain error information generated by each of the multiple local plugins processing the media data processing results; generate error feature values ​​based on the error information; and query and load the data processing plugin corresponding to the current data processing step from the cloud plugin library based on the error feature values.

[0127] In one embodiment, based on the error characteristic value, the second processing module 160 queries and loads the data processing plugin corresponding to the current data processing step from the cloud plugin library. The second processing module 160 is further configured to: query multiple cloud plugins corresponding to the current data processing step in the cloud plugin library; if the characteristic value of any cloud plugin is consistent with the error characteristic value, then the cloud plugin is used as the data processing plugin corresponding to the current data processing step; download the cloud plugin from the cloud plugin library and install it locally.

[0128] In one embodiment, the second processing module 160 is further configured to: obtain the number of successful executions of each of the multiple local plugins corresponding to the current data processing step in the local plugin library; sort the multiple local plugins according to the number of successful executions; and traverse each local plugin according to the sorting result.

[0129] In one embodiment, the media data processing device 100 further includes a first loading module, which is configured to: obtain the number of processing times of each data processing plugin; determine the processing success rate of each data processing plugin based on the ratio of the number of successful processing times to the number of processing times; and preload the data processing plugin locally if the processing success rate of any data processing plugin exceeds a preset success rate.

[0130] In one embodiment, the media data processing device 100 further includes a second loading module, which is used to: obtain the device model of the target device deployed in the target environment; if the device model of the target device is consistent with the device model of the acquisition device, load each data processing plugin corresponding to the acquisition device from the cloud plugin library to the target environment for processing the media data stream pushed by the target device.

[0131] Each module in the aforementioned media data processing device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0132] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores media data streams and media data processing results. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When the computer program is executed by the processor, it implements a media data processing method.

[0133] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0134] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0135] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0136] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0137] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0138] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0139] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0140] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A media data processing method, characterized in that, The method includes: Receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using preset encapsulation and transmission protocols; Obtain a set of preset data processing steps and the execution order of each data processing step; process the media data stream based on the set of data processing steps and the execution order; the data processing steps refer to the steps of protocol parsing and transcoding the media data stream. For each data processing step, the media data processing result of the previous data processing step is obtained according to the execution order; multiple local plugins corresponding to the current data processing step are traversed in the local plugin library; for the current local plugin that is traversed, the media data processing result is processed using the current local plugin; if the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step; if all the local plugins fail, the data processing plugin corresponding to the current data processing step is queried from the cloud plugin library and loaded; the data processing plugin is used to process the media data processing result to obtain the media data processing result of the current data processing step; each plugin in the local plugin library and the cloud plugin library is an encapsulation of each data processing step; Send the media data processing results corresponding to the last data processing step to the client; The step of querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library includes: Obtain error information generated by each of the multiple local plugins in processing the media data processing results; the error information is used to indicate the errors in the processing of the media data processing results by the multiple local plugins; Generate error characteristic values ​​based on multiple error messages; Based on the error characteristic value, query and load the data processing plugin corresponding to the current data processing step from the cloud plugin library.

2. The method according to claim 1, characterized in that, The step of querying and loading the data processing plugin corresponding to the current data processing step from the cloud plugin library based on the error characteristic value includes: Search the cloud plugin library for multiple cloud plugins corresponding to the current data processing step described therein; If any cloud plugin's feature value matches the error feature value, then the cloud plugin is used as the data processing plugin corresponding to the current data processing step; the cloud plugin is downloaded from the cloud plugin library and installed locally.

3. The method according to claim 1, characterized in that, The process of traversing the local plugins corresponding to the current data processing step in the local plugin library includes: Get the number of successful executions of each of the local plugins corresponding to the current data processing step in the local plugin library; The local plugins are sorted according to the number of successful executions, and then each local plugin is traversed according to the sorting result.

4. The method according to claim 1, characterized in that, The method further includes: Get the number of times each data processing plugin has been processed and the number of times it has succeeded; The processing success rate of each data processing plugin is determined based on the ratio of the number of successful attempts to the number of processing attempts. If the success rate of any data processing plugin exceeds the preset success rate, then the data processing plugin is pre-loaded locally.

5. The method according to claim 1, characterized in that, The method further includes: Obtain the device model of the target device deployed in the target environment; If the device model of the target device is the same as that of the acquisition device, then the data processing plugins corresponding to the acquisition device are loaded from the cloud plugin library into the target environment for processing the media data stream pushed by the target device.

6. A media data processing device, characterized in that, The device includes: The receiving module is used to receive media data streams pushed by the acquisition device; the media data streams are obtained by processing raw stream data of a preset media format using a preset encapsulation protocol and a preset transmission protocol. The first processing module is used to acquire a plurality of preset data processing steps and the execution order of each data processing step; and to process the media data stream based on the plurality of data processing steps and the execution order; the data processing steps refer to the steps of protocol parsing and transpackaging the media data stream. The second processing module is used to, for each data processing step, obtain the media data processing result of the previous data processing step according to the execution order; traverse multiple local plugins corresponding to the current data processing step in the local plugin library, and process the media data processing result using the current local plugin. If the processing is successful, the corresponding processing result is used as the media data processing result of the current data processing step; if all the local plugins fail to process, the module queries and loads the data processing plugin corresponding to the current data processing step from the cloud plugin library, and processes the media data processing result using the data processing plugin to obtain the media data processing result of the current data processing step; each plugin in the local plugin library and the cloud plugin library is an encapsulation of each data processing step. The sending module is used to send the media data processing results corresponding to the last data processing step to the client; The second processing module is further configured to obtain error information generated by each of the plurality of local plugins in processing the media data processing results; the error information is used to indicate the error situation in the processing of the media data processing results by the plurality of local plugins; generate error feature values ​​based on the plurality of error information; and query and load the data processing plugin corresponding to the current data processing step from the cloud plugin library based on the error feature values.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 5.

Images