An audio and video stream intelligent processing system based on pipeline transmission

Abstract

The application relates to the technical field of computer data processing, and discloses an audio and video stream intelligent processing system based on pipeline transmission, which comprises a processing module, a data transmission module, a resource management module and a module manager; the processing module is used for performing preset service processing on audio and video stream data; the data transmission module is used for performing asynchronous transmission and buffering of data among the processing modules; the resource management module is used for uniformly pooling and dynamically allocating hardware computing resources in the system; the module manager is used for connecting the processing modules through the data transmission module to build a data processing pipeline, and adaptively scheduling the hardware computing resources according to real-time processing load. Through the dynamic building of the processing pipeline, the adaptive scheduling of the hardware resources and the transactional state management technology, the application improves the resource utilization rate and the fault tolerance capability of the system.

Description

Technical Field

[0001] This invention relates to the field of computer data processing technology, and in particular to an intelligent audio and video stream processing system based on pipeline transmission. Background Technology

[0002] Audio and video streaming technology is a core component of modern information technology, with applications spanning live streaming, video conferencing, security monitoring, and multimedia content distribution. With the surge in data volume and increasing complexity of processing tasks, how to construct an audio and video streaming system that combines high performance, high stability, and high resource utilization has become a technical issue of ongoing concern for those skilled in the art.

[0003] In existing technologies, a common approach is to use a fixed-pipeline architecture to process audio and video streams. This architecture typically breaks down a complete processing flow into multiple independent, functionally fixed processing modules, such as decoding, content analysis, and encoding modules. During system deployment or startup, these modules are sequentially connected in a pre-defined order, forming a static data processing chain. Simultaneously, the hardware computing resources allocated to each processing module, such as CPU cores or GPU resources, are usually configured all at once during system initialization and remain unchanged throughout the entire operation.

[0004] While existing technologies have achieved audio and video stream processing to some extent, their inherent structural design still leads to shortcomings: this static pipeline architecture lacks adaptability to dynamic system changes. This is because the pipeline's topology and resource allocation scheme are fixed at the initial design stage, lacking a mechanism to adjust according to real-time load during operation. When the bitrate or resolution of the input audio and video stream fluctuates, or the computational complexity of a processing module changes dynamically, the preset resource allocation scheme cannot respond. This results in hardware resources being idle for extended periods under low load, causing waste; while under high load, insufficient resources create processing bottlenecks, leading to increased latency or even data loss.

[0005] Furthermore, this architecture has limitations in fault tolerance. This is because the processing modules are tightly coupled and lack a mechanism to track the status of data processing in transit. When any module in the pipeline fails, the data it is processing is lost, and due to the lack of effective fault isolation and data recovery methods, the failure usually propagates upstream or downstream, ultimately causing the entire data processing pipeline to be interrupted. Service recovery often requires restarting the entire system, resulting in a long service interruption time and failing to meet high availability requirements. Summary of the Invention

[0006] The purpose of this invention is to provide an intelligent audio and video stream processing system based on pipeline transmission, which solves the problems of fixed resource allocation and poor fault tolerance in existing audio and video stream processing systems.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] The present invention provides an intelligent audio and video stream processing system based on pipeline transmission, comprising: at least one processing module, at least one data transmission module, a resource management module, and a module manager.

[0009] The processing module is used to perform preset business processing on audio and video stream data.

[0010] The data transmission module has the characteristics of blocking the data transmission operation of the upstream processing module when the internal data storage is full, and blocking the data acquisition operation of the downstream processing module when the internal data storage is empty, which is used to realize asynchronous transmission and flow control between processing modules.

[0011] The resource management module is used to perform unified pooling management and allocation of the hardware computing resources within the system, such as the computing units of the graphics processing unit (GPU) or the usage rights of dedicated hardware accelerators.

[0012] The module manager, which is connected to the processing module, the data transmission module, and the resource management module, is used to connect the processing module through the data transmission module according to the business processing flow, so as to build a data processing pipeline and coordinate the application and use of the hardware computing resources by the processing module.

[0013] To further optimize system resource allocation and dynamically construct processing flows, this invention may also include a module factory. The module factory maintains a module type registry, which records different types of processing modules and their corresponding triplet information. This triplet information includes: a unique identifier for the module type, a constructor for creating an instance of the processing module, and a resource consumption profile describing the standard hardware resource requirements for the processing module to run.

[0014] Based on this structure, the module manager possesses resource-aware pipeline construction capabilities. When constructing a data processing pipeline, the module manager first sends a query command to the resource management module to obtain the total amount of currently available hardware computing resources. Subsequently, the module manager compares the obtained total amount of available resources with the resource consumption profiles of each candidate processing module recorded in the module type registry, selects the processing module type whose resource requirement is less than or equal to the total amount of currently available resources and meets the preset performance priority, and instructs the module factory to create an instance of that type of processing module.

[0015] To achieve adaptive optimization during system operation, the module manager also features resource scheduling based on status feedback. The module manager periodically monitors the operational status of each data transmission module and calculates the processing pressure index L of its upstream processing modules based on this status. m (t) is determined by the following formula:

[0016]

[0017] In the formula, L m (t) represents the processing pressure index of the processing module at time t; The current data depth of the data transmission module is input to it; Q capm This is the maximum capacity of the data transmission module; w represents the change in data depth over a time period Δt. d and w g Δt1 is the preset weighting coefficient; Δt1 is the time period used to measure the change in data depth.

[0018] When the calculated processing pressure index exceeds a preset high threshold or falls below a preset low threshold, the module manager will proactively issue a resource scheduling instruction to the resource management module to increase or reclaim the allocated amount of hardware computing resources associated with the processing module.

[0019] To achieve high fault tolerance of the system, the present invention may further include a transaction state management module. The module manager assigns a unique transaction ID to each data unit entering the data processing pipeline, and the transaction state management module maintains the correspondence between the transaction ID and the processing state of the data unit.

[0020] The internal structure of the processing module includes a transaction interface unit, which is used to report to the transaction status management module when starting and finishing processing any data unit, so as to trigger the transaction status management module to update the processing status of the transaction ID corresponding to the data unit between pending processing, processing and completed.

[0021] When the module manager detects a failure in any processing module, it first queries the transaction status management module to obtain all data units that are under the responsibility of the failure handling module and are currently being processed.

[0022] Subsequently, the module manager initiates the data replay process, which is as follows:

[0023] The data replay interface of the input data transmission module of the fault handling module is invoked to re-inject all the previously acquired but not yet processed data units into the header of the input data transmission module associated with the newly created processing module for fault recovery, so as to ensure the integrity and idempotency of data processing.

[0024] To effectively measure system performance, the module manager is also used to calculate the effective throughput of each processing module based on the completed processing confirmation information returned by the transaction status management module. The effective throughput P m The calculation formula is:

[0025]

[0026] In the formula, P m C represents the effective throughput of the processing module. m (t,Δt) represents the total number of data units that are confirmed as having completed processing by the transaction status management module within the time period Δt; Δt2 is the time period used to calculate the effective throughput.

[0027] In summary, the present invention has at least one of the following beneficial technical effects:

[0028] 1. This invention sets up a module manager that can periodically monitor the data processing load and issue instructions to the resource management module based on the monitoring results to dynamically increase or reclaim the hardware computing resources allocated to the processing module. This avoids resource idleness when the processing load is low and processing bottlenecks caused by insufficient resources when the processing load is high, thereby improving the overall hardware resource utilization of the system.

[0029] 2. This invention tracks the processing status of each data unit by setting up a transaction status management module. When the module manager detects a failure in any processing module, it can query the transaction status management module to obtain all unprocessed data units and instruct the relevant data transmission module to replay these data units. This achieves lossless recovery of processing tasks, ensures the integrity of data processing, greatly shortens service interruption time, and improves the fault tolerance and availability of the system.

[0030] 3. By setting up a data transmission module to perform asynchronous data transmission between various processing modules, this invention decouples the upstream and downstream processing modules in terms of data production and consumption rates. This makes the construction and adjustment of the entire data processing pipeline more flexible, facilitates the dynamic combination of processing flows according to different business needs, and enhances the scalability and modularity of the system. Attached Figure Description

[0031] Figure 1This is a schematic diagram of the system structure of the present invention;

[0032] Figure 2 This is a schematic diagram of the method flow of the present invention;

[0033] Figure 3 This is a schematic diagram of the internal structure of the module manager of the present invention;

[0034] Figure 4 This is a schematic diagram of the internal structure of the module factory of the present invention;

[0035] Figure 5 This is a schematic diagram of the internal structure of the resource management module of the present invention;

[0036] Figure 6 This is a schematic diagram of the internal structure of the transaction status management module of the present invention;

[0037] Figure 7 This is a schematic diagram of the internal structure of the processing module of the present invention;

[0038] Figure 8 This is a schematic diagram of the internal structure of the data transmission module of the present invention. Detailed Implementation

[0039] The following is in conjunction with the appendix Figure 1 -Appendix Figure 8 The present invention will be further described in detail below.

[0040] like Figure 1 As shown, Figure 1 This is a schematic diagram of a system structure according to an embodiment of the present invention. The present invention provides an intelligent audio and video stream processing system based on pipeline transmission, which may include: a module manager 10, a module factory 20, a resource management module 30, a transaction status management module 40, at least one processing module 50, and at least one data transmission module 60.

[0041] Module Manager 10 is used for unified scheduling and control of other modules within the system.

[0042] Module factory 20 is used to create instances of processing module 50 according to instructions.

[0043] The resource management module 30 is used to manage and allocate hardware computing resources within the system in a unified manner.

[0044] The transaction status management module 40 is used to record and maintain the processing status of data units.

[0045] Processing module 50 is used to perform specific business processing on audio and video stream data. Data transmission module 60 is used for asynchronous data transmission between processing modules 50.

[0046] Please see the appendix Figure 2 , Figure 2 This is a schematic flowchart of a method according to an embodiment of the present invention. The present invention provides an intelligent processing method for audio and video streams based on pipeline transmission, which may include the following steps:

[0047] S1: Dynamically construct data processing pipelines based on the current available resources of the system.

[0048] S2: Perform data processing and track the processing status of each data unit during the process.

[0049] S3: During data processing, the processing load is periodically calculated and hardware resource allocation is adaptively adjusted.

[0050] S4: During data processing, detect faults in the processing module and replay the unprocessed data to achieve fault tolerance and recovery.

[0051] In one specific embodiment, the various modules of the system work collaboratively. The module manager 10 is connected to the module factory 20, the resource management module 30, the transaction status management module 40, the processing module 50, and the data transmission module 60.

[0052] After the system starts, step S1 is executed first. The module manager 10 sends a query command to the resource management module 30 to obtain the total amount of currently available hardware computing resources. Then, based on the resource consumption profiles of each candidate processing module 50 obtained from the internal registry of the module factory 20, the module manager 10 selects a type of processing module 50 whose resource requirements are less than or equal to the current total available resources, and instructs the module factory 20 to create an instance of this type. Finally, the data transmission module 60 connects the instances of multiple processing modules 50 to complete the construction of the data processing pipeline.

[0053] After the pipeline is constructed, the system proceeds to step S2. When an external data unit enters the pipeline, the module manager 10 assigns it a unique transaction ID and hands it over to the first processing module 50 for processing. When the processing module 50 begins processing the data unit, it reports the transaction ID to the transaction status management module 40, which then records the ID's status as "processing".

[0054] After processing is completed, the processing module 50 sends the processing result to the next processing module 50 through the data transmission module 60, and reports to the transaction status management module 40 again that the transaction ID has been processed, so that the status is updated to complete.

[0055] While step S2 is being executed, step S3 is being executed in parallel by the system. The module manager 10 periodically monitors the internal data storage depth of the data transmission module 60 connected upstream of the processing module 50, and calculates a processing pressure index based on the proportion of the current data storage depth to its total capacity and the rate of change of the depth over a preset time period. When the calculated result of this index exceeds a preset threshold range, the module manager 10 issues an instruction to the resource management module 30 to increase or reclaim the hardware computing resources allocated to the processing module 50.

[0056] Simultaneously, the system also executes step S4. If any processing module 50 fails during processing, the module manager 10, upon detecting the failure, immediately sends a query command to the transaction status management module 40 to obtain a list of transaction IDs corresponding to all data units that are handled by the failure processing module 50 and whose current processing status is "in progress".

[0057] Subsequently, the module manager 10 instructs the module factory 20 to create a new processing module 50 instance to replace the faulty instance, and instructs the relevant data transmission module 60 to perform data replay, re-injecting all the previously acquired unprocessed data units into the head of the input queue of the newly created processing module 50 instance.

[0058] To measure system performance, the module manager 10 can also calculate the effective throughput of each processing module 50 based on the confirmation information obtained from the transaction status management module 40. This effective throughput is determined as: the total number of data units confirmed as completed by the transaction status management module 40 within a preset time period, divided by the duration of that time period.

[0059] Please see the appendix Figure 3 , Figure 3 This is a schematic diagram of the internal structure of a module manager 10 according to an embodiment of the present invention.

[0060] The module manager 10 is a component used for unified scheduling and control of other modules within the system. In this embodiment, the internal structure of the module manager 10 can be divided into: a resource-aware pipeline construction unit 101, an adaptive resource scheduling unit 102, and a fault-tolerant and transaction control unit 103.

[0061] The resource-aware pipeline construction unit 101 is designed to construct and initialize data processing pipelines based on the current available hardware resources of the system. When this unit receives a pipeline construction instruction, it first sends a query instruction to the resource management module 30 to obtain the total amount of currently available hardware computing resources.

[0062] After receiving available resource data returned by the resource management module 30, the unit reads the resource consumption profiles of one or more candidate processing modules 50 from the internal registry of the module factory 20. The profiles record the amount of standard hardware resources required to run the type of processing module 50.

[0063] Subsequently, the unit performs a comparison operation, comparing the resource requirement of each candidate processing module 50 with the total available resources, and selecting the first processing module 50 type that meets the condition that the resource requirement is less than or equal to the total available resources and meets the preset priority.

[0064] Finally, the unit sends an instantiation instruction containing the selected type identifier to the module factory 20 to create an instance of the processing module 50. This process can be repeated until a complete data processing pipeline consisting of multiple processing modules 50 and data transmission modules 60 is built.

[0065] The adaptive resource scheduling unit 102 functions to dynamically adjust the allocation of hardware computing resources during system operation through a closed-loop feedback mechanism. This unit periodically performs monitoring operations to obtain the internal data storage depth of the data transmission module 60 connected upstream of each processing module 50. Based on the obtained data storage depth and its rate of change within a preset time period, this unit calculates the processing pressure index of the processing module 50.

[0066] The processing pressure index L m (t) is determined by the following formula:

[0067]

[0068] In the formula, L m (t) represents the processing pressure index of the processing module at time t; The current data depth of the data transmission module it inputs; Q capm This is the maximum capacity of the data transmission module; w represents the change in data depth over a time period Δt. d and w g Δt1 is the preset weighting coefficient; Δt1 is the time period used to measure the change in data depth.

[0069] When the processing pressure index L calculated by this unit m If the value of (t) exceeds the preset high threshold, it indicates a data backlog, and the unit sends an instruction to the resource management module 30 to increase resource allocation. Conversely, if the value is below the preset low threshold, it indicates that resources are idle, and an instruction to reclaim some resources is sent.

[0070] The fault tolerance and transaction control unit 103 is responsible for tracking the processing of data units and executing recovery procedures when a processing module 50 fails. Specifically, when a data unit first enters the data processing pipeline, this unit assigns it a unique transaction ID. When this unit detects a failure signal in any processing module 50, it immediately sends a query command to the transaction status management module 40. This command contains the identifier of the faulty module and is used to obtain a list of all data units that are under the responsibility of the faulty module and whose current processing status is recorded as "in progress".

[0071] After obtaining the list, the unit initiates the data replay process. Furthermore, this unit is also responsible for receiving confirmation information forwarded by the transaction status management module 40 regarding the completion of data unit processing, and calculating the effective throughput of the processing module 50 based on this information. The calculation formula is as follows:

[0072]

[0073] In the formula, P m For the effective throughput of the processing module; C m (t,Δt) represents the total number of data units that are confirmed as having been processed within the time period Δt by the transaction status management module; Δt2 is the time period used to calculate the effective throughput.

[0074] Please see the appendix Figure 4 , Figure 4 This is a schematic diagram of the internal structure of a module factory 20 according to an embodiment of the present invention.

[0075] Module factory 20 is a component used to centrally create instances of processing module 50 according to instructions issued by module manager 10. In this embodiment, the internal structure of module factory 20 can be divided into: enhanced registration management unit 201 and module instantiation unit 202.

[0076] The enhanced registration management unit 201 maintains a module type registry and provides an interface for registering and querying the registry. This module type registry stores type information for one or more processing modules 50. Each piece of type information is recorded as a triplet data structure, which includes: a unique identifier for the module type, the constructor of the processing module, and a resource consumption profile.

[0077] The unique identifier for a module type is a string used to distinguish different processing module types 50. The constructor of a processing module is a handle or pointer to the code logic used to create an instance of that processing module 50.

[0078] The resource consumption profile is a structured data record that defines in detail the amount of standard hardware resources required to run this type of processing module 50, such as the number of central processing unit (CPU) cores required, the specific value of graphics processing unit (GPU) memory, and the usage requirements of specific hardware accelerators.

[0079] The enhanced registration management unit 201 provides a registration interface to the system, allowing the new processing module 50 type and its corresponding triplet information to be written to the registry during the system initialization phase. Simultaneously, it provides a query interface to the module manager 10, allowing the module manager 10 to retrieve the corresponding resource consumption profile based on the module type's unique identifier.

[0080] The module instantiation unit 202 is designed to receive and execute module instantiation instructions from the module manager 10. These instructions contain at least one unique identifier for a processing module 50 to be created.

[0081] When the module instantiation unit 202 receives the instantiation instruction, it uses the unique identifier contained in the instruction to search in the module type registry maintained by the enhanced registration management unit 201 to locate the corresponding triplet entry. After finding the corresponding entry, the unit extracts the constructor of the processing module from the triplet.

[0082] Subsequently, the unit executes the extracted constructor, which allocates and initializes resources in system memory, thereby generating a new, fully functional processing module 50 instance. After the instance is created, the unit returns a handle or reference to the newly created processing module 50 instance to the module manager 10 for subsequent use in building data processing pipelines.

[0083] Please see the appendix Figure 5 , Figure 5 This is a schematic diagram of the internal structure of a resource management module 30 according to an embodiment of the present invention.

[0084] The resource management module 30 is a component used for unified pooling management and scheduling allocation of available hardware computing resources within the system. In this embodiment, the internal structure of the resource management module 30 can be divided into: a resource pooling unit 301 and a dynamic scheduling interface unit 302.

[0085] Resource pooling unit 301 is responsible for abstracting and managing the physical hardware computing resources in the system. During system initialization, this unit takes stock of the hardware resources within the system, such as identifying available central processing unit (CPU) cores, graphics processing unit (GPU) computing units and video memory, or dedicated audio and video codec hardware accelerators.

[0086] Subsequently, the unit abstracts these physical resources into logical resources and establishes a resource inventory table. This table records the unique identifier, total quantity, allocated quantity, and currently available quantity for each type of resource. When a resource allocation or reclamation request is received, the unit updates the values ​​in the resource inventory table to reflect the real-time occupancy and availability status of hardware resources in the system.

[0087] The dynamic scheduling interface unit 302 provides a standardized interface for receiving and processing resource requests and instructions from other modules. This unit specifically includes two types of interfaces:

[0088] The first type is the passive request interface. This interface is used to respond to hardware resource requests directly initiated by the processing module 50. When a processing module 50 needs additional computing resources to perform a specific task, it submits a request containing the required resource type and quantity to the resource management module 30 through this interface. Upon receiving the request, the dynamic scheduling interface unit 302 immediately queries the resource inventory table in the resource pooling unit 301. If the inventory meets the request, it performs an allocation operation, updates the inventory table, and returns a success response containing a resource handle to the requesting processing module 50; if the inventory is insufficient, it returns a failure response indicating insufficient resources.

[0089] The second type is the active command interface. This interface is used to receive and execute resource scheduling commands issued by the module manager 10. When the adaptive resource scheduling unit 102 of the module manager 10 decides to adjust the resource allocation quota of a certain processing module 50 based on the calculated processing pressure index, it sends a command to the resource management module 30 through this interface.

[0090] The instruction explicitly specifies the identifier of the target processing module 50 and the specific amount of resources to be added or reclaimed. Upon receiving the instruction, the dynamic scheduling interface unit 302 directly modifies the resource inventory table of the resource pooling unit 301 accordingly, forcibly completes the resource reallocation of the specified processing module 50, and returns the execution result status of the instruction to the module manager 10.

[0091] Please see the appendix Figure 6 , Figure 6 This is a schematic diagram of the internal structure of a transaction status management module 40 according to an embodiment of the present invention.

[0092] The transaction status management module 40 is a component used to track and record the status of the processing of all data units within the system. In this embodiment, the internal structure of the transaction status management module 40 can be divided into: a status table maintenance unit 401 and a query and update interface unit 402.

[0093] The status table maintenance unit 401 is responsible for establishing and maintaining a transaction status table. This transaction status table is a data structure for storing key-value pairs, where the key is a unique transaction ID assigned by the module manager 10, and the value is a data entity that records the current processing status of the transaction ID and the identifier of the processing module 50 responsible for processing the transaction.

[0094] The processing status includes at least three types: pending, in process, and completed. When the unit receives a status update instruction, it looks up the corresponding entry in the table based on the transaction ID provided in the instruction, modifies its status field to the new status specified in the instruction, and updates the identifier of the processing module 50 responsible for processing the data.

[0095] The query and update interface unit 402 provides a standard interface for other modules to interact with the transaction status table. Specifically, this unit provides two interfaces: a status update interface and a status query interface.

[0096] A status update interface is provided for processing module 50 to call. When a processing module 50 obtains a data unit from the upstream data transmission module 60 and is ready to start processing, it submits the transaction ID of the data unit and its own module identifier through this interface. The query and update interface unit 402 then updates the status of the transaction ID in the transaction status table to "processing".

[0097] After the processing module 50 completes the processing of the data unit, it submits the transaction ID again through the interface, and the query and update interface unit 402 then updates the status to completed.

[0098] A status query interface is provided for module manager 10 to call. When module manager 10 needs to obtain all tasks that a fault handling module 50 has not yet completed, it submits the identifier of the fault module through this interface. After receiving the query, query and update interface unit 402 traverses the transaction status table, filters out all entries whose responsible module identifiers match the provided identifiers and whose processing status is "in progress," and returns the list of transaction IDs corresponding to these entries to module manager 10 for it to execute the data replay process.

[0099] Please see the appendix Figure 7 , Figure 7 This is a schematic diagram of the internal structure of a processing module 50 according to an embodiment of the present invention.

[0100] The processing module 50 is a functional component used to perform specific business processing on the audio and video stream data units. Its specific processing logic, such as video decoding, target recognition, or content transcoding, is determined by the module type it is based on when it is created by the module factory 20. In this embodiment, the internal structure of the processing module 50 can be divided into: a core processing unit 501, a transaction interface unit 502, and a resource interface unit 503.

[0101] The core processing unit 501 is designed to execute preset business processing algorithms. This unit receives data units from the upstream data transmission module 60, performs specific calculations or conversions on them, and outputs the processed results. For example, in a video decoding processing module, this unit executes a decoding algorithm; in a target detection processing module, this unit runs a detection model on the decoded video frames.

[0102] The transaction interface unit 502 is used to communicate with the transaction status management module 40 to report the processing progress of the data unit.

[0103] Specifically, after the processing module 50 obtains a data unit from the upstream, but before the core processing unit 501 processes it, the transaction interface unit 502 first calls the status update interface of the transaction status management module 40 to submit the transaction ID of the data unit and the identifier of this module, so as to mark the status of the data unit as being processed in the transaction status table.

[0104] After the core processing unit 501 completes all processing operations on the data unit, the transaction interface unit 502 calls the status update interface again to submit the transaction ID of the data unit in order to update its status to complete.

[0105] The resource interface unit 503 communicates with the resource management module 30 to dynamically request and allocate hardware computing resources. When the core processing unit 501 requires additional, unallocated hardware resources during processing, the resource interface unit 503 sends a resource request to the passive request interface of the resource management module 30. This request includes the type and quantity of the required resources.

[0106] After receiving the resource handle returned by the resource management module 30, this unit delivers the resource to the core processing unit 501 for use. When the core processing unit 501 has finished using the resource, the resource interface unit 503 is responsible for sending a resource return instruction to the resource management module 30 to release the temporarily occupied resource.

[0107] Please see the appendix Figure 8 , Figure 8 This is a schematic diagram of the internal structure of a data transmission module 60 according to an embodiment of the present invention.

[0108] The data transmission module 60 is a component used to implement asynchronous data transmission and caching between the two processing modules 50. In this embodiment, the internal structure of the data transmission module 60 can be divided into: an asynchronous data transmission and flow control unit 601 and a transaction support unit 602.

[0109] The asynchronous data transmission and flow control unit 601 is designed to provide an internal data storage space with limited capacity and to control the data flow of upstream and downstream modules based on the occupancy status of this storage space.

[0110] When the internal data storage space is full, the unit will block the data transmission operation performed by its upstream processing module 50. The transmission operation will be suspended until the internal data storage space of the unit becomes available due to data consumption by the downstream module, at which point it will automatically resume and complete.

[0111] Correspondingly, when the internal data storage space is empty, this unit will block the data acquisition operation performed by its downstream processing module 50. This acquisition operation will be suspended until the upstream module successfully writes a new data unit to the unit's internal data storage space, at which point it will automatically resume and complete. Through this blocking characteristic, this unit achieves decoupling between the upstream and downstream processing modules 50 in terms of data production and consumption rates.

[0112] Transaction support unit 602 provides a data replay interface in response to instructions issued by module manager 10 during fault-tolerant recovery procedures. This data replay interface receives one or more data units to be replayed.

[0113] When the unit receives the data replay instruction and the corresponding data unit, it performs an operation to inject these data units into the header of the internal data storage space of the asynchronous data transmission and flow control unit 601.

[0114] This operation ensures that the replayed data units will be retrieved and processed by the downstream processing module 50 before other data units already existing in the storage space. This function provides the basic data operation support for the system to achieve idempotency in data processing.

[0115] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent audio and video stream processing system based on pipeline transmission, characterized in that, The system includes: At least one processing module, the processing module being used to perform a preset business processing flow on audio and video stream data; At least one data transmission module, the data transmission module being used for asynchronous data transmission between the processing modules; The resource management module is used to perform unified pooling management and allocation of hardware computing resources within the system; A module manager, which is connected to the processing module, the data transmission module, and the resource management module, is used to connect the processing module through the data transmission module according to the business processing flow, so as to build a data processing pipeline and coordinate the application and use of the hardware computing resources by the processing module. The system also includes a module factory, which maintains a module type registry. The module type registry is used to record triplet information, including the constructor of the processing module and its resource consumption profile. The module manager constructs the data processing pipeline by performing the following steps: Query the resource management module for the total amount of currently available hardware computing resources; The total amount of available hardware computing resources is compared with the resource demand recorded in the resource consumption profile. Based on the comparison result, a type that matches the currently available hardware computing resources is selected from multiple candidate processing module types, and then the module factory is instructed to create an instance of that type of processing module.

2. The intelligent audio and video stream processing system based on pipeline transmission according to claim 1, characterized in that, The system also includes a transaction status management module; The module manager assigns a unique transaction ID to each data unit entering the data processing pipeline; When the processing module starts and finishes processing any data unit, it reports to the transaction status management module to trigger the transaction status management module to update the processing status of the transaction ID corresponding to the data unit between pending processing, processing and completed. The module manager is also used to query the transaction status management module to obtain all data units that are handled by the failed processing module and whose processing status is "in progress" when a failure is detected in any processing module, and to start the data replay process.

3. The intelligent audio and video stream processing system based on pipeline transmission according to claim 2, characterized in that, The module manager is also used to calculate the effective throughput of each processing module based on the completed processing confirmation information returned by the transaction status management module. The formula for calculating the effective throughput is: ; In the formula, This represents the effective throughput of the processing module. In the time period The total number of data units that have been confirmed as having completed processing by the transaction status management module; The time period used to calculate the effective throughput.

4. The intelligent audio and video stream processing system based on pipeline transmission according to claim 2, characterized in that, The data replay process includes: the module manager calling the data replay interface provided by the data transmission module associated with the faulty processing module, and re-injecting all data units in the processing state into the head of the input queue of the newly created processing module to recover from the fault.

5. The intelligent audio and video stream processing system based on pipeline transmission according to claim 2, characterized in that, The internal structure of the processing module includes: The core processing unit is used to execute the aforementioned business processing; The transaction interface unit is used to report the transaction ID to the transaction status management module at the start and end of the processing to trigger the update of the processing status; The resource interface unit is used to communicate with the resource management module to request and return the hardware computing resources.

6. The intelligent audio and video stream processing system based on pipeline transmission according to claim 4, characterized in that, The data transmission module also includes a transaction support unit, which is responsible for implementing the data replay interface and responding to the instructions of the module manager to inject the specified data unit into the head of the queue to achieve idempotent fault-tolerant recovery.

Images