Industrial field audio and video dual-channel real-time processing control method and system

Abstract

The application discloses an industrial field audio and video dual-channel real-time processing control method and system, relates to the industrial audio and video processing technical field, and has the following specific steps: collecting audio and video signals output by industrial field equipment to generate original video frame data and original audio frame data; constructing a shared frame system, wherein the shared frame system comprises a frame buffer pool, a reference count module and a timestamp manager; inputting the original video frame data and the original audio frame data into a physical buffer area in the frame buffer pool, and matching timestamps in the original video frame data and the original audio frame data based on the timestamp manager to obtain audio and video frame data, so that memory fragmentation caused by long-term operation is avoided, the reference count module is combined to realize real-time statistics on the business occupation state of each physical storage area, the storage space is recycled and reused only after the area reference count value is zero, and the reading of frame data in a dual-service mode is completed.

Description

Technical Field

[0001] This invention relates to the field of industrial audio and video processing technology, and more specifically, to a method and system for real-time processing and control of dual-channel audio and video in industrial settings. Background Technology

[0002] In industrial automation and intelligent manufacturing on-site monitoring scenarios, CNC machine tools, industrial vision inspection equipment, human-machine interaction screens and other equipment are generally equipped with audio and video output interfaces, which need to achieve low-latency video monitoring and remote cloud streaming transmission simultaneously.

[0003] Existing processing solutions serially couple the display link and the remote streaming link. Acquired audio and video frame data needs to be encoded, transmitted over the network, and rendered sequentially. The two service links share the same set of frame storage resources and scheduling threads. If the remote streaming link experiences network jitter, hardware encoding blockage, or cloud connection interruption, the frame data reading process is directly blocked, causing screen stuttering and audio-visual timing deviations on industrial displays, failing to meet the stringent millisecond-level real-time viewing requirements of industrial production. Furthermore, most existing processing solutions dynamically allocate memory to store audio and video frame data, frequently creating and destroying storage areas during operation, easily leading to memory fragmentation. Embedded edge devices may experience memory overflow and program crashes after prolonged uninterrupted operation. Simultaneously, the existing storage structure lacks a reference counting control mechanism. When the same frame data is read by both remote and mobile services, the storage area occupancy status cannot be tracked, often resulting in frame data being prematurely reclaimed before both services have completed reading, causing screen distortion and audio interruption. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for real-time processing and control of dual-channel audio and video in industrial settings.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method for real-time processing and control of dual-channel audio and video in industrial settings, comprising the following specific steps: Collect audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data; A shared frame system is constructed, which includes a frame buffer pool, a reference counting module, and a timestamp manager. The original video frame data and the original audio frame data are input into the physical buffer in the frame buffer pool. The audio and video frame data are obtained by matching the timestamps in the original video frame data and the original audio frame data based on the timestamp manager. Establish decoupled display consumption queues and remote push streaming consumption queues, and match the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices. The corresponding audio and video frame data is determined based on the queue index of the display consumption queue, and the display terminal is obtained by adapting the corresponding audio and video frame data based on the hardware image processing unit. The system determines the status characteristics of the remote streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped. If the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote streaming terminal. The integrated display terminal and remote streaming terminal obtain the on-site audio and video processing results.

[0006] Preferably, a shared frame system is constructed, which includes a frame buffer pool, a reference counting module, and a timestamp manager, specifically including the following steps: A frame buffer pool is constructed based on contiguous physical memory, and independent storage areas are allocated to hold audio and video frame data collected in the industrial field. A reference counting module is constructed based on the frame buffer pool. The independent reference status of each storage area in the frame buffer pool is matched based on the reference counting module. The business call status of frame data in each storage area is collected to obtain the module operation results. Based on the module's execution results, the timestamp manager is matched with all frame data in the frame buffer pool to obtain timing reference data; Timing calibration of the entire frame data based on the timing reference data yields timing-normalized audio and video frame data; Based on the time-ordered audio and video frame data and the independent reference status of the reference counting module, the storage resources of the frame buffer pool are managed and controlled. Storage areas without business calls are reclaimed and new frame data is reused to obtain a dynamically iterative shared frame system.

[0007] Preferably, the original video frame data and original audio frame data are input into the physical buffer of the frame buffer pool, and the audio and video frame data are obtained by matching the timestamps in the original video frame data and original audio frame data based on the timestamp manager. Specifically, this includes the following steps: The timing baseline is obtained by timing matching the original video frame data and the original audio frame data in the frame buffer pool according to the timestamp manager. Based on the time series reference, the video frames and audio frames corresponding to the time series dimension are selected to obtain the target audio and video frames. The time series deviation is checked on the selected target audio and video frames to obtain the valid video frames and valid audio frames. The valid video frames and valid audio frames are fused together to obtain valid audio and video frames. The frame data storage area of ​​the valid audio and video frames is determined by the reference counting module to obtain the audio and video frame data.

[0008] Preferably, establishing a decoupled display consumption queue and a remote push streaming consumption queue specifically includes the following steps: The service processing links are divided according to the audio and video frame data. The service processing links include screen display services and remote network streaming services. By matching the screen display service with the remote network streaming service through the corresponding data consumption channels, we obtain the display consumption link and the remote streaming consumption link. Based on the resource management mechanism of the shared frame system, corresponding resource scheduling permissions are matched for the display consumption queue and the remote push streaming consumption queue; The business functions of the display consumption link and the remote push stream consumption link are solidified to form an independently operating display consumption queue and a remote push stream consumption queue.

[0009] Preferably, the display consumption queue and the remote push streaming consumption queue are matched to the physical buffer to obtain the queue index, specifically including the following steps: Match the physical storage area in the frame buffer pool based on the audio and video frame data; Divide the storage resource characteristics of display services and remote streaming services in each physical storage area, and connect the mutually decoupled display consumption queue and remote streaming consumption queue according to the storage resource characteristics; The resource mapping relationship is obtained by mapping and associating the display consumption queue, remote push streaming consumption queue, and corresponding physical storage areas. The queue index is formed by generating location identifiers based on the resource mapping relationship.

[0010] Preferably, determining the audio / video frame data based on the queue index of the display consumption queue specifically includes the following steps: The corresponding physical storage area within the frame buffer pool is determined based on the queue index of the displayed consumption queue. Based on the reference counting module of the shared frame system, the resource occupancy status of the physical storage area is verified to obtain the valid frame data within the storage area; Based on the resource mapping relationship of the queue index, filter the number of valid frames in the physical storage area; Extract the valid frame data corresponding to the display scene to obtain audio and video frame data.

[0011] Preferably, matching the physical storage area in the frame buffer pool based on the audio and video frame data specifically includes the following steps: Match audio and video frame data with corresponding storage attribute characteristics, and determine frame data storage attributes based on the resource management mechanism of the shared frame system; The storage attribute characteristics of the collected frame data are compared with the physical storage areas in the frame buffer pool that are in an idle state. The storage compatibility between the frame data storage attributes and the storage of each physical storage area is compared to select the target physical storage area.

[0012] Preferably, determining the corresponding physical storage area within the frame buffer pool based on the queue index of the display consumption queue specifically includes the following steps: The frame data attributes are obtained by filtering the storage adaptation attributes and business adaptation attributes in the audio and video frame data. Based on the extracted frame data attributes, match all initialized physical storage areas in the frame buffer pool, verify the space specifications and resource adaptation attributes of each physical storage area, and filter out the storage areas that are suitable for the current audio and video frame data storage requirements. Combine the reference counting module of the shared frame system to match the valid storage area of ​​the current frame data; The physical storage area is obtained by determining the target storage area for storing and retrieving appropriate audio-visual frame data based on the resource characteristics of the effective storage area.

[0013] Preferably, the valid frame data within the storage area is obtained by verifying the resource occupancy status of the physical storage area using the reference counting module of the shared frame system, specifically including the following steps: The current frame data is obtained by retrieving audio and video frame data from the physical storage area. Based on the timestamp manager of the shared frame system, the temporal integrity and synchronization of the current frame data are verified to obtain a valid frame. The reference frame is obtained by verifying the resource occupancy status of valid frames using the reference counting module; By combining the business adaptation standards for audio and video processing in industrial settings, the integrity and transmission compliance of the baseline frame are verified to obtain valid frame data.

[0014] An industrial field audio and video dual-channel real-time processing and control system includes: Data acquisition module: Acquires audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data; System construction module: Constructs a shared frame system, which includes a frame buffer pool, a reference counting module, and a timestamp manager; Data matching module: Inputs the original video frame data and the original audio frame data into the physical buffer in the frame buffer pool, and matches the timestamps in the original video frame data and the original audio frame data based on the timestamp manager to obtain the audio and video frame data; Queue creation module: Creates mutually decoupled display consumption queues and remote push streaming consumption queues, and matches the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices; Adaptation module: Determines the corresponding audio and video frame data based on the queue index of the display consumption queue, and adapts the corresponding audio and video frame data based on the hardware image processing unit to obtain the display terminal; Judgment Module: Determines the status characteristics of the remote push streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped; if the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote push streaming terminal. Results output module: Integrates the display terminal and remote streaming terminal to obtain the on-site audio and video processing results.

[0015] Compared with existing technologies, this invention has the following advantages: By constructing a shared frame system including a frame buffer pool, a reference counting module, and a timestamp manager, the frame buffer pool is divided into fixed independent storage areas based on contiguous physical memory, avoiding memory fragmentation during long-term operation. Combined with the reference counting module's real-time statistics of the service occupancy status of each physical storage area, reused storage space is only reclaimed after the area's reference count value reaches zero. This solves the problem of prematurely reclaiming frame data before dual-service reading is completed, avoiding defects such as screen tearing, audio interruption, memory overflow, and program crashes, thus adapting to the uninterrupted and stable operation requirements of industrial equipment. Furthermore, the timestamp manager uniformly matches and calibrates the timing of audio and video frames across the entire domain. The system uses a unified timing benchmark to complete frame pairing and deviation verification, and then distributes services after the timing is regulated, ensuring stable audio and video synchronization between the display and remote streaming outputs. By building a decoupled display consumption queue and a remote streaming consumption queue, the two queues only store physical buffer indices to achieve zero-copy distribution of frame data. The two sets of service link scheduling threads and resource channels are completely isolated. The display channel is processed independently based on the resident hardware image processing unit and is not affected by the running status of the remote streaming link. This solves the problems of remote network jitter, encoding blockage, and connection interruption dragging down the picture in the serial coupling architecture, causing display stuttering and excessive latency, and stably maintaining millisecond-level real-time output for display. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating a method for real-time processing and control of dual-channel audio and video in an industrial setting, as provided in an embodiment of the present invention. Figure 2 This is a schematic diagram of a module for an industrial field audio and video dual-channel real-time processing and control system, provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0018] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0019] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0020] Reference Figures 1-2 As shown.

[0021] The embodiments further illustrate the industrial field audio and video dual-channel real-time processing and control method and system proposed in this invention.

[0022] A method for real-time processing and control of dual-channel audio and video in industrial settings, comprising the following specific steps: It collects audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data.

[0023] The topology diagram defines the signal sources on the field / edge side as industrial equipment, i.e., signal sources. These signal sources include various field industrial devices such as PLCs / industrial control computers, industrial touch screens (HMIs), vision inspection equipment, and industrial hosts. HDMI audio and video signals are sent to the system hardware via HDMI-IN. The hardware internally splits into two hardware processing sub-links: video acquisition based on the V4L2 framework and audio acquisition based on the ALSA framework. The HDMI input interface (HDMI-IN) is used to connect the HDMI audio and video signals output by the industrial equipment, supporting RGB, YCbCr4:4:4, YCbCr4:2:2, and YCbCr4:2:0 video formats, and supporting input resolutions of 1920×1080@30fps and above.

[0024] A shared frame system is constructed, which includes a frame buffer pool, a reference counting module, and a timestamp manager. The original video frame data and the original audio frame data are input into the physical buffer in the frame buffer pool. The audio and video frame data are obtained by matching the timestamps in the original video frame data and the original audio frame data based on the timestamp manager. Establish decoupled display consumption queues and remote push streaming consumption queues, and match the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices. The corresponding audio and video frame data is determined based on the queue index of the display consumption queue, and the display terminal is obtained by adapting the corresponding audio and video frame data based on the hardware image processing unit.

[0025] The hardware image processing unit performs image scaling and quality optimization based on the RGA hardware unit. After processing, the audio and video signals are output to the local display terminal consisting of a monitor, TV and speakers via the HDMI-OUT interface, directly completing real-time video playback and audio playback. The HDMI output interface (HDMI-OUT) is used for synchronous video and audio output of the local display terminal, supporting resolutions of 1920×1080 and above, enabling real-time display of industrial equipment operation and sound playback.

[0026] The system determines the status characteristics of the remote streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped. If the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote streaming terminal.

[0027] The video encoding uses the Rockchip-MPP hardware module to perform H.264 / H.265 format compression, and the audio is encoded in AAC format. After the bitstream is encapsulated by FFmpeg, it is pushed outward using the SRT protocol. The data stream is transmitted uplink through an industrial network / Internet consisting of wired, WiFi, 4G, and 5G networks. It is first sent to the streaming media service platform to complete SRT reception, streaming media distribution, and SRT to WebRTC protocol conversion. The cloud management and service platform simultaneously performs device access control, streaming parameter configuration, device operation status alarms, access permissions, and operation log recording. The distributed streaming media can be viewed in real time by remote clients such as PCs, tablets, and mobile phones using the WebRTC protocol. The data stream also supports cross-system data communication with SCADA, MES, and industrial Internet platforms. The device side is equipped with local parameter configuration, operation status monitoring, remote command reception, and overall system operation and maintenance management capabilities. Among them, the integrated Rockchip-MPP hardware codec module supports H.264 / H.265 video hardware encoding, providing hardware acceleration capabilities for remote low-latency audio and video transmission.

[0028] The integrated display terminal and remote streaming terminal obtain the on-site audio and video processing results.

[0029] It is also equipped with a 24V active cooling fan and a metal heat dissipation structure to achieve continuous heat dissipation of the processor and hardware coding unit, meeting the needs of long-term continuous operation in industrial sites.

[0030] The system performs screen adaptation processing by reading the queue index of the display consumption queue and accurately locating the corresponding audio and video frame data in the frame buffer pool based on the resource mapping relationship stored in the index. The hardware image processing unit is then invoked to perform hardware-level adaptation operations on this set of audio and video frame data. The adaptation operations include screen resolution scaling, pixel format conversion, and color parameter adjustment. The hardware processing time for a single frame is equal to the original frame processing time minus the CPU software processing time. The hardware unit completes all screen adjustment work based on dedicated computing power. After processing, the output is a display terminal adapted to the specifications of the industrial display. This processing flow is in a resident running state and will not be affected by the operation of the remote streaming link. For example, if the remote link experiences network congestion, the hardware adaptation process will continue to output the screen stably.

[0031] The system performs branch determination processing for remote streaming links. It collects real-time status characteristic values ​​of the corresponding remote streaming consumption links to determine their status. Links operate in two modes: dormant and active. When the status characteristic corresponds to the dormant state, the system directly terminates all encoding, encapsulation, and network transmission operations for audio and video frame data, releasing hardware encoding units and network transmission resources to reduce device computing power and bandwidth consumption. When the status characteristic corresponds to the active state, the system retrieves the queue index of the remote streaming consumption queue to locate the audio and video frame data, and starts the hardware encoding unit to perform compression encoding operations on the frame data. The calculation formula is: Total capacity of the encoded output stream = Total capacity of the original frame data ÷ Encoding compression ratio. The encoding ratio is dynamically adjusted based on the remote transmission bandwidth. After the hardware encoding unit completes the audio and video compression and encapsulation, it outputs a remote streaming terminal that can be transmitted over the public network. The remote link occupies independent computing resources and will not preempt the scheduling rights of the hardware image processing unit.

[0032] The system integrates the results of dual-channel operations, simultaneously aggregating the continuously output display terminal video signal and the remote streaming terminal bitstream data that can be started and stopped on demand. The two output data share the same source audio and video frame data, and the processing flow is completely isolated. A failure in any one link will not interrupt the output flow of the other. After the two output information is aggregated and integrated, a complete on-site audio and video dual-channel processing result is formed, which is simultaneously provided to on-site operators for viewing and remotely retrieved and used by the remote operation and maintenance platform. The hardware part receives the audio and video signals output by the industrial equipment through the HDMI-IN interface. After the system completes the acquisition, processing and distribution, one channel is synchronously output to the on-site display terminal through the HDMI-OUT interface to achieve local real-time display; the other channel completes the encoding and network transmission and sends it to the remote monitoring platform to realize the local display and remote sharing of the audio and video information of the industrial equipment.

[0033] Constructing a shared frame system, which includes a frame buffer pool, a reference counting module, and a timestamp manager, involves the following steps: A frame buffer pool is constructed based on contiguous physical memory, and independent storage areas are allocated to hold audio and video frame data collected in the industrial field. A reference counting module is constructed based on the frame buffer pool. The independent reference status of each storage area in the frame buffer pool is matched based on the reference counting module. The business call status of frame data in each storage area is collected to obtain the module operation results. Based on the module's execution results, the timestamp manager is matched with all frame data in the frame buffer pool to obtain timing reference data; Timing calibration of the entire frame data based on the timing reference data yields timing-normalized audio and video frame data; Based on the time-ordered audio and video frame data and the independent reference status of the reference counting module, the storage resources of the frame buffer pool are managed and controlled. Storage areas without business calls are reclaimed and new frame data is reused to obtain a dynamically iterative shared frame system.

[0034] The frame buffer pool is built based on contiguous physical memory. Several independent storage areas are divided within the contiguous physical memory. Each independent storage area is dedicated to storing audio and video frame data collected from the industrial site. For example, when 40 independent storage areas are allocated, the frame buffer pool can cache up to 40 complete audio and video frames at the same time. The calculation formula is: memory capacity required for a single area = memory occupied by a single video frame + memory occupied by a single audio frame. Before the calculation, the two types of memory values ​​are uniformly converted to MB to ensure a consistent calculation standard. All the divided independent storage areas constitute the basic storage carrier of the frame buffer pool.

[0035] A reference counting module is built based on the established frame buffer pool. The reference counting module assigns a unique independent reference state to each independent storage area in the frame buffer pool. The value of the independent reference state changes dynamically with business calls. For each new business that reads frame data in the area, the corresponding state value is incremented by 1. For each business that completes a read, the corresponding state value is decremented by 1. The reference counting module continuously counts the frequency and duration of business calls to frame data in all storage areas, summarizes the occupancy information of all areas to generate the module's running results, and fully records the real-time independent reference state value of each storage area as the basis for subsequent timing matching and resource reclamation.

[0036] Based on the module's execution results, the timestamp manager and all frame data within the frame buffer pool are bound and matched one by one to generate unified time-series baseline data. The time-series baseline data consists of time-series values ​​labeled with a unified measurement standard for each group of audio and video frames. The time-series values ​​are uniformly converted to milliseconds. The calculation formula is: single frame time-series baseline value = hardware acquisition start millisecond value + frame sampling offset millisecond value. The two types of time parameters are added together after the units are unified to complete the calculation. All frame data share the same set of time-series measurement rules to eliminate the time standard differences of frame data in different storage areas.

[0037] Using the timing reference data as a unified reference, full-domain timing calibration is performed on all audio and video frame data in the frame buffer pool. During the calibration process, the timing values ​​of video frames and audio frames in the same group are compared. If the difference between the timing values ​​of video frames and audio frames exceeds the allowable difference, the frame with the earlier timing is marked as delayed; if the difference between the timing values ​​of audio frames and video frames exceeds the allowable difference, the audio frame is marked as delayed. After unified correction, the timing deviation of all audio and video frames is within the allowable range, resulting in audio and video frame data with regular timing.

[0038] Finally, combining the time-ordered audio and video frame data with the independent reference status recorded by the reference counting module to manage storage resources, the system reads the status value corresponding to each independent storage area block by block. When the independent reference status value = 0, it means that there are no business calls in that area, and the system directly reclaims the idle storage area. The reclaimed area is restored to an empty state and can be rewritten with newly collected frame data. When the independent reference status value > 0, it means that the frame data in the area is still occupied by business, and the reclamation operation is not performed temporarily. The entire process of acquisition, counting statistics, timing matching calibration, and resource reclamation and reuse runs continuously in a loop. The frame buffer pool storage resources can be continuously reused, ultimately forming a complete shared frame system that can be dynamically iteratively updated.

[0039] The original video frame data and original audio frame data are input into the physical buffer of the frame buffer pool. The audio and video frame data are obtained by matching the timestamps in the original video frame data and original audio frame data based on the timestamp manager. The specific steps include: The timing baseline is obtained by timing matching the original video frame data and the original audio frame data in the frame buffer pool according to the timestamp manager. Based on the time series reference, the video frames and audio frames corresponding to the time series dimension are selected to obtain the target audio and video frames. The time series deviation is checked on the selected target audio and video frames to obtain the valid video frames and valid audio frames. The valid video frames and valid audio frames are fused together to obtain valid audio and video frames. The frame data storage area of ​​the valid audio and video frames is determined by the reference counting module to obtain the audio and video frame data.

[0040] Based on the timestamp manager, timing matching operations are performed on the raw video and audio frame data already stored in the frame buffer pool to generate a unified timing benchmark. The timestamp manager extracts the acquisition time values ​​of each set of raw video and audio frames, and converts all time values ​​into milliseconds. The calculation formula is: Single frame timing difference = Video frame acquisition millisecond value - Audio frame acquisition millisecond value. The system uses the sum of the timing differences of all frames as a unified timing benchmark for the entire domain. For example, after synchronously acquiring 10 sets of raw audio and video frames, the timing differences of all frames are summarized to determine a unified timing reference standard that can be used for comparison of all frames, providing a unified judgment basis for subsequent frame pairing and screening.

[0041] Using the generated timing benchmark as a screening criterion, video and audio frames with corresponding timing dimensions are selected from all original frames and combined to form target audio-video frames. A timing deviation verification operation is then performed on the selected target audio-video frames to distinguish qualified and valid video and audio frames. During the verification process, the system retrieves the allowable deviation value corresponding to the timing benchmark and compares the timing difference of a single set of target audio-video frames with the allowable deviation value. If the absolute value of the calculated result of a single set of timing differences is less than or equal to the allowable deviation value, the corresponding video and audio frames are determined to be valid frames. If the absolute value of the calculated result of a single set of timing differences is greater than the allowable deviation value, the frame data of that set is discarded and will not participate in subsequent fusion processes. For example, when the allowable deviation value is set to 20 milliseconds, if a set of frames has a timing difference of 12 milliseconds, it meets the verification criterion, and the corresponding frame is retained as a valid frame.

[0042] The valid video and audio frames obtained after verification and filtering are fused to generate complete valid audio and video frames. Then, the record information of the reference counting module is retrieved to locate the frame data storage area where the current valid audio and video frames are stored. After the area binding is completed, the final required audio and video frame data is obtained. The reference counting module marks the number of services occupied by frames corresponding to each physical buffer. The fused valid audio and video frames are bound to the original physical buffers, and the independent reference status value of the corresponding storage area is updated synchronously. For each set of valid audio and video frames generated, the reference value of the corresponding storage area is incremented by 1 to lock the frame data belonging area. Finally, the audio and video frame data that has completed timing matching, verification, fusion and storage location is output.

[0043] Establishing a decoupled display consumption queue and a remote push streaming consumption queue includes the following steps: The service processing links are divided according to the audio and video frame data. The service processing links include screen display services and remote network streaming services. By matching the screen display service with the remote network streaming service through the corresponding data consumption channels, we obtain the display consumption link and the remote streaming consumption link. Based on the resource management mechanism of the shared frame system, corresponding resource scheduling permissions are matched for the display consumption queue and the remote push streaming consumption queue; The business functions of the display consumption link and the remote push stream consumption link are solidified to form an independently operating display consumption queue and a remote push stream consumption queue.

[0044] Based on audio and video frame data, two independent business processing links are split. These two types of business processing links correspond to screen display services and remote network streaming services, respectively. The processing objectives of the two types of services are completely different. For example, the same set of audio and video frame data can be supplied to the on-site display output and transmitted to the remote cloud server at the same time. The two types of services will not share the same set of processing logic. The link separation is completed from the source of the business. The calculation formula is: the maximum concurrent capacity of a single service = the total number of storage areas in the frame buffer pool ÷ 2. The storage resource limit is equally allocated to each type of service. The result of the integer division operation is used to ensure equal resource allocation.

[0045] The two types of services are each assigned a dedicated data consumption channel, resulting in mutually independent display consumption links and remote streaming consumption links. The display service is matched with a low-latency hardware read / write channel, while the remote network streaming service is matched with a network encoding transmission channel. The data read / write rules for the two types of channels are isolated from each other. The single frame read time of the channel is much shorter than that of the remote channel. The difference between the remote channel read time and the channel read time represents the additional processing time caused by network transmission. The two links read frame data independently, and the blockage of one link will not occupy the data channel of the other link.

[0046] The resource management mechanism built into the shared frame system is invoked, and based on this, exclusive resource scheduling permissions are assigned to the two consumer queues that are yet to be formed. The shared frame system relies on the reference counting module to manage all physical buffer resources. The explicit consumer queue is assigned priority scheduling permission, while the remote push-stream consumer queue is assigned regular scheduling permission. When each queue retrieves a physical buffer, the reference status value of the corresponding storage area is incremented by 1. The resource retrieval operations of the two types of queues do not compete for scheduling priority. Queues can obtain free storage areas first, and remote queues are only allocated remaining storage resources after the queue has completed scheduling.

[0047] The established display consumption link and remote push streaming consumption link are subjected to business function solidification operation. The solidification operation will lock the dedicated data channel, resource scheduling authority and business processing logic corresponding to each link. After solidification, the two links can run completely independently, and finally generate mutually decoupled display consumption queue and remote push streaming consumption queue. When the remote push streaming link experiences network lag or encoding delay, it will not interfere with the frame reading and output process of the display queue, thus completely realizing the parallel isolation of the two types of services.

[0048] The process involves matching the displayed consumption queue with the remote push streaming consumption queue and then matching them to the physical buffer to obtain the queue index. This includes the following steps: Match the physical storage area in the frame buffer pool based on the audio and video frame data; Divide the storage resource characteristics of display services and remote streaming services in each physical storage area, and connect the mutually decoupled display consumption queue and remote streaming consumption queue according to the storage resource characteristics; The resource mapping relationship is obtained by mapping and associating the display consumption queue, remote push streaming consumption queue, and corresponding physical storage areas. The queue index is formed by generating location identifiers based on the resource mapping relationship.

[0049] Based on the time-ordered audio and video frame data, the physical storage area inside the frame buffer pool that carries the set of audio and video frames is matched. Each set of audio and video frame data will uniquely correspond to one physical storage area. A single physical storage area can hold one complete set of audio and video frames. The storage capacity of a single area = the memory occupied by the video frame + the memory occupied by the audio frame. Before calculation, the two types of memory values ​​are uniformly converted to KB units and the summation is performed to lock the underlying storage carrier corresponding to the frame data. For example, when a set of 1080P video frames is paired with 48kHz audio frames, the total space occupied by the physical area is obtained by adding the memory of the two types of frames.

[0050] For each physical storage area, two types of storage resource characteristics are defined for each service. One type of characteristic is adapted to the low-latency read requirements of display services, and the other type of characteristic is adapted to the encoding and transmission requirements of remote streaming services. Based on the defined storage resource characteristics, two sets of decoupled consumption queues are connected. The display consumption queue is matched with low-latency read and write resource characteristics, and the remote streaming consumption queue is matched with encoding and forwarding resource characteristics. The two types of queues are connected only through resource characteristics, and there is no data interaction or resource contention between the queues.

[0051] Complete the bidirectional mapping association between queues and physical storage areas, establish a complete resource mapping relationship, and synchronously record and display the access path of the consumer queue reading the storage area, as well as the access path of the remote push-stream consumer queue reading the same storage area. The two business access paths are independent of each other. The calculation formula is: total reference count value of a single area = business reference value + remote business reference value. The reference values ​​generated by the two types of businesses are added together to obtain the real-time occupancy count of the storage area.

[0052] Based on the complete resource mapping relationship, a unique location identifier is generated for each group of queues and corresponding physical storage areas. All location identifiers are integrated and summarized to form a unified and usable queue index. The queue index enables two sets of consumer queues to quickly locate the target physical storage area, read the corresponding audio and video frame data, and realize zero-copy data retrieval. Network fluctuations of remote streaming services will not interfere with the display service reading frame data in the storage area through the queue index.

[0053] The audio and video frame data is determined based on the queue index of the displayed consumption queue, specifically including the following steps: The corresponding physical storage area within the frame buffer pool is determined based on the queue index of the displayed consumption queue. Based on the reference counting module of the shared frame system, the resource occupancy status of the physical storage area is verified to obtain the valid frame data within the storage area; Based on the resource mapping relationship of the queue index, filter the number of valid frames in the physical storage area; Extract the valid frame data corresponding to the display scene to obtain audio and video frame data.

[0054] The system reads the queue index corresponding to the consumption queue and locks the matching physical storage area in the frame buffer pool according to the position identifier recorded in the index. The queue index pre-stores the mapping information that corresponds one-to-one between the queue and the physical storage area. Each index value uniquely corresponds to a single physical storage area. For example, when the queue index value is equal to 5, the system directly locates the physical storage area numbered 5 in the frame buffer pool, thus completing the rapid location of the underlying storage carrier.

[0055] The reference counting module within the shared frame system is invoked to perform resource occupancy status verification on the located physical storage area. The reference counting module will output the current total reference value of the area. The calculation formula is: Total reference value = Display service reference value + Remote push streaming service reference value. The occupancy counts generated by the two types of services are added together to obtain the real-time occupancy index. If the total reference value is greater than 0, it is determined that there is non-expired frame data in the physical storage area, and valid frame data is read from it. If the total reference value is equal to 0, it is determined that the frame data in the area has expired, and the data reading operation is abandoned.

[0056] The system retrieves the built-in resource mapping relationship of the queue index and filters the frame data stored in the physical storage area according to the mapping rules. The resource mapping relationship distinguishes between two types of frame data: those adapted for display scenarios and those adapted for remote streaming scenarios. The system removes frame data that is only adapted for remote streaming services and retains only frame data adapted for display services. The number of valid frames that meet the business standards is calculated using the formula: Total number of valid frames in a single area = Number of frames adapted for the scenario, without adding the number of frames in the remote scenario.

[0057] Valid frame data suitable for the display scenario is extracted separately from the filtered and retained frame data. After extraction, the complete audio and video frame data required for the display business is obtained. The extraction operation only reads the picture and audio information that matches the business specifications in the current physical storage area. It will not change the frame storage content corresponding to the remote push streaming business, ensuring that the two decoupled queues do not interfere with each other when reading data, and meeting the usage requirements of real-time picture retrieval.

[0058] Matching the physical storage area in the frame buffer pool based on audio and video frame data includes the following steps: Match audio and video frame data with corresponding storage attribute characteristics, and determine frame data storage attributes based on the resource management mechanism of the shared frame system; The storage attribute characteristics of the collected frame data are compared with the physical storage areas in the frame buffer pool that are in an idle state. The storage compatibility between the frame data storage attributes and the storage of each physical storage area is compared to select the target physical storage area.

[0059] The system matches the audio and video frame data to be stored with corresponding storage attribute characteristics, and then locks the dedicated frame data storage attribute for this group of frame data according to the resource management mechanism built into the shared frame system. The storage attribute characteristics of audio and video frame data include two core parameters: single frame memory usage capacity and read / write latency threshold. The calculation formula is: single frame memory usage capacity = video frame memory usage value + audio frame memory usage value. During the calculation, the two types of memory values ​​are uniformly converted to MB units before the addition operation is performed. The resource management mechanism will divide the storage attribute category according to the business purpose corresponding to the audio and video frames. Frame data corresponding to display business is matched with low-latency storage attribute, and frame data corresponding to remote push streaming business is matched with high-capacity storage attribute. For example, a group of 1080P audio and video frames used for real-time display is calculated to have a total single frame memory usage capacity of 3MB, and the system assigns it a low-latency frame data storage attribute.

[0060] Two types of reference information are collected simultaneously: one is the complete storage attribute characteristics corresponding to the current audio and video frame data, and the other is the information of all idle physical storage areas within the frame buffer pool. Then, the frame data storage attributes are compared one by one with the storage adaptability parameters inherent in each idle physical storage area. Areas with a perfect fit are selected as target physical storage areas. The storage adaptability parameters include two indicators: the maximum capacity of the area and the read / write response time. The system performs two sets of numerical comparison calculations: the first is the physical area capacity minus the capacity occupied by a single frame; a result greater than or equal to 0 indicates space capacity adaptability. The second is the physical area read / write response time minus the frame latency threshold; a result less than or equal to 0 indicates read / write speed adaptability. Only idle physical storage areas that simultaneously meet the standards of both sets of calculations are determined to be adaptable areas and selected as target physical storage areas. After selection, the audio and video frame data can be written to the target physical storage area, achieving a precise correspondence between frame data and the storage medium.

[0061] Based on the queue index of the displayed consumption queue, the corresponding physical storage area within the frame buffer pool is determined, specifically including the following steps: The frame data attributes are obtained by filtering the storage adaptation attributes and business adaptation attributes in the audio and video frame data. Based on the extracted frame data attributes, match all initialized physical storage areas in the frame buffer pool, verify the space specifications and resource adaptation attributes of each physical storage area, and filter out the storage areas that are suitable for the current audio and video frame data storage requirements. Combine the reference counting module of the shared frame system to match the valid storage area of ​​the current frame data; The physical storage area is obtained by determining the target storage area for storing and retrieving appropriate audio-visual frame data based on the resource characteristics of the effective storage area.

[0062] Based on the audio and video frame data to be retrieved, storage adaptation attributes and business adaptation attributes are extracted and separated. These two types of attributes are then integrated to form complete frame data attributes. Storage adaptation attributes include the total memory usage per frame, calculated as: Total memory usage per frame = Video frame memory usage + Audio frame memory usage. Before calculation, both memory values ​​are converted to MB and added. Business adaptation attributes mark the corresponding business type for this set of frame data. For example, frames specific to display services are marked for low-latency reading requirements, while frames for remote streaming services are marked for encoding and forwarding requirements. The combination of these two types of attributes constitutes a complete basis for distinguishing the usage scenarios of frame data.

[0063] Based on the extracted frame data attributes, the system iterates through all initialized physical storage areas within the frame buffer pool, verifying the space specifications and resource adaptation attributes of each physical storage area. The space specification parameters include the maximum memory capacity of the area. The system performs a calculation of the maximum memory capacity of the area minus the total memory capacity occupied by a single frame. A result greater than or equal to 0 indicates that the space specifications meet the standards. The resource adaptation attributes are matched against the business adaptation attributes carried in the frame data to meet the read and write response standards. Physical storage areas that pass both checks are initially selected as candidate storage areas that meet the basic storage conditions.

[0064] The built-in reference counting module of the shared frame system is invoked to perform validity matching and judgment on all candidate storage areas. The reference counting module records the current total reference value of each storage area. The calculation formula is: total reference value = business reference value + remote business reference value. Only candidate areas with a total reference value greater than 0 are determined to be valid storage areas containing complete frame data. Storage areas with a reference value of 0 or whose frame data has been recycled and expired are eliminated, thus narrowing the scope of storage area screening.

[0065] Read the resource characteristics corresponding to all valid storage areas. The resource characteristics include two types of parameters: read and write response speed and the number of concurrent reads supported. Select the area that meets both the audio and video frame data storage requirements and the display queue's fast retrieval requirements as the target storage area. This target storage area is the physical storage area that the queue index finally corresponds to. This completes the entire positioning process from the index to the underlying storage carrier, ensuring that the display queue can accurately read the required audio and video frame data.

[0066] Based on the reference counting module of the shared frame system, the resource occupancy status of the physical storage area is verified to obtain the valid frame data within the storage area. This specifically includes the following steps: The current frame data is obtained by retrieving audio and video frame data from the physical storage area. Based on the timestamp manager of the shared frame system, the temporal integrity and synchronization of the current frame data are verified to obtain a valid frame. The reference frame is obtained by verifying the resource occupancy status of valid frames using the reference counting module; By combining the business adaptation standards for audio and video processing in industrial settings, the integrity and transmission compliance of the baseline frame are verified to obtain valid frame data.

[0067] The system reads audio and video frame data stored in the target physical storage area, defines the read content as the current frame data, and then calls the timestamp manager built into the shared frame system to perform dual checks on the timing integrity and audio-visual synchronization of the current frame data, thereby filtering out valid frames that meet the timing requirements. During the check, the time parameters of video and audio frames are uniformly converted to milliseconds. The calculation formula is: Single frame timing deviation value = Video frame timestamp value - Audio frame timestamp value. The system sets a fixed allowable deviation value. If the absolute value of the timing deviation value is less than or equal to the allowable deviation value, it means that the audio and video timing synchronization meets the standard. At the same time, the timestamp continuity of the frame sequence is checked. If the difference between the timestamps of two adjacent frames remains within a fixed range, the timing is considered complete. Frame data that meets both standards is retained as valid frames, and other frames are directly discarded.

[0068] The reference counting module is activated to verify the resource occupancy status of the filtered valid frames. The reference counting module records the real-time occupancy count value of the physical storage area corresponding to the frame. The calculation formula is: total occupancy count value = display service reference value + remote push stream service reference value. When the total occupancy count value is greater than 0, it means that there is still a service call requirement for the frame data and the storage resources are occupied. The frame will be marked as the baseline frame. If the total occupancy count value is equal to or less than 0, it means that there is no service occupation and the frame data has expired. It will be directly excluded and will not enter the next layer of verification process. For example, if the total occupancy count value of a frame is 2, it corresponds to two services reading the frame at the same time, which meets the occupancy status standard and is retained as the baseline frame.

[0069] The business adaptation standards corresponding to the audio and video processing in the industrial field are retrieved, and the obtained reference frames are subjected to dual verification. The first verification is the frame integrity verification, which is performed by subtracting the actual byte value of the frame from the standard frame byte value. If the difference between the actual byte value of the frame and the standard frame byte value is within the allowable fluctuation range, the frame content is determined to be complete. The second verification is the transmission compliance verification, which checks whether the frame encoding format and sampling parameters match the industrial transmission specifications. The reference frame that passes both verifications is the valid frame data in the final storage area.

[0070] An industrial field audio and video dual-channel real-time processing and control system includes: Data acquisition module: Acquires audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data; System construction module: Constructs a shared frame system, which includes a frame buffer pool, a reference counting module, and a timestamp manager; Data matching module: Inputs the original video frame data and the original audio frame data into the physical buffer in the frame buffer pool, and matches the timestamps in the original video frame data and the original audio frame data based on the timestamp manager to obtain the audio and video frame data; Queue creation module: Creates mutually decoupled display consumption queues and remote push streaming consumption queues, and matches the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices; Adaptation module: Determines the corresponding audio and video frame data based on the queue index of the display consumption queue, and adapts the corresponding audio and video frame data based on the hardware image processing unit to obtain the display terminal; Judgment Module: Determines the status characteristics of the remote push streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped; if the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote push streaming terminal. Results output module: Integrates the display terminal and remote streaming terminal to obtain the on-site audio and video processing results.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for real-time processing and control of dual-channel audio and video in industrial settings, characterized in that, The method includes the following specific steps: Collect audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data; A shared frame system is constructed, which includes a frame buffer pool, a reference counting module, and a timestamp manager. The original video frame data and the original audio frame data are input into the physical buffer in the frame buffer pool. The audio and video frame data are obtained by matching the timestamps in the original video frame data and the original audio frame data based on the timestamp manager. Establish mutually decoupled display consumption queues and remote push streaming consumption queues, and match the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices; The corresponding audio and video frame data is determined based on the queue index of the display consumption queue, and the display terminal is obtained by adapting the corresponding audio and video frame data based on the hardware image processing unit. The system determines the status characteristics of the remote streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped. If the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote streaming terminal. The integrated display terminal and remote streaming terminal obtain the on-site audio and video processing results.

2. The industrial site audio and video dual-channel real-time processing and control method according to claim 1, characterized in that, Constructing a shared frame system, which includes a frame buffer pool, a reference counting module, and a timestamp manager, involves the following steps: A frame buffer pool is constructed based on contiguous physical memory, and independent storage areas are allocated to hold audio and video frame data collected in the industrial field. A reference counting module is constructed based on the frame buffer pool. The independent reference status of each storage area in the frame buffer pool is matched based on the reference counting module. The business call status of frame data in each storage area is collected to obtain the module operation results. Based on the module's execution results, the timestamp manager is matched with all frame data in the frame buffer pool to obtain timing reference data; Timing calibration of the entire frame data based on the timing reference data yields timing-normalized audio and video frame data; Based on the time-ordered audio and video frame data and the independent reference status of the reference counting module, the storage resources of the frame buffer pool are managed and controlled. Storage areas without business calls are reclaimed and new frame data is reused to obtain a dynamically iterative shared frame system.

3. The industrial field audio and video dual-channel real-time processing and control method according to claim 2, characterized in that, The original video frame data and original audio frame data are input into the physical buffer of the frame buffer pool. The audio and video frame data are obtained by matching the timestamps in the original video frame data and original audio frame data based on the timestamp manager. The specific steps include: The timing baseline is obtained by timing matching the original video frame data and the original audio frame data in the frame buffer pool according to the timestamp manager. Based on the time series reference, the video frames and audio frames corresponding to the time series dimension are selected to obtain the target audio and video frames. The time series deviation is checked on the selected target audio and video frames to obtain the valid video frames and valid audio frames. The valid video frames and valid audio frames are fused together to obtain valid audio and video frames. The frame data storage area of ​​the valid audio and video frames is determined by the reference counting module to obtain the audio and video frame data.

4. The industrial field audio and video dual-channel real-time processing and control method according to claim 3, characterized in that, Establishing a decoupled display consumption queue and a remote push streaming consumption queue includes the following steps: The service processing links are divided according to the audio and video frame data. The service processing links include screen display services and remote network streaming services. By matching the screen display service with the remote network streaming service through the corresponding data consumption channels, we obtain the display consumption link and the remote streaming consumption link. Based on the resource management mechanism of the shared frame system, corresponding resource scheduling permissions are matched for the display consumption queue and the remote push streaming consumption queue; The business functions of the display consumption link and the remote push stream consumption link are solidified to form an independently operating display consumption queue and a remote push stream consumption queue.

5. The industrial field audio and video dual-channel real-time processing and control method according to claim 4, characterized in that, The process involves matching the displayed consumption queue with the remote push streaming consumption queue and then matching them to the physical buffer to obtain the queue index. This includes the following steps: Match the physical storage area in the frame buffer pool based on the audio and video frame data; Divide the storage resource characteristics of display services and remote streaming services in each physical storage area, and connect the mutually decoupled display consumption queue and remote streaming consumption queue according to the storage resource characteristics; The resource mapping relationship is obtained by mapping and associating the display consumption queue, remote push streaming consumption queue, and corresponding physical storage areas. The queue index is formed by generating location identifiers based on the resource mapping relationship.

6. The industrial field audio and video dual-channel real-time processing and control method according to claim 5, characterized in that, The audio and video frame data is determined based on the queue index of the displayed consumption queue, specifically including the following steps: The corresponding physical storage area within the frame buffer pool is determined based on the queue index of the displayed consumption queue. Based on the reference counting module of the shared frame system, the resource occupancy status of the physical storage area is verified to obtain the valid frame data within the storage area; Based on the resource mapping relationship of the queue index, filter the number of valid frames in the physical storage area; Extract the valid frame data corresponding to the display scene to obtain audio and video frame data.

7. The industrial field audio and video dual-channel real-time processing and control method according to claim 5, characterized in that, Matching the physical storage area in the frame buffer pool based on audio and video frame data includes the following steps: Match audio and video frame data with corresponding storage attribute characteristics, and determine frame data storage attributes based on the resource management mechanism of the shared frame system; The storage attribute characteristics of the acquired frame data are compared with the physical storage areas in the frame buffer pool that are in an idle state. The storage compatibility between the frame data storage attributes and the storage of each physical storage area is then selected to identify the target physical storage area.

8. The industrial field audio and video dual-channel real-time processing and control method according to claim 6, characterized in that, Based on the queue index of the displayed consumption queue, the corresponding physical storage area within the frame buffer pool is determined, specifically including the following steps: The frame data attributes are obtained by filtering the storage adaptation attributes and business adaptation attributes in the audio and video frame data. Based on the extracted frame data attributes, match all initialized physical storage areas in the frame buffer pool, verify the space specifications and resource adaptation attributes of each physical storage area, and filter out the storage areas that are suitable for the current audio and video frame data storage requirements. Combine the reference counting module of the shared frame system to match the valid storage area of ​​the current frame data; The physical storage area is obtained by determining the target storage area for storing and retrieving appropriate audio-visual frame data based on the resource characteristics of the effective storage area.

9. A method for real-time processing and control of dual-channel audio and video in industrial settings according to claim 8, characterized in that, Based on the reference counting module of the shared frame system, the resource occupancy status of the physical storage area is verified to obtain the valid frame data within the storage area. This specifically includes the following steps: The current frame data is obtained by retrieving audio and video frame data from the physical storage area. Based on the timestamp manager of the shared frame system, the temporal integrity and synchronization of the current frame data are verified to obtain a valid frame. The reference frame is obtained by verifying the resource occupancy status of valid frames using the reference counting module; By combining the business adaptation standards for audio and video processing in industrial settings, the integrity and transmission compliance of the baseline frame are verified to obtain valid frame data.

10. An industrial site audio and video dual-channel real-time processing and control system, applied to the industrial site audio and video dual-channel real-time processing and control method described in claims 1-9, characterized in that, include: Data acquisition module: Acquires audio and video signals output from industrial field equipment to generate raw video frame data and raw audio frame data; System construction module: Constructs a shared frame system, which includes a frame buffer pool, a reference counting module, and a timestamp manager; Data matching module: Inputs the original video frame data and the original audio frame data into the physical buffer in the frame buffer pool, and matches the timestamps in the original video frame data and the original audio frame data based on the timestamp manager to obtain the audio and video frame data; Queue creation module: Creates mutually decoupled display consumption queues and remote push streaming consumption queues, and matches the display consumption queues and remote push streaming consumption queues to the physical buffer to obtain queue indices; Adaptation module: Determines the corresponding audio and video frame data based on the queue index of the display consumption queue, and adapts the corresponding audio and video frame data based on the hardware image processing unit to obtain the display terminal; Judgment Module: Determines the status characteristics of the remote push streaming consumption link. If the status characteristics are in a dormant state, the processing of audio and video frame data is stopped; if the status characteristics are in an active state, the audio and video frame data is processed according to the queue index and hardware encoding unit to obtain the remote push streaming terminal. Results output module: Integrates the display terminal and remote streaming terminal to obtain the on-site audio and video processing results.

Images