A method and system for low-latency audio-video synchronization intercom in a cage

Abstract

The embodiment of the application provides a cage audio and video low-delay synchronous intercom method and system, which is applied to the technical field of mine hoisting communication and audio and video synchronous transmission, and through association of cage running state data, audio frame data, video frame data and link quality data according to a unified time identifier, determination of a section synchronization risk level according to an audio and video arrival offset, an offset direction and a transmission queue state and adjustment of a corresponding transmission queue, feedback calibration is carried out according to an actual ground playing time; when a main link is abnormal, audio frames, video key frames and device state data are cached, after the main link is restored, cached data with a high event level is first supplemented, cached data with the same level and an early occurrence time is first supplemented, and the supplementing only uses a remaining part of a current available bandwidth after deduction of a bandwidth required for real-time audio transmission; the system continuously supplies power through a backup power supply when the main power supply is abnormal. The application can reduce an audio and video actual playing misplacement rate and reduce the influence of cache supplementing on real-time intercom.

Description

Technical Field

[0001] This application relates to the field of mine hoisting communication and audio-visual synchronous transmission technology, and in particular to a method and system for low-latency synchronous audio-visual intercom in a mine cage. Background Technology

[0002] Coal mine hoisting cages are crucial facilities for transporting personnel, equipment, and materials in mines. To meet the needs of dispatching and command, personnel communication, on-site monitoring, and emergency response during cage operation, existing cage communication systems typically install audio and video acquisition devices and communication terminals on the cage side. Audio and video data are transmitted to a ground monitoring platform via wireless or wired communication links for real-time viewing and two-way communication by dispatchers.

[0003] When the cage operates within the shaft, it sequentially undergoes several states, including stopping, acceleration, constant speed operation, deceleration at the dock, and abnormal stopping. The communication link quality varies depending on the location and operating state, easily leading to link latency, jitter, packet loss, and reduced available bandwidth. Existing technologies typically employ fixed buffer durations, fixed video parameters, or uniformly reduce the video bitrate when bandwidth is insufficient. These technologies fail to adequately consider the cage's operating section, the direction of audio and video arrival offset, transmission queue backlog, and the importance of the target audio, potentially causing audio-visual asynchrony, intercom delays, or missing key visuals.

[0004] Existing technologies include measures to mitigate anomalies such as backup power takeover, reducing video transmission load, local caching, and uploading after link recovery. However, these measures primarily maintain power supply during anomalies or preserve data that has not yet been uploaded, failing to establish a coordinated adjustment relationship between audio / video arrival offset, transmission queue status, actual ground playback offset, and recovery / retransmission bandwidth. When fluctuations occur in the shaft link, audio and video frame data may experience lags in different directions within their respective transmission queues and playback paths. After the main link recovers, retransmission of cached data may further consume real-time audio intercom bandwidth, increasing intercom latency. Therefore, joint control of the audio / video transmission queue, actual playback offset, and recovery / retransmission process is necessary. Summary of the Invention

[0005] This application provides a method and system for low-latency synchronous audio and video intercom in a cage, mainly addressing the technical problem that fluctuations in the shaft link cause audio and video frame data to lag in different directions in the transmission queue and playback path, leading to bandwidth contention between the buffered retransmission after the main link recovers and real-time audio, resulting in asynchronous audio and video playback on the ground and increased real-time intercom latency. To achieve the above objectives, this application adopts the following technical solution: A method for low-latency synchronous audio and video intercom in a cage, the method comprising: The system acquires cage operation status data, audio frame data, video frame data, equipment status data, and link quality data, and generates a collection cycle synchronization record according to a unified time identifier. Based on the cage operation status data, it determines the operating segment data and generates segment synchronization characteristic data, including audio and video arrival offsets, offset directions, and audio and video transmission queue status, based on the collection cycle synchronization record and link quality data. Based on the segment judgment threshold group corresponding to the operating segment data and the segment synchronization characteristic data, it determines the segment synchronization risk level. According to the segment synchronization risk level and offset direction, it generates synchronization transmission strategy data to adjust the upper limit of the queue length and the upper limit of the queue waiting time for the audio or video transmission queue. The ground monitoring platform plays the audio frame data according to the unified time identifier. Based on the actual playback time of the audio and video frame data, synchronous feedback calibration data is generated. Based on the synchronous feedback calibration data, the compensation buffer duration of the relatively advanced audio or video playback path is increased, and the relatively lagging audio or video transmission queue parameters in the synchronous transmission strategy data are updated. When the main link is abnormal, audio frame data, video key frame data, and device status data are written to the local cache and event cache data is generated. When the main link recovers, retransmission order data is generated according to the event level and occurrence time of the event cache data. The larger value between the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero is taken to obtain the retransmission bandwidth limit. The retransmission parameters are determined based on the retransmission bandwidth limit, and the data in the local cache is retransmitted according to the retransmission order data and the retransmission parameters.

[0006] A low-latency synchronous audio-visual intercom system for a cage includes a cage-side edge control terminal, a ground monitoring platform, a main link connecting the two, and a backup power supply. The backup power supply is connected to the cage-side edge control terminal and is used to take over power supply when the main power supply fails. The cage-side edge control terminal includes a data association module, a segment determination module, a feature generation module, a risk determination module, a strategy generation module, a local caching module, a strategy update module, and an evidence transmission module. The ground monitoring platform includes a synchronization feedback module. The data association module generates synchronous records for the acquisition cycle according to a unified time identifier. The segment determination module determines the operating segment data based on the cage's operating status data. The feature generation module generates segment synchronization feature data including audio-visual arrival offset, offset direction, and transmission queue status. The risk determination module determines the segment synchronization risk level by combining the operating segment data and the segment synchronization feature data. The strategy generation module adjusts the corresponding transmission queue parameters according to the segment synchronization risk level and offset direction. The synchronization feedback module generates synchronization feedback based on the actual playback time of the audio-visual content. The calibration data and strategy update module is used to generate compensation buffer adjustment instructions based on synchronous feedback calibration data and update the relatively lagging transmission queue parameters. The synchronous feedback module is also used to increase the compensation buffer duration of the relatively advanced playback path according to the compensation buffer adjustment instructions. The local cache module is used to generate event cache data after the backup power supply takes over or when the main link is abnormal. The strategy update module is also used to select data with an exception type of main link abnormality and an uploading confirmation mark from the event cache data when the main link is restored, or to select data with an exception type of main power abnormality or main power and main link combined abnormality and an uploading confirmation mark from the event cache data when the main power status is restored to normal and the main link is available. Based on the event level and occurrence time of the selected data, the retransmission order data is generated, and the larger value between the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero is taken to obtain the retransmission bandwidth limit. The retransmission parameters are determined based on the retransmission bandwidth limit. The evidence transmission module is used to retransmit the cache data within the retransmission bandwidth limit according to the retransmission order data and the retransmission parameters.

[0007] As can be seen from the above technical solution, this application has the following beneficial effects: 1. This application associates audio frame data and video frame data based on a unified time identifier, determines the segment synchronization risk level according to the arrival offset, offset direction and transmission queue status of audio and video, and adjusts the transmission queue parameters corresponding to the relatively lagging data stream according to the offset direction, thereby reducing the continuous arrival misalignment of audio frame data and video frame data caused by queue backlog differences.

[0008] 2. This application generates synchronous feedback calibration data based on the actual playback time of audio and video recorded by the ground monitoring platform, adds compensation buffer time to the relatively advanced playback path, and updates the relatively lagging transmission queue parameters, thereby simultaneously processing the already formed playback offset and the queue backlog that causes subsequent offsets, forming a synchronous feedback closed loop from the transmission end to the actual playback end.

[0009] 3. When the main link is restored, or the main power supply is restored to normal and the main link is available, this application selects the event cache data to be retransmitted according to the type of abnormality, and determines the retransmission order according to the event level and occurrence time. Only the remaining part of the currently available bandwidth after deducting the bandwidth required for real-time audio transmission is used to retransmit the cache data, which can reduce the bandwidth occupation of real-time intercom for cache retransmission; the backup power supply provides a continuous power supply basis for cache and policy updates during the main power supply abnormality. Attached Figure Description

[0010] The present application will be further described below with reference to the accompanying drawings.

[0011] Figure 1 A flowchart illustrating a low-latency synchronous audio and video intercom method for cages provided in this application embodiment; Figure 2 Flowchart for determining the segment synchronization risk level and generating the synchronization transmission strategy provided in the embodiments of this application; Figure 3 This is a schematic diagram of a low-latency synchronous audio and video intercom system for cages, provided as an embodiment of this application. Detailed Implementation

[0012] The terms "first," "second," and "third," etc., used in this application specification, claims, and drawings are used to distinguish different objects, not to limit a specific order.

[0013] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0014] Research has revealed that existing cage-based audio-visual intercom systems typically employ fixed synchronization buffer parameters and fixed audio-visual transmission strategies. They fail to differentiate between different operational phases, such as cage docking, acceleration, constant speed operation, deceleration at the dock, and abnormal stops, to address audio-visual arrival offsets, link fluctuations, and transmission queue status. When the shaft communication link experiences increased latency, jitter, packet loss, or bandwidth degradation, audio-visual misalignment, target voice transmission delays, and missing video keyframes can easily occur. Furthermore, existing systems lack feedback calibration based on actual ground playback time, as well as a mechanism for hierarchical caching and orderly retransmission of critical data in the event of main power supply or main link anomalies, making it difficult to balance real-time intercom continuity with the integrity of abnormal data.

[0015] To address the aforementioned issues, this application provides a method and system for low-latency synchronous audio and video intercom within a cage: Example: like Figures 1-3 As shown, this embodiment relates to a method and system for low-latency synchronous audio and video intercom in coal mine hoisting cages, which is particularly suitable for real-time scheduling intercom and video monitoring scenarios where communication link fluctuations, audio and video transmission latency changes, and main power supply or main link abnormalities may occur during the operation of coal mine hoisting cages in the shaft.

[0016] Specific implementation steps: S1. Acquire cage operating status data, audio frame data, video frame data, equipment status data, and link quality data, and generate a synchronous record of the acquisition cycle according to a unified time identifier; In this embodiment, step S1 includes the following process: Acquire cage operation status data including cage position, running speed, acceleration, running direction, and door opening / closing status; acquire audio frame data including audio frame number, audio acquisition time, audio arrival time, voice activity identifier, distress call identifier, and dispatch voice identifier; when the distress call identifier is valid, write the target voice identifier into the corresponding audio frame data and determine the target voice type as distress call voice; when the distress call identifier is not valid but the dispatch voice identifier is valid, write the target voice identifier into the corresponding audio frame data and determine the target voice type as dispatch voice; when both the distress call identifier and the dispatch voice identifier are not valid but the voice activity identifier is valid, write the target voice identifier into the corresponding audio frame data. Voice identification is performed, and the target voice type is determined as ordinary voice. Video frame data including video frame number, video acquisition time, video arrival time, and keyframe identifier are acquired. Video frame data with keyframe identifiers are identified as video keyframe data, and the remaining video frame data are identified as ordinary video frame data. Device status data including main power supply status, backup power supply status, communication status, storage status, and device online status are acquired. Link quality data including link latency, link jitter, packet loss rate, signal strength, and available bandwidth are acquired. Audio frame data, video frame data, cage operation status data, device status data, and link quality data within the same acquisition cycle are associated with a unified time identifier to generate a synchronous record for the acquisition cycle.

[0017] Specifically, an edge control terminal is installed on the cage side, which is connected to the hoist control system, explosion-proof audio acquisition terminal, explosion-proof video acquisition terminal, power management unit, communication unit, and local storage unit. A ground monitoring platform is installed on the ground side, and the ground monitoring platform establishes a data communication connection with the edge control terminal on the cage side.

[0018] The edge control terminal divides the data acquisition cycle into continuous and non-overlapping cycles according to a preset acquisition cycle length. The preset acquisition cycle length, as a system configuration parameter, is set during the device debugging phase based on audio frame duration, video frame interval, and link quality detection cycle, and then written into the edge control terminal's configuration file. A single acquisition cycle can contain multiple audio frame data and multiple video frame data. Cage operating status data, device status data, and link quality data are written into the corresponding acquisition cycle according to their respective acquisition times.

[0019] A unified time identifier refers to a time base shared by the edge control terminal on the cage side and the ground monitoring platform. The edge control terminal and the ground timing equipment use a precise time protocol for clock calibration. After calibration, the edge control terminal writes the acquisition time based on the same clock for each piece of data collected, and writes the acquisition period number according to the acquisition period in which the acquisition time falls. When the communication link is temporarily interrupted, the edge control terminal uses the calibrated local crystal oscillator for continuous timing; after the communication link is restored, it corrects the subsequent time identifier based on the clock offset returned by the ground timing equipment, without modifying the original acquisition time, and simultaneously writes a clock calibration status to the corresponding record to preserve the actual time relationship during data acquisition.

[0020] The cage's operating status data is acquired through the hoist control system and the cage door control device. The hoist control system includes a programmable logic controller (PLC). The edge control terminal reads the register data corresponding to the cage position, running speed, acceleration, and running direction stored in the PLC via an industrial communication interface. The cage position is the position data of the cage relative to the preset origin of the shaft coordinate system along the running direction; the running direction is determined according to the upward or downward control status output by the hoist control system; the door opening / closing status is acquired through the door opening and closing signals from the cage door controller. When the door opening signal is valid, the open state is written; when the door closing signal is valid, the closed state is written; when both signals are invalid, a door status transition flag is written. The edge control terminal correlates the cage position, running speed, acceleration, running direction, door opening / closing status, and their acquisition time to form the cage operating status data.

[0021] The range of docking positions is determined during the system commissioning phase based on the designed docking positions at each wellhead or intermediate level and the detection error of the cage position. This range is then written into the position configuration table as upper and lower limits for each docking position. Each docking position range in the position configuration table is associated with a corresponding docking position number, which is used when determining the docking section and deceleration section.

[0022] Audio frame data is acquired through an explosion-proof microphone, audio acquisition circuit, and audio encoder. The explosion-proof microphone collects sound from inside the cage, and an analog-to-digital converter converts the sound signal into a digital audio signal. The audio encoder divides the digital audio signal into continuous audio frames according to a preset audio frame duration. The edge control terminal sequentially writes the audio frame number, audio acquisition time, and audio data content to each audio frame.

[0023] The audio arrival time is written by the ground monitoring platform when the corresponding audio frame enters the audio receiving buffer. The ground monitoring platform writes the audio frame number, audio arrival time, and acquisition cycle number into the receiving receipt and sends the receiving receipt to the edge control terminal; the edge control terminal then adds the audio arrival time to the corresponding acquisition cycle synchronization record based on the audio frame number. When the segment synchronization feature data is generated by the ground monitoring platform, the ground monitoring platform directly reads the audio arrival time recorded in the audio receiving buffer and does not perform receipt addition.

[0024] The edge control terminal performs existing speech activity detection on the audio frame data. When the detection result shows that there are continuous speech components in the corresponding audio frame, a valid speech activity identifier is written for the audio frame; when the detection result shows that the corresponding audio frame is mainly stable background noise, an invalid speech activity identifier is written for the audio frame.

[0025] The edge control unit utilizes keyword detection to identify distress keywords in audio frames with valid voice activity identifiers. A keyword configuration table pre-stores keywords corresponding to target distress semantics such as distress call, life-saving, parking, and malfunction. When the keyword recognition result of an audio frame matches the target distress semantic in the keyword configuration table, a valid distress identifier is written.

[0026] The dispatch voice identifier does not rely on voice content recognition, but is determined based on the data source of the audio frame. When the audio frame is sent from the dispatch intercom channel of the ground monitoring platform to the edge control terminal on the cage side, the edge control terminal writes a valid dispatch voice identifier according to the dispatch channel number in the data packet; when the audio frame originates from the explosion-proof microphone on the cage side, the dispatch voice identifier is invalid.

[0027] When the distress call flag is valid, the edge control terminal writes the target voice flag to the corresponding audio frame and determines the target voice type as distress call voice. When the distress call flag is invalid but the dispatch voice flag is valid, the edge control terminal writes the target voice flag to the corresponding audio frame and determines the target voice type as dispatch voice. When both the distress call flag and the dispatch voice flag are invalid but the voice activity flag is valid, the edge control terminal writes the target voice flag to the corresponding audio frame and determines the target voice type as normal voice. By determining the target voice type in the order of distress call voice, dispatch voice, and normal voice, multiple target voice types are avoided from being written to the same audio frame simultaneously.

[0028] Video frame data is acquired via explosion-proof cameras and video encoders. The explosion-proof cameras continuously capture video images of the interior of the cage, the cage door area, and the area where personnel stand. The video encoder encodes the video images into video frames according to a preset video frame rate. The edge control terminal writes a video frame number, video acquisition time, and keyframe identifier to each video frame.

[0029] Keyframe identifiers are directly output by the video encoder. The video encoder marks I-frames and InstantaneousDecoder Refresh frames as video keyframes and sets the corresponding keyframe identifiers to a valid state; video frames that depend on preceding and following video frames for decoding are marked as ordinary video frames, and their corresponding keyframe identifiers are set to an invalid state. The edge control terminal determines the video frame data with valid keyframe identifiers as video keyframe data, and determines the remaining video frame data as ordinary video frame data.

[0030] The video arrival time is written by the ground monitoring platform when the corresponding video frame enters the video receiving buffer. The ground monitoring platform writes the video frame number, video arrival time, and acquisition cycle number into the receiving receipt, and then adds them to the corresponding acquisition cycle synchronization record according to the video frame number. Both audio and video arrival times are recorded using the unified clock of the ground monitoring platform to avoid false audio and video offsets caused by clock errors from different receiving devices.

[0031] Equipment status data includes main power status, backup power status, communication status, storage status, and equipment online status. The main power status and backup power status are acquired by the power management unit. The power management unit collects the voltage status, power supply channel status, and switching status of the main power input and backup power output terminals, and generates normal or abnormal flags based on whether the power supply channel is within the normal output range. Communication status is determined by the network interface connection status of the communication unit and the heartbeat response status of the ground monitoring platform; when the network interface is disconnected or no heartbeat response is received continuously, a communication abnormal status is written. Storage status is determined by the remaining storage capacity, write status, and read status returned by the local storage unit; when the remaining capacity is lower than a preset capacity threshold or the write test fails, a storage abnormal status is written. Equipment online status is determined by the equipment heartbeat data sent by each acquisition terminal according to a preset heartbeat cycle; if no equipment heartbeat data is received within a preset offline judgment period, the corresponding equipment is written to the offline status.

[0032] Link quality data is jointly acquired by the communication unit and the ground monitoring platform, including link latency, link jitter, packet loss rate, signal strength, and available bandwidth. The edge control unit sends link detection data packets with transmission time and consecutive sequence numbers according to a preset detection period. The ground monitoring platform records the reception time and returns a reception acknowledgment. Link latency is obtained from the difference between the reception time and the transmission time; link jitter is obtained from the variation in link latency between consecutive detection data packets; packet loss rate is obtained from the ratio between the number of missing packets in consecutive sequence numbers and the number of packets sent; signal strength is read through the communication module driver interface; available bandwidth is obtained by dividing the amount of successfully confirmed data within a preset detection time window by the length of the detection time window. All of the above link quality data are associated with corresponding detection times and acquisition period numbers.

[0033] The acquisition cycle synchronization record refers to a data record used to represent the time correspondence between audio frame data, video frame data, cage operation status data, equipment status data, and link quality data within the same acquisition cycle. The edge control terminal uses the acquisition cycle number and a unified time identifier as the primary key for the record, writing data acquired between the start and end times of that acquisition cycle into the same acquisition cycle synchronization record. Because the audio frame rate, video frame rate, and status data acquisition frequencies differ, the acquisition cycle synchronization record separately stores lists of audio frame numbers, video frame numbers, and the most recently acquired cage operation status data, equipment status data, and link quality data within that acquisition cycle. If a certain type of data is not acquired within the corresponding acquisition cycle, a data missing flag is written to the corresponding field for that type of data; data from the previous acquisition cycle is not used to directly replace it.

[0034] S2. Determine the operating section data based on the cage's operating status data; In this embodiment, step S2 includes the following process: The system correlates the cage position, running speed, acceleration, running direction, and door opening / closing status along the time axis to generate running status change data. When the cage position is within the docking position range and the running speed is lower than the preset stationary speed threshold, it is identified as a docking section, and a person boarding / alighting identifier is written for the docking section when the door is in the open state. When the cage leaves the docking position range and the running speed increases for a preset number of consecutive collection cycles, it is identified as a start-up acceleration section. When the running speed is within the preset speed fluctuation range for a preset number of consecutive collection cycles, it is identified as a constant speed running section. When the cage enters the docking position range and the running speed decreases for a preset number of consecutive collection cycles, it is identified as a docking deceleration section. When the cage is outside the docking position range, the running speed is lower than the preset stationary speed threshold, and the duration exceeds the preset parking time, it is identified as an abnormal parking section. The system correlates the section type, section start time, section end time, running direction, and person boarding / alighting identifier to generate running section data.

[0035] Specifically, operating segment data refers to the data formed by dividing the continuous operation process according to the actual operating status of the cage. Each operating segment data includes at least the segment type, segment start time, segment end time, operating direction, and personnel boarding / alighting indicators. Segment types include stopping segments, acceleration segments, constant speed operating segments, deceleration segments, and abnormal stopping segments.

[0036] The edge control terminal sorts the synchronous records of the acquisition cycle according to a unified time identifier, reads the cage position, running speed, acceleration, running direction and door opening and closing status from the synchronous records of each acquisition cycle, and matches the data of the current acquisition cycle with the data of the previous acquisition cycle to form the running status change data.

[0037] The operational status change data includes the direction of cage position change, speed increase, speed decrease, stable speed, change in direction, and door opening / closing status changes between adjacent acquisition cycles. Speed ​​increase is determined by whether the current acquisition cycle's speed is greater than the previous acquisition cycle's speed; speed decrease is determined by whether the current acquisition cycle's speed is less than the previous acquisition cycle's speed; and stable speed is determined by whether the speed within consecutive acquisition cycles remains within a preset speed fluctuation range.

[0038] The preset stationary speed threshold is used to distinguish between the actual parking state and zero-point fluctuations caused by position detection or speed detection. During the equipment commissioning phase, the cage is kept stationary at each docking position, and the operating speed data is continuously collected. The maximum absolute value of the operating speed in the stationary state is added to the preset safety margin to obtain the preset stationary speed threshold, which is then written into the operating section configuration table.

[0039] The preset speed fluctuation range is determined based on the operating speed data during the normal uniform speed operation phase of the cage. During the equipment commissioning phase, normal uniform speed operation records without stopping, acceleration, or deceleration are selected, and the upper and lower speed limits within these records are obtained. A preset speed margin is then added outside the upper and lower speed limits to form the preset speed fluctuation range.

[0040] The preset stopping time is determined based on the duration of the maximum short-term speed fluctuation during normal hoist control and the requirements for confirming abnormal stops. The preset stopping time should be greater than the duration of short-term low speed caused by normal reversing, speed adjustment, or vibration detection, and should be entered into the operating section configuration table.

[0041] A preset number of consecutive acquisition cycles is used to filter out sudden speed changes within a single acquisition cycle. The preset number of consecutive acquisition cycles is determined based on a preset minimum duration and a preset acquisition cycle length, using the following formula: ; In the formula, This represents a preset number of consecutive data collection cycles; This represents the preset minimum duration of the corresponding operating state; This represents the preset data collection period length; This represents the floor function. The larger the preset minimum duration, the more consecutive preset collection cycles there are, and the higher the tolerance of the running segment determination for short-term speed fluctuations; the larger the preset collection cycle length, the fewer consecutive preset collection cycles there are when the preset minimum duration remains unchanged.

[0042] The preset minimum duration is set during the equipment commissioning phase for the startup acceleration state, constant speed state, and station deceleration state. During setting, the state transition records during normal lifting are read, the upper limit of the duration of short-term speed fluctuations to be filtered is taken, and an acquisition cycle length is added to this upper limit to obtain the preset minimum duration for the corresponding operating state. This setting method ensures that data fluctuations with durations shorter than the normal state transition time do not directly form new operating segments.

[0043] The edge control terminal first compares the cage position with the various docking position ranges in the position configuration table. When the cage position is within any docking position range and the running speed is lower than the preset stationary speed threshold, the current state is determined as a docking state. When the door opening / closing state corresponding to the docking state is open, a loading / unloading identifier is written for the docking state. When the docking state is continuously maintained, the start time of the first collection cycle that meets the docking state is determined as the start time of the docking segment. When the cage leaves the corresponding docking position range or the running speed is no longer lower than the preset stationary speed threshold, the end time of the previous collection cycle is determined as the end time of the docking segment, thus generating the docking segment.

[0044] When the cage position changes from within the docking position range to outside the docking position range, and the running speed increases continuously for a preset number of data acquisition cycles, the edge control terminal determines the start time of the data acquisition cycle in which the continuous speed increase first occurs as the start time of the acceleration segment. When the running speed stops increasing continuously and enters the preset speed fluctuation range, or when the cage enters the docking position range, the end time of the previous data acquisition cycle is determined as the end time of the acceleration segment, thus generating the acceleration segment.

[0045] When the operating speed remains within a preset speed fluctuation range for a consecutive preset number of acquisition cycles, and the cage is not within the stopping or docking position range, the edge control terminal determines the corresponding state as a uniform speed operation state. The start time of the acquisition cycle that first satisfies the uniform speed operation state is determined as the start time of the uniform speed operation segment; when the operating speed leaves the preset speed fluctuation range or the cage enters the docking position range, the end time of the previous acquisition cycle is determined as the end time of the uniform speed operation segment, thus generating the uniform speed operation segment.

[0046] The docking position range is determined by the target docking position and the docking deceleration distance, and is written into the position configuration table. The docking deceleration distance is determined during the equipment commissioning phase based on the normal deceleration curve of the hoist and the maximum allowable docking error. When the cage enters the docking position range corresponding to the target docking position, and the running speed decreases for a preset number of consecutive acquisition cycles, the edge control terminal determines that the cage has entered the docking deceleration state. The start time of the acquisition cycle that first meets the docking deceleration state is determined as the start time of the docking deceleration section; when the cage enters the docking position range and the running speed is lower than the preset stationary speed threshold, the end time of the previous acquisition cycle is determined as the end time of the docking deceleration section, and the docking deceleration section is generated.

[0047] When the cage is outside the designated stopping position range, its operating speed is below the preset stationary speed threshold, and the duration of the low-speed state exceeds the preset stopping time, the edge control terminal determines that the cage has entered an abnormal stopping state. The start time of the abnormal stopping section is the start time of the data collection cycle corresponding to the first occurrence of the low-speed state, not the current time when the preset stopping time has been exceeded; when the operating speed exceeds the preset stationary speed threshold again or the cage enters the stopping position range, the end time of the previous data collection cycle is determined as the end time of the abnormal stopping section, and the abnormal stopping section is generated.

[0048] When multiple operating segment determination conditions are met simultaneously within the same data acquisition cycle, the edge control terminal determines a unique segment type in the order of abnormal parking segment, docking segment, deceleration segment, acceleration segment, and constant speed operating segment. Abnormal parking segments are prioritized over other segments because continuous low-speed states outside the docking position range need to be preserved; docking segments are prioritized over acceleration and constant speed operating segments to avoid minor positional fluctuations of the cage near the docking position that could create erroneous operating segments.

[0049] The running direction is directly read from the running direction output by the hoist control system within the current running section. When the running direction changes within the same running section, the original running section is split into two running sections, with the acquisition period corresponding to the change in running direction as the boundary, ensuring that the data of each running section corresponds to only one running direction.

[0050] The edge control terminal associates the final determined segment type, segment start time, segment end time, running direction, and personnel boarding / alighting identifiers to generate running segment data, and establishes an index relationship between the running segment data and the entire collection cycle covered by the segment.

[0051] S3. Generate segment synchronization feature data containing audio and video arrival offset, offset direction, link fluctuation and queue status based on the acquisition cycle synchronization record and link quality data; In this embodiment, step S3 includes the following process: For each video frame within the same acquisition period, the audio frame with the closest video acquisition time to the audio acquisition time is selected to generate an audio-video corresponding frame group. The audio and video arrival offset is determined based on the audio and video arrival times in the corresponding frame group, and the offset direction is determined according to their arrival order. The changes in link latency, link jitter, packet loss rate, and available bandwidth between adjacent acquisition periods are determined to generate link fluctuation data. Audio queue status data and video queue status data are generated based on the queue length and waiting time of the audio and video transmission queues. It is determined whether video frame data with keyframe identifiers arrives within the preset arrival time limit, generating video keyframe arrival status data. According to the running segment data, the audio-video arrival offset, offset direction, link fluctuation data, audio queue status data, video queue status data, target voice identifier, and video keyframe arrival status data are correlated to generate segment synchronization feature data.

[0052] Specifically, segment synchronization characteristic data refers to data formed by organizing the audio and video transmission and reception status according to the operating segment data. It includes at least audio and video arrival offset, offset direction, link fluctuation data, audio queue status data, video queue status data, target voice identifier, and video keyframe arrival status data. Segment synchronization characteristic data does not directly indicate the segment synchronization risk level, but rather serves as the data basis for subsequently calling the segment judgment threshold group and determining the segment synchronization risk level.

[0053] The edge control terminal or ground monitoring platform reads the synchronization record of the acquisition cycle according to the acquisition cycle number. When the segment synchronization feature data is generated on the ground monitoring platform, the audio arrival time, video arrival time and receiving queue data recorded by the ground monitoring platform are directly read. When the segment synchronization feature data is generated on the edge control terminal, the ground monitoring platform returns the audio and video arrival time, receiving queue status and video keyframe arrival status according to the acquisition cycle. The edge control terminal then writes the returned data into the corresponding acquisition cycle synchronization record and performs subsequent processing.

[0054] For each video frame within the same acquisition period, the video acquisition time of that video frame is read, and the audio frame data within the same acquisition period is traversed. The absolute time difference between the video acquisition time and each audio acquisition time is calculated. The audio frame data with the smallest absolute time difference is determined as the corresponding audio frame data. The video frame data and the corresponding audio frame data form an audio-video corresponding frame group.

[0055] An audio-video corresponding frame group refers to a set of audio and video frame data that are associated based on the principle of closest acquisition time. A preset frame association time limit serves as a validity parameter for the audio-video corresponding frame group. It is determined by the audio frame duration in the audio encoding configuration and the video frame interval in the video encoding configuration, taking the larger value between the audio frame duration and the video frame interval, and writing it into the synchronization configuration table. When the minimum absolute time difference does not exceed the preset frame association time limit, the audio-video corresponding frame group is written with a valid correspondence identifier; when the minimum absolute time difference exceeds the preset frame association time limit, or when no audio frame data exists within the same acquisition period, a corresponding frame missing identifier is written. Data with a corresponding frame missing identifier is not used to calculate the audio-video arrival offset, but is stored as a data missing status in the segment synchronization feature data.

[0056] For audio and video corresponding frame groups with valid corresponding identifiers, the audio arrival time of the corresponding audio frame data and the video arrival time of the corresponding video frame data are read respectively. The audio and video arrival offset is determined based on the difference between the two arrival times. The audio and video arrival offset is calculated using the following formula: ; In the formula, This represents the audio / video arrival offset of the k-th audio / video frame group; This represents the video arrival time of the video frame data in the kth audio / video frame group; This represents the audio arrival time of the audio frame data in the k-th audio / video frame group; k represents the sequential number of the corresponding audio / video frame group. The larger the absolute value of the difference between the video arrival time and the audio arrival time, the larger the audio / video arrival offset, indicating a more significant difference in arrival time between the audio and video frame data at the receiving end. The offset direction is determined based on the order of the video and audio arrival times, using the following formula: ; In the formula, This represents the offset direction of the k-th audio / video frame group; Represents a symbolic function; This represents the video arrival time of the video frame data in the kth audio / video frame group; This represents the audio arrival time of the audio frame data in the k-th audio / video frame group. When the value is positive, the video frame data arrives later than the audio frame data, and the offset direction represents the lag between the video frame data and the audio frame data; when... When the value is negative, the audio frame data arrives later than the video frame data, and the offset direction represents the lag between the audio frame data and the video frame data; when When the value is zero, the arrival time of audio frame data and video frame data is the same.

[0057] To avoid the inability to directly represent the degree of change due to the different units used for link latency, link jitter, packet loss rate, and available bandwidth, the changes in each type of link quality parameter between adjacent acquisition periods are processed using dimensionless methods to generate link fluctuation data. The dimensionless relative changes are expressed by the following formula: ; In the formula, This represents the dimensionless relative change in the quality parameters of the q-th link during the k-th acquisition period; The parameter value represents the quality parameter of the q-th link within the k-th acquisition period; The parameter value represents the q-th type of link quality parameter within the (k-1)-th acquisition period; q is used to distinguish link latency, link jitter, packet loss rate, and available bandwidth. This represents the minimum reference value corresponding to the quality parameter of the q-th link, used to avoid an abnormal increase in the relative change when the parameter value in the previous acquisition cycle is zero or close to zero.

[0058] The minimum reference values ​​are set during the equipment commissioning phase based on the minimum effective resolution of the corresponding communication statistics interface. The minimum reference values ​​for link latency and link jitter are the minimum non-zero time that the communication statistics interface can identify; the minimum reference value for packet loss rate is the minimum non-zero packet loss ratio of a detection data packet within a preset detection time window; and the minimum reference value for available bandwidth is the minimum non-zero transmission rate that the communication unit can stably identify. These minimum reference values ​​are written into the link quality configuration table.

[0059] When the link quality parameter value in the previous acquisition period is not lower than the minimum reference value, the larger the parameter difference between adjacent acquisition periods, the larger the dimensionless relative change. When the parameter difference is the same, the larger the parameter value in the previous acquisition period, the smaller the dimensionless relative change, indicating that the relative fluctuation caused by the same absolute change under a larger base value is lower. When the parameter value in the previous acquisition period is less than the minimum reference value, the minimum reference value is used as the denominator to avoid excessive amplification of link fluctuation data by zero or near-zero values.

[0060] The link fluctuation data does not combine link latency, jitter, packet loss rate, and available bandwidth into a single comprehensive score. Instead, it stores the dimensionless relative changes and directions of these four types of link quality parameters separately. Specifically, an increase in link latency, jitter, and packet loss rate is recorded as a deterioration; a decrease in available bandwidth is also recorded as a deterioration; and a decrease in link latency, jitter, and packet loss rate or an increase in available bandwidth is recorded as an improvement. Subsequent segment synchronization risk assessment compares each type of dimensionless relative change with the corresponding threshold in the segment link fluctuation threshold group to avoid the possibility of a certain abnormal link parameter being offset by other normal parameters through fixed-weight synthesis.

[0061] Audio queue status data and video queue status data are read from the audio transmission queue manager and video transmission queue manager, respectively. The transmission queue is a buffer queue at the edge control terminal that temporarily stores data frames according to data transmission priority before sending audio or video frame data. Audio queue status data includes the current audio queue length, the maximum audio queue length, the enqueue time of the first audio frame, and the waiting time for the first audio frame. Video queue status data includes the current video queue length, the maximum video queue length, the enqueue time of the first video frame, and the waiting time for the first video frame.

[0062] The queue length is obtained by reading the number of data frames that have not yet been sent in the corresponding transmission queue. The queue waiting time is obtained by subtracting the enqueue time of the first data frame from the current unified time identifier. When the transmission queue is empty, both the queue length and the queue waiting time are set to zero. Both audio and video queue status data include the queue length and queue waiting time of the corresponding transmission queue; the queue length characterizes the backlog of data frames that have not yet been sent, and the queue waiting time characterizes the transmission lag of the first data frame. Because even when the number of data frames to be sent in the transmission queue is small, the first data frame may still wait for a long time due to link congestion, priority scheduling, or data retransmission, both the queue length and the queue waiting time are recorded simultaneously to avoid missing audio and video transmission lag states based solely on the queue length.

[0063] The arrival status data for video keyframes is generated by the ground monitoring platform based on the keyframe identifier, video frame number, and video arrival time. The preset arrival time limit is set in the synchronization configuration table, with its starting time being the video acquisition time of the video keyframe data. When the ground monitoring platform receives the corresponding video keyframe data within the preset arrival time limit, it writes an "on-time arrival" status; when it receives the corresponding video keyframe data after the preset arrival time limit has expired, it writes a "delayed arrival" status; if the corresponding video keyframe data is not received by the end of the next acquisition cycle, it writes a "not arrived" status. The preset arrival time limit is determined based on the basic synchronization buffer duration and the upper limit of normal link latency, and the set result should be greater than the upper limit of the total time for video keyframe data from acquisition, encoding, queuing to reception under normal link conditions.

[0064] The edge control terminal queries the operating segment data to which each audio and video frame group belongs based on its video capture time, and writes the corresponding segment type, operating direction, and on / off person identifiers into the segment synchronization feature data. When the audio frame data and video frame data are located before and after the operating segment switch, the operating segment to which they belong is determined by the video capture time of the video frame data. This is because subsequent video parameters and video queue parameters mainly affect the video frame data, and using the video capture time can keep the segment synchronization risk judgment consistent with the video transmission adjustment object.

[0065] The data acquisition cycle number, operating segment data, audio and video arrival offset, offset direction, link fluctuation data, audio queue status data, video queue status data, target speech identifier, target speech type, video keyframe arrival status data, and data missing status are correlated to generate segment synchronization feature data. When an operating segment covers multiple acquisition cycles, multiple segments of synchronization feature data are saved in the order of the acquisition cycles. The feature data from different acquisition cycles are not averaged in advance to preserve the instantaneous state when the link suddenly fluctuates, the queue suddenly backlogs, and the target speech appears.

[0066] S4. Based on the running segment data, call the corresponding segment judgment threshold group, and combine the segment synchronization characteristic data to determine the segment synchronization risk level; In this embodiment, step S4 includes the following process: The corresponding segment judgment threshold group is called based on the operating segment data. This group includes the segment synchronization offset threshold, segment link fluctuation threshold, segment bandwidth threshold, and segment video queue threshold. The segment synchronization offset threshold for stopping segments, deceleration segments, and abnormal parking segments with passenger boarding / alighting indicators is lower than the segment synchronization offset threshold for acceleration segments and constant speed operation segments. A unique segment synchronization risk level is determined according to the order of synchronization imbalance level, congestion level, fluctuation level, and normal level. When the audio / video arrival offset exceeds the segment synchronization offset threshold, [the risk level is determined]. The following conditions are met: Synchronization imbalance level; Congestion level; Fluctuation level; Normal level.

[0067] Specifically, the segment judgment threshold group refers to the set of synchronization risk judgment parameters configured separately for different operating segments. Each segment judgment threshold group includes at least the segment synchronization offset threshold, segment link fluctuation threshold group, segment bandwidth threshold, and segment video queue threshold group. The segment judgment threshold group is stored in the segment threshold configuration table at the edge control terminal on the cage side. The segment threshold configuration table uses segment type and personnel entry / exit identifiers as calling indexes.

[0068] The segment types in the segment threshold configuration table correspond one-to-one with stopping segments, acceleration segments, constant speed operation segments, deceleration segments, and abnormal parking segments. Stop segments are further configured with different segment judgment threshold groups based on whether they have passenger pick-up / drop-off indicators. The edge control terminal reads the operating segment data corresponding to the current acquisition cycle, and queries the segment threshold configuration table based on the segment type and passenger pick-up / drop-off indicators to obtain the segment judgment threshold group corresponding to the current operating segment.

[0069] The segment synchronization offset threshold is used to determine whether unacceptable arrival misalignment has occurred in the audio and video frame data within the same audio / video frame group. The segment synchronization offset threshold is obtained during the system debugging phase. During debugging, under normal conditions of the main power supply, main link, and audio / video acquisition equipment, the cage is made to complete the processes of stopping, starting acceleration, constant speed operation, deceleration at the station, and normal stopping, continuously acquiring the audio / video arrival offset corresponding to each operating segment. Acquisition cycles with missing data markers, equipment abnormality markers, or link interruption records are excluded. The remaining audio / video arrival offsets are arranged in ascending order, and the corresponding value is selected according to the preset percentile position in the segment threshold configuration table as the segment synchronization offset threshold for that operating segment.

[0070] The preset percentile positions corresponding to the stop sections with passenger boarding / alighting indicators, deceleration sections, and abnormal parking sections are lower than the preset percentile positions corresponding to the acceleration and constant speed operation sections. This results in stricter section synchronization offset thresholds being applied to the aforementioned operating sections with higher safety concerns. After initial threshold selection, the edge control terminal compares the section synchronization offset thresholds corresponding to each operating section. When the section synchronization offset threshold corresponding to the stop sections with passenger boarding / alighting indicators, deceleration sections, or abnormal parking sections is not less than the section synchronization offset threshold corresponding to the acceleration and constant speed operation sections, the section synchronization offset threshold for the aforementioned high-safety-concern sections is adjusted to a configuration value lower than the smaller threshold between the acceleration and constant speed operation sections. The adjusted threshold is then verified through normal operation playback data to ensure that it does not lead to continuous misjudgments.

[0071] The segment link fluctuation threshold group includes the link delay variation threshold, link jitter variation threshold, packet loss rate variation threshold, and available bandwidth variation threshold. Since step S3 has already performed dimensionless processing on the changes of each link quality parameter between adjacent acquisition periods, all thresholds in the segment link fluctuation threshold group are dimensionless parameters, and there is no issue of direct comparison between different physical quantities.

[0072] The segment link fluctuation threshold group is also obtained through system debugging data. The edge control terminal extracts the changes in link latency, link jitter, packet loss rate, and available bandwidth for each operating segment under normal communication conditions. These are sorted according to parameter type, and the values ​​are read from their respective preset percentile positions to form the segment link fluctuation threshold group. Each link fluctuation data is only compared with the segment link fluctuation threshold of the same type; the changes in link latency, link jitter, packet loss rate, and available bandwidth are not added or weighted together.

[0073] The segment bandwidth threshold refers to the minimum available bandwidth required to maintain the continuous transmission of target voice data, video keyframe data, and device status data indicating anomalies. The segment bandwidth threshold is obtained through communication testing tools. During testing, the average transmission volume of target voice data, video keyframe data, and device status data within a preset statistical time window is statistically analyzed. The average transmission rates of the three types of data are added together, and a preset bandwidth margin for data packet headers, transmission acknowledgments, and short-term retransmissions is added to obtain the segment bandwidth threshold for the corresponding operating segment. Larger preset bandwidth margins are used in stopping segments with passenger boarding / alighting indicators, deceleration segments, and abnormal parking segments to reduce the bandwidth consumption of target voice data and video keyframe data by ordinary video frame data during short-term link fluctuations in these segments.

[0074] The segment video queue threshold group includes a queue length threshold and a queue waiting time threshold for video transmission queues. The queue waiting time threshold is set based on the allowed video transmission waiting time for the corresponding operating segment; the queue length threshold is obtained by multiplying the allowed waiting time by the base video frame rate and rounding the product down. Stopping segments with passenger pick-up / drop-off indicators, deceleration segments, and abnormal parking segments use shorter allowed waiting times, thus obtaining smaller video queue length thresholds and queue waiting time thresholds.

[0075] After the segment judgment threshold group is called, the edge control terminal judges the segment synchronization characteristic data of the current collection period in the order of synchronization imbalance level, congestion level, fluctuation level and normal level. When the judgment condition is met for the first time, the subsequent level judgment stops, thereby determining a unique segment synchronization risk level for the current collection period.

[0076] The edge control unit first compares the audio / video arrival offset with the segment synchronization offset threshold corresponding to the current operating segment. When the audio / video arrival offset exceeds the segment synchronization offset threshold, the current acquisition cycle is determined to be at a synchronization imbalance level, and the offset direction is written into the risk judgment record to distinguish whether video frame data lags behind audio frame data or vice versa.

[0077] When the audio / video arrival offset does not exceed the segment synchronization offset threshold, the edge control terminal continues to assess available bandwidth, video queue status data, and target voice identifier. When available bandwidth is lower than the segment bandwidth threshold, the video queue status data indicating queue length or queue waiting time exceeds the corresponding threshold in the segment video queue threshold group, and a target voice identifier exists in the current acquisition period, a congestion level is determined. The target voice identifier, as a necessary condition for a congestion level, ensures that the same bandwidth reduction and video queue backlog receive higher processing priority when intercom, dispatch voice, or distress voice is present.

[0078] When the criteria for determining synchronization imbalance level and congestion level are not met, the edge control terminal compares the changes in link latency, link jitter, packet loss rate, and available bandwidth with the corresponding thresholds in the segment link fluctuation threshold group. If at least one link fluctuation data point exceeds the corresponding threshold, it is determined to be a fluctuation level, and the type of link quality parameter exceeding the threshold is recorded in the risk assessment record. When the segment synchronization characteristic data does not meet the criteria for determining synchronization imbalance level, congestion level, and fluctuation level, it is determined to be a normal level.

[0079] Video keyframe arrival status data is used to verify the risk assessment results. When the video queue status data indicates queue backlog, and the video keyframe arrival status data is in a delayed arrival or non-arrival state, the video keyframe anomaly flag is written into the risk assessment record corresponding to the current segment's synchronization risk level to determine the video keyframe interval and transmission priority. Video keyframe arrival status data does not generate new risk levels independently to avoid multiple risk levels forming simultaneously in the same acquisition cycle.

[0080] The segment synchronization risk level includes at least the risk level type, corresponding operating segment, judgment time, triggering condition, offset direction, link quality parameter type exceeding the threshold, and video keyframe anomaly identifier. The edge control terminal associates the segment synchronization risk level with the corresponding acquisition cycle synchronization record and sends it to the policy generation module.

[0081] S5. Generate synchronization transmission strategy data, including synchronization buffer parameters, transmission priority, video parameters, and queue parameters, based on the segment synchronization risk level. In this embodiment, step S5 includes the following process: When the segment synchronization risk level is normal, the preset basic synchronization buffer duration, basic video bitrate, basic video frame rate, and basic keyframe interval are written into the synchronization transmission strategy data. When the segment synchronization risk level is fluctuating, the synchronization buffer duration is increased within the preset maximum buffer duration range, the video keyframe interval is shortened, and the video bitrate of ordinary video frame data is reduced. When the segment synchronization risk level is congested, the transmission priority of audio frame data with target voice identifiers, video keyframe data, and device status data indicating abnormalities is increased, the upper limit of the queue length corresponding to ordinary video frame data is reduced, and the number of ordinary video frames is reduced. According to at least one of the video frame rate and video resolution; when the segment synchronization risk level is the synchronization imbalance level and the offset direction indicates that the video frame data lags behind the audio frame data, reduce at least one of the upper limit of the queue length and the upper limit of the queue waiting time of the video transmission queue, reduce the video bitrate of the ordinary video frame data, and increase the transmission priority of the video key frame data; when the segment synchronization risk level is the synchronization imbalance level and the offset direction indicates that the audio frame data lags behind the video frame data, reduce at least one of the upper limit of the queue length and the upper limit of the queue waiting time of the audio transmission queue, and increase the transmission priority of the audio frame data with target speech identifier.

[0082] Specifically, the synchronous transmission strategy data refers to the set of parameters generated by the cage-side edge control terminal and used to control the transmission process of audio frame data, video frame data, device status data, and voice-accompanied evidence data. Under normal transmission conditions, the synchronous transmission strategy data includes synchronization buffer parameters, transmission priority, video parameters, and queue parameters. When the main power supply or main link malfunctions and generates event buffer data, emergency transmission parameters are written based on the event buffer data. When the main link recovers, or the main power supply returns to normal and the main link becomes available, retransmission parameters are written based on the selected event buffer data to be retransmitted.

[0083] Synchronization buffer parameters include synchronization buffer duration and preset maximum buffer duration; video parameters include video bitrate, video frame rate, video resolution, and video keyframe interval; queue parameters include the upper limit of the audio transmission queue length and the upper limit of the queue waiting time, as well as the upper limit of the video transmission queue length and the upper limit of the queue waiting time; emergency transmission parameters include emergency video bitrate, emergency video frame rate, and emergency data category; retransmission parameters include retransmission bandwidth ratio, single retransmission data volume, and real-time audio reserved bandwidth.

[0084] The preset basic synchronization buffer duration is obtained through normal link test data. Under the condition that the main link is stable and there is no data loss, the ground monitoring platform calculates the total time required for audio and video frame data to go from acquisition, encoding, queuing, transmission to entering the playback buffer, reads the upper limit of the difference between audio processing time and video processing time, and adds a collection cycle length as the basic buffer margin to obtain the preset basic synchronization buffer duration.

[0085] The preset maximum buffer duration is set based on the maximum playback waiting time allowed for low-latency intercom, and its value is greater than the preset basic synchronization buffer duration. The adjustment of the synchronization buffer duration must not exceed the preset maximum buffer duration to avoid continuously increasing the buffer to absorb link jitter, which would cause unacceptable overall delays in dispatch voice and on-site video.

[0086] The base video bitrate, base video frame rate, base video resolution, and base keyframe interval are read through the video encoder configuration interface. During the equipment commissioning phase, under the condition that the cage door area, personnel standing area, and cage interior environment can all be identified, the video encoder parameters are adjusted step by step, and the parameter set that can meet the preset image recognition requirements and does not exceed the normal available link bandwidth is selected as the base video parameters.

[0087] The upper limits of the basic queue length and queue waiting time for the audio and video transmission queues are set based on the basic audio frame rate, basic video frame rate, preset basic synchronization buffer time, and allowed data frame waiting time, and written into the transmission configuration table. To determine the specific adjustment amounts for each parameter, the edge control terminal calculates the risk exceedance coefficient for the current acquisition cycle. The risk exceedance coefficient characterizes the maximum degree of exceedance of the current segment's synchronization characteristic data relative to the corresponding segment's judgment threshold group, using the following formula: ; In the formula, This represents the risk exceedance coefficient for the p-th data acquisition cycle; p represents the sequential number of the data acquisition cycle. This represents the audio / video arrival offset for the p-th acquisition cycle; This represents the segment synchronization offset threshold corresponding to the current running segment r; r represents the running segment number. Represents the set of link quality parameter types; g represents the link quality parameter type number; This represents the dimensionless change in the quality parameter of the g-th link during the p-th acquisition period; This represents the link fluctuation threshold of the g-th segment corresponding to the current operating segment r; This represents the length of the video transmission queue for the p-th acquisition cycle; This represents the video queue length threshold corresponding to the current running segment r; The video transmission queue waiting time represents the p-th acquisition cycle. This represents the video queue waiting time threshold corresponding to the current running segment r; This represents the segment bandwidth threshold corresponding to the currently running segment r; This represents the available bandwidth for the p-th acquisition cycle.

[0088] Among them, the larger the audio and video arrival offset, the better. The larger the value, the greater the dimensionless change in any link quality parameter. The larger the value, the longer the video transmission queue length or the longer the video transmission queue waiting time. The larger the value, the smaller the available bandwidth; the larger the ratio between the segment bandwidth threshold and the available bandwidth. The larger the value, the better. When the available bandwidth is zero, the corresponding division operation is not performed; instead, the current acquisition cycle is directly matched to the preset highest adjustment level.

[0089] The strategy adjustment mapping table is pre-stored at the edge control terminal. The strategy adjustment mapping table sets multiple adjustment levels according to the segment synchronization risk level and risk over-limit coefficient range. Each adjustment level corresponds to the increase in buffer time, the decrease in video keyframe interval, the reduction in video bitrate, the reduction in video frame rate, the reduction in video resolution level, the reduction in the upper limit of audio queue length, the reduction in the upper limit of video queue length, and the reduction in the upper limit of queue waiting time.

[0090] The strategy adjustment mapping table is obtained through link playback tests during the system debugging phase. During debugging, different levels of latency, jitter, packet loss, and bandwidth limitations are injected into the communication link, and the synchronization buffer, video, and queue parameters are changed step by step. Under the conditions that the target voice data can be transmitted continuously, the video keyframe data can arrive within the preset arrival time limit, and the audio and video arrival offset recovers to within the segment synchronization offset threshold, the set of parameter adjustment values ​​with the smallest change is selected and written into the corresponding adjustment level. In this way, each parameter adjustment value can be directly matched with the strategy adjustment mapping table through the risk over-limit coefficient, rather than being arbitrarily set during operation.

[0091] When the segment synchronization risk level is normal, the edge control terminal writes the preset basic synchronization buffer duration, basic video bitrate, basic video frame rate, basic video resolution, basic keyframe interval, and basic queue parameters into the synchronization transmission strategy data. Emergency transmission parameters and retransmission parameters retain their current configuration and do not trigger emergency transmission or buffer retransmission.

[0092] When the segment synchronization risk level is at the fluctuation level, the edge control terminal adjusts the mapping table according to the risk exceedance coefficient matching strategy, reading the buffer duration increase, video keyframe interval reduction, and video bitrate reduction ratio. The edge control terminal first increases the corresponding buffer duration based on the current synchronization buffer duration; if the adjusted synchronization buffer duration exceeds the preset maximum buffer duration, the preset maximum buffer duration is used. Subsequently, the video keyframe interval reduction is subtracted from the current video keyframe interval, and the video bitrate of ordinary video frame data is reduced according to the video bitrate reduction ratio.

[0093] Increasing the synchronization buffer duration at different fluctuation levels is to prioritize absorbing short-duration link fluctuations and avoid immediately degrading image quality when minor fluctuations occur. Shortening the video keyframe interval after buffer adjustment facilitates faster recovery of independent decoding after video data loss. Finally, reducing the video bitrate of ordinary video frame data reduces the transmission load when link fluctuations persist.

[0094] When the segment synchronization risk level is congestion level, the edge control terminal first sets audio frame data with target voice identifiers as the first data category transmission priority, sets video keyframe data and device status data indicating anomalies as the second data category transmission priority, and sets ordinary video frame data as a data category transmission priority lower than the aforementioned data. The communication scheduler reads the transmission queue from high to low according to the data category transmission priority.

[0095] After completing the transmission priority adjustment, the edge control terminal reads the reduction amount of the upper limit of the video queue length based on the risk exceeding the limit coefficient, and deducts this reduction amount from the upper limit of the queue length corresponding to the current ordinary video frame data. When the adjusted upper limit of the queue length is lower than the preset minimum video queue length, the preset minimum video queue length is adopted. Subsequently, the video frame rate reduction ratio or video resolution reduction level in the mapping table is adjusted according to the strategy to reduce the video frame rate or video resolution of the ordinary video frame data.

[0096] Under congestion levels, the transmission priority of target voice data, video keyframe data, and abnormal device status data is increased first, and then the upper limit of the queue length of ordinary video frame data is reduced. This allows us to first determine the data categories that must be retained and then release the queuing space occupied by ordinary video frame data. On this basis, reducing the ordinary video frame rate or video resolution can reduce the amount of data entering the video transmission queue later and prevent the queue from accumulating rapidly again.

[0097] When the segment synchronization risk level is at the synchronization imbalance level, and the offset direction indicates that video frame data lags behind audio frame data, the edge control terminal reads the reduction amount of the upper limit of video queue length, the reduction amount of the upper limit of video queue waiting time, and the reduction ratio of video bitrate from the policy adjustment mapping table according to the risk exceeding limit coefficient. The edge control terminal first reduces at least one of the upper limit of video transmission queue length and the upper limit of queue waiting time, causing ordinary video frame data with longer waiting times to be removed from the queue in advance; then it reduces the video bitrate of ordinary video frame data and increases the transmission priority of video key frame data. When the video key frame arrival status data indicates delayed arrival or non-arrival, the video key frame interval is shortened simultaneously.

[0098] When the segment synchronization risk level is at the synchronization imbalance level, and the offset direction indicates that audio frame data lags behind video frame data, the edge control terminal reads the reduction amount of the upper limit of audio queue length and the reduction amount of the upper limit of audio queue waiting time based on the risk exceeding limit coefficient. It then reduces at least one of the upper limit of the audio transmission queue length and the upper limit of the queue waiting time, and increases the transmission priority of audio frame data with target voice identifiers. The parameters of ordinary video frame data remain at their current values ​​to avoid irrelevant video quality adjustments when audio path lag occurs.

[0099] After the synchronization transmission strategy data is generated, the edge control terminal writes the strategy version number, generation time, applicable operating segment, segment synchronization risk level, risk exceedance coefficient, and effective collection cycle to it. The new synchronization transmission strategy data takes effect from the next collection cycle. Data frames that have already entered the transmission process in the previous collection cycle continue to be transmitted according to the original synchronization transmission strategy data, avoiding parameter switching of the same data frame during transmission.

[0100] S6. When the audio frame data contains a target voice identifier, the corresponding audio frame data, video key frame data and device status data are bound together as voice-accompanying evidence data and transmitted according to the synchronous transmission strategy. In this embodiment, step S6 includes the following process: Audio frame data with target voice identifiers and continuous time are merged to determine the start time, end time, and target voice type of the voice event, generating voice event data. Based on the start time, end time, and unified time identifier of the voice event, video keyframe data within a preset evidence time window is extracted. Cage position, door opening / closing status, equipment status data, operating section data, and link quality data are extracted from the synchronous records of the acquisition cycle corresponding to the video keyframe data. The voice event data, video keyframe data, cage position, door opening / closing status, equipment status data, operating section data, and link quality data are written into the same event index to generate voice-accompanied evidence data. Within the data category transmission priority determined by the synchronous transmission strategy data, the evidence transmission order between different voice-accompanied evidence data is determined according to the target voice type. Specifically, the evidence transmission order corresponding to distress calls precedes dispatch calls, and the evidence transmission order corresponding to dispatch calls precedes ordinary voices. The voice-accompanied evidence data is written into the transmission queue corresponding to the synchronous transmission strategy data and transmitted according to the evidence transmission order.

[0101] Specifically, voice-accompanied evidence data refers to a data set formed by time-binding corresponding audio frame data, video keyframe data, cage operation status data, equipment status data, operating section data, and link quality data, using the target voice event as an index. Voice-accompanied evidence data is used to enable ground monitoring platforms to simultaneously obtain corresponding on-site images and equipment operation status when receiving distress calls, dispatch voices, or ordinary intercom voices.

[0102] The edge control terminal sequentially reads the target speech identifier and target speech type from the audio frame data. When consecutive audio frame data all have target speech identifiers, the acquisition time interval between adjacent audio frames does not exceed the preset speech merging interval, and the target speech type is the same, the corresponding audio frame data are merged into the same speech event.

[0103] The preset speech merging interval is used to allow for short pauses within the same continuous speech segment. The preset speech merging interval is determined based on the speech hold time and audio frame duration used for speech activity detection, taking the larger value between the speech hold time and the audio frame duration, and writing it into the speech event configuration table. When the time interval between adjacent target speech frames exceeds the preset speech merging interval, or when the target speech type changes, the current speech event ends, and a new speech event is generated starting from the next target speech frame.

[0104] The edge control terminal determines the audio acquisition time of the first audio frame data in the same voice event as the start time of the voice event, and the audio acquisition end time of the last audio frame data as the end time of the voice event. It also associates the corresponding target voice type, audio frame number list, audio acquisition time range and unified time identifier to generate voice event data.

[0105] The preset evidence time window includes a pre-confirmation evidence period before the start time of the voice event and a post-confirmation evidence period after the end time of the voice event. The pre-confirmation and post-confirmation evidence periods are stored in the voice event configuration table and are set separately for distress calls, dispatch calls, and normal voices. The pre-confirmation evidence period is used to retain the cage view and equipment status before the target voice occurs, while the post-confirmation evidence period is used to retain changes in the scene after the target voice ends.

[0106] The edge control terminal queries the video keyframe data whose acquisition time falls within a preset evidence time window from the synchronous record of the acquisition cycle, based on the start time, end time, and unified time identifier of the voice event. If multiple video keyframes exist within the preset evidence time window, all video keyframes are extracted in the order of video acquisition time.

[0107] When no video keyframe data exists within the preset evidence time window, the edge control terminal reads the video keyframe data closest to the start time of the voice event before the preset evidence time window. Simultaneously, it sends a keyframe generation command through the video encoder control interface, causing the video encoder to encode the next video frame after receiving the command as video keyframe data, and then inserts this video keyframe data into the accompanying voice evidence data. Thus, when the target voice occurs, at least one frame of footage before the target voice occurs and one frame of footage after the target voice occurs can be preserved.

[0108] For each video keyframe, the edge control terminal queries the corresponding acquisition cycle synchronization record based on its video acquisition time and unified time identifier, and extracts the cage position, door opening and closing status, equipment status data and link quality data from it; at the same time, it queries the corresponding running section data based on the video acquisition time, and extracts the section type, running direction and personnel boarding and alighting identifiers.

[0109] When the preset evidence time window covers multiple acquisition cycles, device status data and link quality data are saved in the order of the acquisition cycles, without overwriting data from the previous acquisition cycle with data from the later acquisition cycle. Therefore, the ground monitoring platform can reconstruct the changes in main power status, communication status, and link quality before and after the occurrence of the target voice based on the time sequence.

[0110] An event index is a data field used to uniquely identify a set of voice-accompanied evidence data. The edge control terminal sequentially concatenates the cage device number, the voice event start time, and the local event sequence number to generate the event index. The cage device number is read from the edge control terminal's device configuration file; the local event sequence number is incremented sequentially by the edge control terminal each time a new voice event is generated. By combining the cage device number, the voice event start time, and the local event sequence number, voice-accompanied evidence data generated in different cages or at different times from the same cage can have different event indices.

[0111] The edge control terminal writes the event index, voice event data, video keyframe data, cage location, door opening / closing status, equipment status data, operating section data, and link quality data into the same voice-accompanied evidence data record. The voice-accompanied evidence data record also includes the data generation time, target voice type, data integrity status, and the corresponding synchronization transmission strategy version number.

[0112] The data integrity status is determined based on whether the voice event data, video keyframe data, and device status data are complete. When all three types of data have been generated, a complete status is written; when video keyframe data or device status data contains missing data indicators, a partially missing status is written, and the missing data type is recorded. A partially missing status does not prevent voice-related evidence data from entering the transmission queue, avoiding delays in the transmission of distress calls or dispatch voice messages due to missing auxiliary data.

[0113] The voice-accompanied evidence data is transmitted in a two-level sequence. The first level is the transmission priority of data categories determined by the synchronous transmission strategy data, which is used to determine the transmission order among target voice data, video keyframe data, device status data, and ordinary video frame data. The second level is the evidence transmission order among different voice-accompanied evidence data within the same data category.

[0114] The edge control terminal determines the transmission order of the second-level evidence based on the target voice type. Voice-accompanied evidence data corresponding to distress calls is transmitted before voice-accompanied evidence data corresponding to dispatch calls, and voice-accompanied evidence data corresponding to dispatch calls is transmitted before voice-accompanied evidence data corresponding to ordinary voices. When multiple voice-accompanied evidence data exist for the same target voice type, they are transmitted in ascending order of the voice event start time.

[0115] After determining the order of evidence transmission, the edge control terminal serializes the voice-accompanying evidence data into evidence data packets. Each evidence data packet contains at least an event index, data type, data packet sequence number, total number of data packets, and data content. The data packet sequence number is used by the ground monitoring platform to reassemble the voice-accompanying evidence data in the original order, and the total number of data packets is used to determine whether the data for the corresponding event has been received completely.

[0116] The evidence transmission module writes evidence data packets into the corresponding transmission queues according to the data category transmission priority and evidence transmission order in the synchronous transmission strategy data. Audio frame data with target voice identifiers enters the target voice transmission queue, video keyframe data enters the video keyframe transmission queue, and equipment status data and operating segment data enter the status data transmission queue. The communication scheduler reads evidence data packets from each transmission queue according to the priority allocated in the synchronous transmission strategy data and the currently available bandwidth, and sends them to the ground monitoring platform through the main link.

[0117] After receiving the evidence data packets, the ground monitoring platform establishes a corresponding event reception record based on the event index and reassembles the voice-accompanied evidence data according to the data packet sequence number. Once all evidence data packets have been received, the ground monitoring platform returns transmission confirmation data containing the event index and a reception completion indicator to the edge control terminal on the cage side. Upon receiving the transmission confirmation data, the edge control terminal marks the corresponding voice-accompanied evidence data as transmitted successfully and removes it from the real-time transmission queue.

[0118] If no transmission confirmation data is received within the preset confirmation period, the edge control terminal retains the corresponding evidence data packet and retransmits it according to the current transmission priority in the synchronous transmission strategy data. When the link quality data indicates that the main link is abnormal, real-time retransmission is stopped, and the voice-accompanying evidence data that has not yet obtained the reception completion mark is retained in the local cache for subsequent event caching and recovery retransmission steps.

[0119] S7. The ground monitoring platform synchronously plays audio frame data and video frame data according to a unified time identifier, generates synchronous feedback calibration data based on the actual playback time, and updates the synchronous buffer parameters, transmission priority, video parameters and queue parameters of subsequent acquisition cycles based on the synchronous feedback calibration data. In this embodiment, step S7 includes the following process: The ground monitoring platform synchronously plays audio and video frame data based on a unified time identifier. It generates synchronization feedback calibration data, including playback offset direction and amount, based on the difference between the actual video playback time and the actual audio playback time. When the playback offset direction indicates that video frame data lags behind audio frame data, the upper limit of the video transmission queue length in subsequent acquisition cycles is reduced, the transmission priority of key video frame data is increased, and the video bitrate of ordinary video frame data is reduced. Conversely, when the playback offset direction indicates that audio frame data lags behind video frame data, the upper limit of the audio transmission queue length in subsequent acquisition cycles is reduced, and the transmission priority of audio frame data with target voice identifiers is increased.

[0120] Specifically, the ground monitoring platform receives audio frame data, video frame data, equipment status data, and accompanying voice evidence data sent by the edge control terminal on the cage side, and writes them into the audio receiving buffer, video receiving buffer, and status data buffer respectively according to a unified time identifier. Based on the acquisition cycle number, audio frame number, and video frame number in the synchronous record of the acquisition cycle, the ground monitoring platform establishes a playback correlation between the audio frame data, video frame data, and equipment status data.

[0121] Synchronous playback refers to a ground monitoring platform not directly playing audio and video frames according to their network arrival order, but rather sending them into their respective decoding and playback processes based on the acquisition time relationship represented by a unified timestamp. The ground monitoring platform first reads the synchronization buffer parameters from the current synchronous transmission strategy data and determines the audio and video playback buffer durations respectively. When no compensation buffer parameters are written, both audio and video playback buffer durations use the preset basic synchronization buffer duration; when compensation buffer parameters are written, only the data streams with relatively advanced playback times are given additional compensation buffer durations, while data streams with relatively lagging playback times are not further buffered.

[0122] After being decoded by the audio decoder, the audio frame data is written to the audio output buffer. The ground monitoring platform obtains the time when the audio frame data begins to leave the audio output buffer and enter the speaker playback link through the playback callback interface of the audio output device, and determines this time as the actual audio playback time. The actual audio playback time is recorded using the unified clock of the ground monitoring platform and is associated with and stored in conjunction with the audio frame number and unified time identifier.

[0123] After video frame data is decoded by the video decoder, it is written to the video rendering queue. The ground monitoring platform obtains the time when the video frame data is submitted to the display device and the screen refresh is completed through the screen rendering callback interface of the video player, and determines this time as the actual video playback time. The actual video playback time is also recorded using the unified clock of the ground monitoring platform and is associated with and stored with the video frame number and unified time identifier.

[0124] The actual playback time is recorded using a playback callback interface, rather than directly using the time when audio or video frame data enters the receive buffer as the actual playback time. This is because audio decoding, video decoding, playback buffering, and image rendering all introduce additional processing time. The actual playback time reflects the true relationship between the time the scheduler ultimately hears the sound and the time they see the image.

[0125] The ground monitoring platform, based on corresponding audio and video frame groups, reads the actual audio playback time of audio frame data and the actual video playback time of video frame data within the same corresponding audio and video frame group, and determines the directional time difference between the two as the playback offset value. The specific formula is as follows: ; In the formula, This represents the playback offset value of the u-th audio / video frame group; The actual playback time of the video frame data in the u-th audio / video frame group; This represents the actual playback time of the audio frame data in the u-th audio / video frame group; u represents the sequential number of the audio / video frame group.

[0126] When the actual video playback time is later than the actual audio playback time, the playback offset value is positive, and the direction of the playback offset indicates that the video frame data lags behind the audio frame data; when the actual video playback time is earlier than the actual audio playback time, the playback offset value is negative, and the direction of the playback offset indicates that the audio frame data lags behind the video frame data; when both are equal, the playback offset value is zero. The ground monitoring platform takes the absolute value of the playback offset value to obtain the playback offset amount, specifically using the following formula: ; In the formula, This represents the playback offset of the u-th audio / video frame group; This represents the playback offset value of the u-th audio / video frame group. The larger the absolute value of the playback offset, the greater the playback offset, indicating a more significant time misalignment between the audio content actually heard by the dispatcher and the video image seen.

[0127] The ground monitoring platform associates the corresponding frame group number of audio and video, playback offset direction, playback offset amount, corresponding running segment data, current synchronization transmission strategy version number, audio playback buffer duration, video playback buffer duration, audio frame loss status, and video frame loss status to generate synchronization feedback calibration data.

[0128] Audio frame loss status is determined by the continuity of audio frame numbers and the decoding status returned by the audio decoder; video frame loss status is determined by the continuity of video frame numbers, the arrival status data of video keyframes, and the decoding status returned by the video decoder. When an audio or video frame number is missing, the ground monitoring platform writes the missing data frame number into the synchronization feedback calibration data, enabling the cage-side edge control terminal to distinguish whether the playback offset is caused by queue backlog or by data frame loss and waiting for keyframe decoding again.

[0129] The ground monitoring platform aggregates and synchronously feeds back calibration data according to a preset feedback cycle. The preset feedback cycle is an integer multiple of the acquisition cycle length and is written into the synchronization configuration table. When there are multiple audio and video corresponding frame groups within a preset feedback cycle, the ground monitoring platform retains the playback offset direction and playback offset amount of each audio and video corresponding frame group, and selects the audio and video corresponding frame group with the largest playback offset amount as the target calibration object for that feedback cycle, avoiding the masking of obvious audio and video misalignment that occurs within a short period of time through averaging.

[0130] The synchronization calibration adjustment table is stored at the edge control terminal on the cage side. The synchronization calibration adjustment table sets multiple calibration levels according to the operating segment, playback offset direction, and playback offset range. Each calibration level corresponds to the adjustment amount of synchronization buffer, the adjustment amount of the upper limit of queue length, the adjustment amount of the upper limit of queue waiting time, the adjustment ratio of video bitrate, and the adjustment level of transmission priority.

[0131] The synchronization calibration adjustment table is obtained through audio and video playback debugging. During debugging, artificial delays of different durations are injected into the audio and video transmission paths to create playback offsets of different directions and sizes on the ground monitoring platform. Subsequently, the synchronization buffer, transmission queue, video bitrate, and data transmission priority are adjusted step by step. Within a preset number of acquisition cycles, if the playback offset recovers to within the synchronization offset threshold corresponding to the current operating segment without causing interruption of the target voice, the adjustment group with the smallest parameter change is selected and written to the corresponding calibration level.

[0132] The ground monitoring platform will synchronously send the calibration data to the cage-side edge control terminal. The cage-side edge control terminal will then query the synchronous calibration adjustment table based on the operating section, playback offset direction, and playback offset amount, and read the corresponding parameter adjustment values.

[0133] When the playback offset direction indicates that the video frame data lags behind the audio frame data, the cage-side edge control terminal generates an audio compensation buffer adjustment command based on the synchronization buffer adjustment amount and sends the audio compensation buffer adjustment command to the ground monitoring platform. The ground monitoring platform increases the compensation buffer duration of the audio playback path according to the audio compensation buffer adjustment command, so that the relatively advanced audio frame data is delayed in playback, and the increase in the compensation buffer duration does not exceed the preset maximum compensation buffer duration. The cage-side edge control terminal then reduces at least one of the upper limit of the video transmission queue length and the upper limit of the queue waiting time in subsequent acquisition cycles, increases the transmission priority of video key frame data, and reduces the video bitrate of ordinary video frame data according to the video bitrate adjustment ratio.

[0134] The adjustment value for the upper limit of the video transmission queue length is equal to the upper limit of the video transmission queue length before adjustment minus the adjustment amount of the upper limit of the queue length recorded in the corresponding level of the synchronization calibration adjustment table; if the adjustment result is lower than the preset minimum video queue length, the preset minimum video queue length shall be adopted. The adjustment value of the video bitrate is determined by multiplying the current ordinary video bitrate by the video bitrate adjustment ratio in the synchronization calibration adjustment table, and the adjusted ordinary video bitrate shall not be lower than the preset minimum video bitrate required to ensure basic identification of the cage door area and personnel area.

[0135] When video frame data lags behind audio frame data, the playback offset that has already formed is temporarily aligned first through audio compensation buffering. Then, by reducing the video transmission queue and the normal video bitrate, the queuing and transmission time of subsequent video frame data is reduced. This can simultaneously handle the current playback misalignment and the transmission cause of the continuous video delay.

[0136] When the playback offset direction indicates that the audio frame data lags behind the video frame data, the cage-side edge control terminal generates a video compensation buffer adjustment command based on the synchronization buffer adjustment amount and sends the video compensation buffer adjustment command to the ground monitoring platform. The ground monitoring platform increases the compensation buffer duration of the video playback path according to the video compensation buffer adjustment command, causing the relatively advanced video frame data to be played later. The cage-side edge control terminal then reduces at least one of the upper limit of the audio transmission queue length and the upper limit of the queue waiting time in subsequent acquisition cycles, and increases the transmission priority of audio frame data with target voice identifiers according to the transmission priority adjustment level.

[0137] The adjustment value for the upper limit of the audio transmission queue length is equal to the upper limit of the audio transmission queue length before adjustment minus the corresponding upper limit adjustment amount in the synchronization calibration adjustment table; the adjustment result must not be lower than the preset minimum audio queue length that can accommodate a complete target speech coding frame group. The number of levels by which the audio transmission priority is increased each time is directly read from the synchronization calibration adjustment table, and must not exceed the preset highest transmission priority of the target speech data.

[0138] When the playback offset does not exceed the recovery threshold corresponding to the current operating segment for a preset number of consecutive feedback cycles, the edge control terminal on the cage side gradually reduces the compensation buffer duration of the relatively leading data stream and gradually restores the normal video bitrate and transmission queue parameters. The recovery threshold is less than the segment synchronization offset threshold used to trigger calibration adjustment, which is used to prevent repeated increases and decreases in the compensation buffer when the playback offset fluctuates near the threshold. The amount of parameters restored each time is executed according to the recovery adjustment amount of the corresponding calibration level in the synchronization calibration adjustment table, until the synchronization transmission strategy data corresponding to the current segment synchronization risk level is restored.

[0139] S8. When the main power supply is abnormal, generate event cache data after the backup power supply takes over the power supply, or generate event cache data when the main link is abnormal. When the main link is restored, or the main power supply status is restored to normal and the main link is available, select the event cache data to be retransmitted according to the type of abnormality, determine the retransmission order, real-time audio reserved bandwidth and retransmission parameters, and retransmit the data in the local cache. In this embodiment, step S8 includes the following process: When device status data indicates a main power supply failure, the backup power supply takes over. After the backup power supply takes over, audio frame data, video keyframe data, device status data, and generated accompanying voice evidence data are written to the local cache. Based on emergency transmission parameters, the video bitrate and frame rate of ordinary video frame data are reduced to preset emergency values. When link quality data indicates a main link failure, audio frame data, video keyframe data, device status data, and generated accompanying voice evidence data are written to the local cache. Individual main power supply failures, individual main link failures, and main power supply and main link failures occurring during the same event are respectively identified as main power supply failures, main link failures, and combined main power supply and main link failures. Based on the failure type and operating segment data, and when a target voice identifier exists, the event level is determined by matching the target voice type with preset event classification conditions. The event level, occurrence time, and pending confirmation identifier are associated with the data written to the local cache to generate event cache data. When the link quality data indicates that the main link has recovered, data with an exception type of main link exception and an pending upload confirmation flag are selected from the event cache data. When the device status data indicates that the main power status has returned to normal and the link quality data indicates that the main link is available, data with an exception type of main power exception or main power and main link joint exception and an pending upload confirmation flag are selected from the event cache data. The selected event cache data are then used to generate retransmission sequence data in descending order of event level and in ascending order of occurrence time. The target voice transmission bandwidth, recent audio retransmission bandwidth, and control data reserved bandwidth of the current output of the audio encoder are added together to obtain the real-time audio reserved bandwidth. The larger of the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero is taken to obtain the upper limit of the retransmission bandwidth. The retransmission bandwidth ratio and the amount of data transmitted in a single retransmission are determined based on the upper limit of the retransmission bandwidth. Data in the local cache is retransmitted according to the retransmission sequence data, the retransmission bandwidth ratio, and the amount of data transmitted in a single retransmission. After receiving the retransmission confirmation data, the corresponding pending upload confirmation flag is updated to the uploaded confirmation flag.

[0140] Specifically, event cache data refers to the data records formed by writing audio frame data, video key frame data, device status data, and generated voice-accompanied evidence data that need to be retained into the local cache during main power supply or main link anomalies, and associating the anomaly type, event level, running segment data, occurrence time, and upload confirmation status with the cache data.

[0141] The cage-side edge control terminal determines the main power supply status and main link status based on equipment status data and link quality data, respectively.

[0142] Main power supply anomalies are detected through the power management unit. When the supply voltage of the main power input channel exceeds the preset normal voltage range, the main power supply channel returns to the disconnected state, or the power management unit returns a main / standby power switching command, the device status data is written to the main power anomaly flag. After the main power is restored, when the supply voltage of the main power input channel remains within the preset normal voltage range for a preset number of consecutive sampling cycles, and the power supply channel returns to the normal state, the device status data is written to the main power recovery flag.

[0143] The main link anomaly is jointly determined by the communication unit and the ground monitoring platform. When the communication interface returns to a disconnected state, a preset number of consecutive link detection data packets fail to receive reception acknowledgments, or the available bandwidth is lower than the minimum bandwidth required to maintain target voice data transmission for a preset number of consecutive acquisition cycles, link quality data is written to the main link anomaly flag. When the communication interface reconnects, link detection data packets receive consecutive reception acknowledgments, and the available bandwidth is higher than the preset link recovery bandwidth threshold for a preset number of consecutive acquisition cycles, link quality data is written to the main link recovery flag. Confirming the main link anomaly or recovery through multiple consecutive acquisition cycles reduces frequent buffer switching caused by short-term signal fluctuations within the wellbore.

[0144] In this embodiment, the main link is considered available when the communication interface is in a connected state, the link detection data packets can receive reception acknowledgments, and the available bandwidth is not less than the minimum bandwidth required to maintain the target voice data transmission. The main link is determined to be available when it does not experience any abnormalities but continues to meet the above conditions, or when a main link recovery flag has been written after an abnormality.

[0145] When device status data indicates a main power supply failure, the power management unit controls the backup power supply to take over power supply. When the backup power supply status indicates that the power supply channel switch is complete and the backup power supply output is within the normal output range, the edge control terminal records the unified time stamp of the first occurrence of the main power supply failure, determines this as the event occurrence time, and initiates the local cache writing process. When link quality data indicates a main link failure and the edge control terminal is in a normal power supply state, the edge control terminal records the unified time stamp of the first occurrence of the main link failure, determines this as the event occurrence time, and initiates the local cache writing process.

[0146] Local caching uses an independent data directory built according to event indexes. The edge control unit writes audio frame data with target voice identifiers, audio frame data without target voice identifiers but within the current acquisition cycle, video keyframe data, device status data indicating the anomaly, and voice-accompanying evidence data that has been generated but has not yet received a transmission completion indicator, generated during anomalies, into the corresponding event directory. Ordinary video frame data is not forcibly cached during main link anomalies to reduce the storage space occupied by continuous video; when device status data storage capacity is sufficient, a portion of ordinary video frame data is additionally saved according to the cache range set by the emergency transmission parameters.

[0147] The edge control unit associates each piece of data written to the local cache with its data type, data number, collection time, data length, and verification result. The verification result is generated using the existing Cyclic Redundancy Check (CRC) method: a checksum is calculated based on the data content before writing; after writing, the cached data is read and the checksum is calculated again; the two checksums are compared. If the two checksums match, the written data is considered complete; otherwise, the corresponding data is rewritten. CRC is only used to determine whether errors occur during the writing and reading of cached data and does not participate in event level determination.

[0148] Anomaly types include main power supply anomaly, main link anomaly, and combined main power supply and main link anomaly. When a main power supply anomaly and a main link anomaly occur successively during the same event duration, the anomaly type will be updated to combined main power supply and main link anomaly, and the event catalog will not be repeatedly created for the same consecutive anomaly process.

[0149] Preset event classification conditions are stored in the event classification configuration table. The event classification configuration table uses exception type, running segment type, person entry / exit identifier, and target voice type as matching conditions, and divides the event level into first event level, second event level, and third event level. The first event level is higher than the second event level, and the second event level is higher than the third event level.

[0150] When the anomaly type is a combined anomaly of the main power supply and the main link, or when the operating section data indicates an abnormal parking section and there is a distress call voice, it is determined to be a first-level event; when the anomaly type is a main power supply anomaly, or when the operating section data indicates a parking section or deceleration section with passenger pick-up / drop-off markers and there is a distress call voice or dispatch voice, it is determined to be a second-level event; all other events that only involve a main link anomaly and do not involve a distress call voice are determined to be third-level events.

[0151] When the same event meets the criteria for multiple event levels simultaneously, the highest event level is used. If no target voice type exists at the start of the event, but a distress call or dispatch voice is detected during the event, the edge control terminal re-invokes the event level configuration table and updates the event level if the re-matched event level is higher than the original event level, without downgrading the already determined event level.

[0152] The edge control terminal associates the event index, exception type, event level, running segment data, event occurrence time, cached data number list, and pending upload confirmation flag to generate event cache data. The pending upload confirmation flag is written to a valid status when the event cache data is generated, indicating that the corresponding cached data has not yet been fully received by the ground monitoring platform.

[0153] After the backup power supply takes over, the edge control terminal reads the emergency transmission parameters in the synchronous transmission strategy data, stops the transmission of ordinary video frame data that does not belong to the emergency data category, or reduces the video bitrate and video frame rate of ordinary video frame data to the preset emergency value, while retaining the acquisition, buffering and transmission of target voice data, video key frame data, abnormal device status data and control data.

[0154] The preset emergency video bitrate and preset emergency video frame rate were obtained through backup power supply endurance testing. During the test, the power management unit recorded the remaining capacity of the backup power supply and the power consumption status of each device, and gradually reduced the video bitrate and video frame rate of ordinary video frame data. Under the condition that the backup power supply can maintain the preset emergency power supply duration and the video key frames can still identify the cage door area and personnel area, the highest video bitrate and video frame rate were selected as the preset emergency values ​​and written into the emergency transmission configuration table.

[0155] When the main power supply fails, the video bitrate adjustment is the difference between the current normal video bitrate and the preset emergency video bitrate; the video frame rate adjustment is the difference between the current normal video frame rate and the preset emergency video frame rate. If the current normal video bitrate or video frame rate is already lower than the corresponding preset emergency value, the current parameters are maintained, and no reverse increase is performed.

[0156] When the synchronous feedback calibration data indicates that the video frame data lags behind the audio frame data, the edge control terminal updates the upper limit of the video transmission queue length, the priority of video key frame data transmission, the normal video bitrate, and the audio compensation buffer duration for subsequent acquisition cycles according to the synchronous calibration adjustment table; when the synchronous feedback calibration data indicates that the audio frame data lags behind the video frame data, the upper limit of the audio transmission queue length, the priority of target voice data transmission, and the video compensation buffer duration are updated.

[0157] When synchronous feedback calibration and adjustment occur simultaneously with main power supply or main link failures, emergency transmission parameters have a higher execution priority than ordinary synchronous calibration parameters. When ordinary video bitrate, video frame rate, and queue parameters are simultaneously limited by both types of parameters, parameter values ​​that minimize the amount of ordinary video data transmitted are used. The transmission priority of target audio data, video keyframe data, and abnormal device status data adopts the higher of the two strategies, thereby avoiding the synchronous feedback calibration process from weakening the protection of critical data under abnormal conditions.

[0158] When link quality data indicates that the main link has recovered, the edge control terminal selects event cache data from its local cache that shows the main link as the exception type and has a valid pending upload confirmation flag. When device status data indicates that the main power supply status has returned to normal and link quality data indicates that the main link is available, the edge control terminal selects event cache data from its local cache that shows the main power supply as the exception type or the main power supply and main link as the combined exception type and has a valid pending upload confirmation flag. The edge control terminal sorts the selected event cache data in descending order of event level. Event cache data with the same event level are then sorted in chronological order of event occurrence time to generate retransmission order data. Data within the same event directory is retransmitted in the following order: target voice audio frame data, voice accompanying evidence data, video keyframe data, abnormal device status data, and additional ordinary video frame data.

[0159] To prevent buffered retransmissions from consuming real-time audio intercom bandwidth, the edge control terminal first determines the reserved real-time audio bandwidth in each retransmission acquisition cycle. The reserved real-time audio bandwidth is obtained by adding the current audio encoder output bandwidth, the recent audio retransmission bandwidth, and the reserved control data bandwidth, using the following formula: ; In the formula, Represents the real-time audio reserved bandwidth for the nth retransmission acquisition cycle; This represents the target voice transmission bandwidth currently output by the audio encoder; The recent audio retransmission bandwidth is obtained by dividing the amount of audio data retransmitted within the preset statistical time window before the nth retransmission acquisition cycle by the length of the statistical time window. The reserved bandwidth for control data represents the data required for device status, link detection, and transmission confirmation; n represents the sequential number of the retransmission acquisition cycle.

[0160] The larger the current output bandwidth of the audio encoder, the larger the reserved bandwidth for real-time audio; the larger the recent audio retransmission volume, the larger the recent audio retransmission bandwidth, and the larger the reserved bandwidth for real-time audio; the larger the reserved bandwidth for control data, the larger the reserved bandwidth for real-time audio. When the reserved bandwidth for real-time audio increases, the bandwidth available for buffering and retransmission decreases accordingly.

[0161] After obtaining the real-time audio reserved bandwidth, the edge control terminal determines the upper limit of the retransmission bandwidth based on the currently available bandwidth and the real-time audio reserved bandwidth, using the following formula: ; In the formula, Represents the upper limit of the retransmission bandwidth for the nth retransmission acquisition cycle; This represents the currently available bandwidth for the nth retransmission acquisition cycle; This represents the reserved bandwidth for real-time audio during the nth retransmission acquisition cycle.

[0162] The larger the available bandwidth, the larger the upper limit of the retransmission bandwidth when the reserved bandwidth for real-time audio remains unchanged; conversely, the larger the reserved bandwidth for real-time audio, the smaller the upper limit of the retransmission bandwidth when the available bandwidth remains unchanged. When the available bandwidth is not higher than the reserved bandwidth for real-time audio, the upper limit of the retransmission bandwidth is zero, and the edge control end pauses buffered retransmission, maintaining only the transmission of real-time audio, control data, and current video keyframe data.

[0163] The retransmission bandwidth percentage is obtained by dividing the maximum retransmission bandwidth by the currently available bandwidth. When the currently available bandwidth is zero, the retransmission bandwidth percentage is set to zero. The edge control terminal writes the retransmission bandwidth percentage into the synchronization transmission policy data, so that the communication scheduler allocates the buffered retransmission bandwidth according to this percentage in each retransmission acquisition cycle.

[0164] The amount of data transmitted in a single retransmission is determined based on the maximum retransmission bandwidth and the preset duration of a single retransmission, using the following formula: ; In the formula, This represents the amount of data transmitted in a single transmission during the nth transmission acquisition cycle. Represents the upper limit of the retransmission bandwidth for the nth retransmission acquisition cycle; This represents the preset duration of a single retransmission.

[0165] The higher the upper limit of the retransmission bandwidth, the larger the amount of data transmitted in a single retransmission; the longer the preset duration of a single retransmission, the larger the amount of data transmitted in a single retransmission. The preset duration of a single retransmission is written into the retransmission configuration table, and its value does not exceed one retransmission scheduling time window to avoid continuously occupying the communication link during a single retransmission process and failing to respond to new target voice data in a timely manner.

[0166] The edge control unit reads cached data from its local buffer, up to the amount of data to be retransmitted in a single batch, according to the retransmission order, and writes it into the retransmission queue. After completing the scheduling of the current real-time target voice data and control data, the communication scheduler reads the cached data from the retransmission queue and sends it to the ground monitoring platform.

[0167] The ground monitoring platform reassembles the cached data based on the event index, data number, and data packet sequence number. After receiving all the forced retransmission data within the same event directory, it returns retransmission confirmation data containing the event index and a reception completion identifier to the edge control terminal. Upon receiving the retransmission confirmation data, the edge control terminal updates the pending upload confirmation identifier of the corresponding event cached data to the uploaded confirmation identifier.

[0168] If the main link fails again during the retransmission process, the edge control terminal immediately stops the retransmission and retains the pending upload confirmation flag for data that has not yet received retransmission confirmation. After the main link is restored, the retransmission order data is regenerated according to the event level and occurrence time, and data that already has an upload confirmation flag will not be retransmitted again.

[0169] When a new target voice identifier is detected during the retransmission process, the communication scheduler immediately pauses reading the retransmission queue and prioritizes the transmission of newly generated target voice data and voice-accompanying evidence data; after the target voice event ends and the current available bandwidth is once again higher than the real-time audio reserved bandwidth, the buffered retransmission continues.

[0170] like Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of this application. The low-latency synchronous audio and video intercom system for cages provided in this embodiment includes the following modules at the cage side edge control terminal: The data association module is used to acquire cage operation status data, audio frame data, video frame data, equipment status data, and link quality data, and generate collection cycle synchronization records according to a unified time identifier; The section determination module is used to determine the operating section data based on the cage's operating status data; The feature generation module is used to generate segment synchronization feature data, including audio and video arrival offset, offset direction, link fluctuation and queue status, based on the acquisition cycle synchronous record and link quality data. The risk determination module is used to call the corresponding segment judgment threshold group based on the running segment data, and determine the segment synchronization risk level by combining the segment synchronization characteristic data. The strategy generation module is used to generate synchronization transmission strategy data, including synchronization buffer parameters, transmission priority, video parameters, and queue parameters, based on the segment synchronization risk level. The evidence transmission module is used to bind the corresponding audio frame data, video key frame data, and device status data into voice-accompanying evidence data when the audio frame data contains a target voice identifier, and transmit the data according to the synchronous transmission strategy. The local caching module is used to store audio frame data, video key frame data, device status data, and generated voice-accompanied evidence data when the main power supply or main link fails, and to generate event cache data. The strategy update module is used to update the synchronization buffer parameters, transmission priority, video parameters, and queue parameters for subsequent acquisition cycles based on the synchronization feedback calibration data; when the local cache module generates event cache data, it generates emergency transmission parameters based on the event cache data; when the main link is restored, or the main power status is restored to normal and the main link is available, it selects the event cache data to be retransmitted according to the anomaly type, and generates retransmission order data and retransmission parameters to update the synchronization transmission strategy data. The ground monitoring platform includes a synchronization feedback module, which is used to synchronously play audio frame data and video frame data according to a unified time identifier, generate synchronization feedback calibration data based on the actual playback time of the two, and send the synchronization feedback calibration data to the edge control terminal on the cage side.

[0171] Furthermore, the cage-side edge control terminal is connected to the ground monitoring platform via a main link. This main link is used to transmit audio frame data, video frame data, equipment status data, voice-related evidence data, and synchronous feedback calibration data. The system also includes a backup power supply, which is connected to the cage-side edge control terminal and is used to take over power supply in the event of a main power failure. The local caching module classifies isolated main power supply anomalies, isolated main link anomalies, and main power supply and main link anomalies occurring during the same event duration into main power supply anomalies, main link anomalies, and combined main power supply and main link anomalies, respectively. Based on the anomaly type and operating segment data, and when a target voice identifier is present, it matches the target voice type with preset event classification conditions to determine the event level and generate event cache data. When the main link recovers, the policy update module selects event cache data with the anomaly type of main link anomaly and an pending upload confirmation identifier. When the main power status returns to normal and the main link is available, the policy update module selects event cache data with the anomaly type of main power supply anomaly or combined main power supply and main link anomaly and an pending upload confirmation identifier, and generates retransmission order data and retransmission parameters based on the selected event cache data. The evidence transmission module reads and retransmits the corresponding data in the local caching module according to the retransmission order data and retransmission parameters.

[0172] The foregoing has shown and described the basic principles, main features, and advantages of this application. Those skilled in the art should understand that this application is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of this application. Various changes and modifications can be made to this application without departing from the spirit and scope thereof, and all such changes and modifications fall within the scope of this application as claimed. The scope of protection of this application is defined by the appended claims and their equivalents.

Claims

1. A method for low-latency synchronous audio and video intercom in a cage, characterized in that, include: Acquire cage operation status data, audio frame data, video frame data, equipment status data, and link quality data, and generate a collection cycle synchronization record according to a unified time identifier; Based on the cage operation status data, the operation section data is determined, and based on the acquisition cycle synchronization record and link quality data, section synchronization characteristic data including audio and video arrival offset, offset direction and audio and video transmission queue status are generated. The risk level of segment synchronization is determined based on the segment judgment threshold group and segment synchronization feature data corresponding to the running segment data. Synchronization transmission strategy data is generated based on the segment synchronization risk level and offset direction to adjust the upper limit of the queue length and the upper limit of the queue waiting time for the audio or video transmission queue. The ground monitoring platform plays audio and video frame data according to a unified time identifier and generates synchronous feedback calibration data based on the actual playback time of the two. Based on synchronous feedback calibration data, the compensation buffer duration of relatively advanced audio or video playback paths is increased, and the audio or video transmission queue parameters of relatively lagging audio or video in the synchronous transmission strategy data are updated. When the main link fails, audio frame data, video key frame data, and device status data are written to the local cache and event cache data is generated. When the main link recovers, the retransmission order data is generated based on the event level and occurrence time of the event cache data. The larger value between the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero is taken to obtain the upper limit of the retransmission bandwidth. The retransmission parameters are determined based on the upper limit of the retransmission bandwidth, and the data in the local cache is retransmitted according to the retransmission order data and the retransmission parameters.

2. The method for low-latency synchronous audio and video intercom in a cage according to claim 1, characterized in that, Generate a data acquisition cycle synchronization record, specifically including: Acquire cage operating status data including cage position, running speed, acceleration, running direction, and door opening / closing status; Acquire audio frame data including audio frame number, audio acquisition time, audio arrival time, voice activity identifier, distress identifier, and dispatch voice identifier; When the distress call identifier is in a valid state, the target voice identifier is written into the corresponding audio frame data, and the target voice type is determined to be distress call voice; When the distress call identifier is not in a valid state but the dispatch voice identifier is in a valid state, the target voice identifier is written to the corresponding audio frame data, and the target voice type is determined to be dispatch voice. When both the distress call identifier and the dispatch voice identifier are not in a valid state, but the voice activity identifier is in a valid state, write the target voice identifier into the corresponding audio frame data and determine the target voice type as ordinary voice. Acquire video frame data including video frame number, video acquisition time, video arrival time and key frame identifier, identify video frame data with key frame identifier as video key frame data, and identify the remaining video frame data as ordinary video frame data. Acquire device status data including main power status, backup power status, communication status, storage status, and device online status; Acquire link quality data including link latency, link jitter, packet loss rate, signal strength, and available bandwidth; The audio frame data, video frame data, cage operation status data, equipment status data, and link quality data within the same acquisition cycle are associated with a unified time identifier to generate a synchronous record for the acquisition cycle.

3. The low-latency synchronous audio and video intercom method for cages according to claim 2, characterized in that, Determine the data for the operating segment, specifically including: The cage position, running speed, acceleration, running direction, and door opening and closing status are correlated along the time axis to generate running status change data; When the cage is in the stopping position range and the running speed is lower than the preset static speed threshold, it is determined as a stopping section, and when the door is in the open state, a person loading / unloading identifier is written for the stopping section. When the cage moves away from the parking position and the running speed increases for a preset number of data collection cycles, it is determined to be the start-up acceleration section; When the running speed is within a preset speed fluctuation range for a preset number of consecutive collection cycles, it is determined as a uniform speed running section; When the cage enters the docking position range and its running speed decreases for a preset number of consecutive data collection cycles, it is determined to be the docking deceleration section. When the cage is located outside the parking position range, the running speed is lower than the preset stationary speed threshold, and the duration exceeds the preset parking time, it is determined to be an abnormal parking section. Associate the segment type, segment start time, segment end time, running direction, and passenger boarding / alighting identifiers to generate running segment data.

4. The low-latency synchronous audio and video intercom method for cages according to claim 3, characterized in that, Generate segment synchronization feature data, specifically including: For each video frame data within the same acquisition period, select the audio frame data whose video acquisition time is closest to the audio acquisition time, and generate a corresponding audio and video frame group. The audio and video arrival offset is determined based on the audio arrival time and video arrival time in the corresponding frame group, and the offset direction is determined based on the order of their arrival. Determine the changes in link latency, link jitter, packet loss rate, and available bandwidth between adjacent acquisition periods to generate link fluctuation data; Based on the queue length and queue waiting time of the audio and video transmission queues, generate audio queue status data and video queue status data; Determine whether video frame data with keyframe identifiers arrives within a preset arrival time limit, and generate video keyframe arrival status data; Based on the data of the operating segment, the arrival offset, offset direction, link fluctuation data, audio queue status data, video queue status data, target voice identifier, and video keyframe arrival status data are correlated to generate segment synchronization feature data.

5. The low-latency synchronous audio and video intercom method for cages according to claim 4, characterized in that, Determine the segment synchronization risk level, specifically including: The corresponding segment judgment threshold group is called based on the data of the running segment. The segment judgment threshold group includes the segment synchronization offset threshold, the segment link fluctuation threshold group, the segment bandwidth threshold, and the segment video queue threshold group. The segment synchronization offset thresholds for stopping sections with passenger boarding / alighting markings, deceleration sections at stops, and abnormal stopping sections are less than the segment synchronization offset thresholds for starting acceleration sections and constant speed running sections. The unique segment synchronization risk level is determined by prioritizing synchronization imbalance level, congestion level, fluctuation level, and normal level. When the audio and video offset exceeds the segment synchronization offset threshold, it is determined to be a synchronization imbalance level. When the conditions for determining the level of synchronization imbalance are not met, the available bandwidth is lower than the segment bandwidth threshold, the video queue status data indicates that the queue length or queue waiting time exceeds the corresponding threshold in the segment video queue threshold group, and a target voice identifier is present, it is determined to be a congestion level. When the conditions for determining the synchronization imbalance level and the congestion level are not met, and at least one change in the link fluctuation data exceeds the corresponding threshold in the segment link fluctuation threshold group, it is determined to be a fluctuation level. When the criteria for determining the synchronization imbalance level, congestion level, and fluctuation level are not met, it is determined to be at the normal level.

6. The method for low-latency synchronous audio and video intercom in a cage according to claim 5, characterized in that, Generate synchronization transmission strategy data, specifically including: When the segment synchronization risk level is normal, the preset basic synchronization buffer duration, basic video bitrate, basic video frame rate and basic key frame interval will be written into the synchronization transmission strategy data. When the segment synchronization risk level is fluctuating, the synchronization buffer time is increased within the preset maximum buffer time range, the video key frame interval is shortened, and the video bitrate of ordinary video frame data is reduced. When the segment synchronization risk level is congestion level, increase the transmission priority of audio frame data with target voice identifier, video key frame data and device status data representing abnormalities, reduce the upper limit of queue length corresponding to ordinary video frame data, and reduce at least one of the video frame rate and video resolution of ordinary video frame data. When the segment synchronization risk level is the synchronization imbalance level and the offset direction indicates that the video frame data lags behind the audio frame data, reduce at least one of the upper limit of the video transmission queue length and the upper limit of the queue waiting time, reduce the video bitrate of ordinary video frame data, and increase the transmission priority of video key frame data. When the segment synchronization risk level is the synchronization imbalance level and the offset direction indicates that the audio frame data lags behind the video frame data, reduce at least one of the upper limit of the audio transmission queue length and the upper limit of the queue waiting time, and increase the transmission priority of audio frame data with target voice identifier.

7. The method for low-latency synchronous audio and video intercom in a cage according to claim 6, characterized in that, It also includes generating and transmitting voice-accompanying evidence data, specifically including: The audio frame data with target speech identifiers and continuous time are merged to determine the start time, end time and target speech type of the speech event, and generate speech event data. Based on the start time, end time and unified time identifier of the voice event, extract video keyframe data within the preset evidence time window; Extract cage location, door opening / closing status, equipment status data, operating section data, and link quality data from the synchronous records of the acquisition cycle corresponding to the video keyframe data; Voice event data, video keyframe data, cage location, door opening and closing status, equipment status data, operating section data, and link quality data are written into the same event index to generate voice-accompanied evidence data. Within the data category transmission priority determined by the synchronous transmission strategy data, the evidence transmission order between different voice-accompanying evidence data is determined according to the target voice type. Among them, the evidence transmission order corresponding to the distress call voice is before the dispatch voice, and the evidence transmission order corresponding to the dispatch voice is before the ordinary voice. According to the order of evidence transmission, the voice-accompanying evidence data is written into the transmission queue corresponding to the synchronous transmission strategy data and transmitted.

8. The method for low-latency synchronous audio and video intercom in a cage according to claim 7, characterized in that, This also includes updating the synchronous transmission strategy data and performing abnormal cache retransmission, specifically including: The ground monitoring platform generates synchronous feedback calibration data, including playback offset direction and playback offset amount, based on the difference between the actual video playback time and the actual audio playback time. When the playback offset direction indicates that the video frame data lags behind the audio frame data, the compensation buffer duration of the audio playback path is increased, the upper limit of the video transmission queue length in subsequent acquisition cycles is reduced, and the transmission priority of video key frame data is increased. When the playback offset direction indicates that the audio frame data lags behind the video frame data, the compensation buffer duration of the video playback path is increased, the upper limit of the audio transmission queue length in subsequent acquisition cycles is reduced, and the transmission priority of audio frame data with target speech identifiers is increased. When the device status data indicates an abnormality in the main power supply, the backup power supply is controlled to take over the power supply. After the backup power supply status indicates that the power supply has taken over the power supply, the audio frame data, video key frame data, device status data, and the generated voice-accompanied evidence data are written to the local cache. When link quality data indicates an anomaly in the main link, audio frame data, video key frame data, device status data, and generated voice-accompanied evidence data are written to the local cache. The main power supply failure, the main link failure, and the main power supply failure and main link failure that occur separately during the same event duration are respectively identified as main power supply failure, main link failure, and main power supply and main link failure. Based on the failure type and the data of the operating segment, and combined with the target voice type when the target voice identifier exists, the event level is determined. The event level, occurrence time, and pending confirmation identifier are associated with the data written to the local cache to generate event cache data. When the main link recovers, select data from the event cache data that is classified as having a main link failure and has an pending confirmation flag. When the device status data indicates that the main power supply status has returned to normal and the link quality data indicates that the main link is available, select data from the event cache data that is classified as having a main power supply failure or a combined main power supply and main link failure and has an pending confirmation flag. Generate retransmission order data based on the selected data in descending order of event level and in ascending order of occurrence time. The target voice transmission bandwidth, recent audio retransmission bandwidth, and control data reserved bandwidth of the audio encoder are added together to obtain the real-time audio reserved bandwidth; the larger of the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero is taken to obtain the upper limit of the retransmission bandwidth, and the retransmission bandwidth ratio and the amount of data transmitted in a single retransmission are determined based on the upper limit of the retransmission bandwidth. Data in the local cache is retransmitted according to the retransmission order, retransmission bandwidth ratio, and single retransmission data volume. After receiving the retransmission confirmation data, the corresponding pending upload confirmation flag is updated to the uploaded confirmation flag.

9. A low-latency synchronous audio-visual intercom system for cages, characterized in that, It includes a cage-side edge control terminal, a ground monitoring platform, a main link connecting the two, and a backup power supply. The backup power supply is connected to the cage-side edge control terminal and is used to take over power supply when the main power supply is abnormal. The cage side edge control end includes: The data association module is used to associate audio frame data, video frame data, and cage operation status data according to a unified time identifier, and generate a synchronous record of the acquisition cycle. The section determination module is used to determine the operating section data based on the cage's operating status data; The feature generation module, risk determination module, and strategy generation module are respectively used to generate segment synchronization feature data containing audio and video arrival offset, offset direction, and transmission queue status based on the synchronous recording of the acquisition cycle and the data of the running segment; to determine the segment synchronization risk level based on the data of the running segment and the segment synchronization feature data; and to adjust the corresponding transmission queue parameters according to the segment synchronization risk level and offset direction. The local caching module is used to generate event cache data after the backup power supply takes over or when the main link fails. The strategy update module is used to generate compensation buffer adjustment instructions based on synchronous feedback calibration data and update the relatively lagging transmission queue parameters. When the main link is restored, it generates retransmission order data based on the event cache data formed during the main link abnormality. Or, when the main power status is restored to normal and the main link is available, it generates retransmission order data based on the event cache data formed during the main power abnormality or the joint abnormality of the main power and the main link. It also generates retransmission parameters based on the current available bandwidth and the real-time audio reserved bandwidth. The evidence transmission module is used to retransmit cached data according to the retransmission order and retransmission parameters; The ground monitoring platform includes a synchronization feedback module, which generates synchronization feedback calibration data based on the actual playback time of audio and video and sends it to the edge control terminal on the cage side. It also increases the compensation buffer duration of the relatively advanced playback path according to the compensation buffer adjustment instruction.

10. The cage audio-visual low-latency synchronous intercom system according to claim 9, characterized in that: The data association module is also used to acquire device status data, including main power supply status and backup power supply status, as well as link quality data; The synchronous feedback module is also used to determine the playback offset direction and playback offset amount based on the difference between the actual video playback time and the actual audio playback time. The strategy update module is also used to generate an audio compensation buffer adjustment instruction to increase the duration of the audio playback path compensation buffer, reduce the upper limit of the video transmission queue length, and increase the transmission priority of video keyframe data when the playback offset direction indicates that the video frame data lags behind the audio frame data; and to generate a video compensation buffer adjustment instruction to increase the duration of the video playback path compensation buffer, reduce the upper limit of the audio transmission queue length, and increase the transmission priority of audio frame data with target speech identifiers when the playback offset direction indicates that the audio frame data lags behind the video frame data. The local caching module is also used to identify main power supply failure, main link failure, and main power supply failure and main link failure that occur during the same event as main power supply failure, main link failure, and main power supply and main link combined failure, respectively. Based on the failure type and running segment data, and when there is a target voice identifier, the event level is determined by combining the target voice type. The event level, occurrence time and pending confirmation identifier are associated with the cached data to generate event cache data. The strategy update module is also used to select event cache data with an pending upload confirmation flag when the main link recovers, or select event cache data with an pending upload confirmation flag when the main power status recovers and the main link is available, and generate retransmission sequence data according to the selected event cache data in descending order of event level and in descending order of occurrence time; add the target voice transmission bandwidth, recent audio retransmission bandwidth and control data reserved bandwidth of the audio encoder to obtain the real-time audio reserved bandwidth; take the larger value between the difference between the current available bandwidth and the real-time audio reserved bandwidth and zero to obtain the upper limit of the retransmission bandwidth, and determine the retransmission bandwidth ratio and the amount of data to be retransmitted in a single time based on the upper limit of the retransmission bandwidth; The evidence transmission module is also used to retransmit cached data according to the retransmission order, the retransmission bandwidth ratio, and the amount of data retransmitted in a single transmission. After receiving the retransmission confirmation data, it updates the corresponding pending upload confirmation flag to the uploaded confirmation flag.

Images