Digital exhibition hall intelligent central control and audio-video linkage control system
By using high-precision clock synchronization and streaming media parsing, combined with linkage mapping and scheduling, the synchronous control of audio-visual equipment and environmental devices in the digital exhibition hall was achieved, solving the problem of timing misalignment and ensuring precise alignment between equipment actions and audio-visual presentation.
Patent Information
- Application Number
- CN202611221790.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-03
AI Technical Summary
In existing digital exhibition hall control systems, there is a timing misalignment between audio and video playback and environmental equipment control. Especially when there is network packet loss or fluctuations in decoding frame rate, the system cannot dynamically adapt to the actual presentation progress of the media stream, resulting in a disconnect between lighting changes, mechanical device actions, and audio and video images.
A master-slave clock topology is constructed using a high-precision clock synchronization server. The streaming media parsing gateway extracts the presentation timestamps from the audio and video data packets in real time. The linkage mapping and scheduling server binds the timestamps with the environmental device status parameters. The device execution terminal node performs local timer compensation to ensure that the actions of the environmental device are synchronized with the audio and video media stream.
It achieves synchronization between environmental equipment actions and audio/video media streams, eliminates accumulated timing errors, adapts to abnormal media stream scenarios, and avoids timing misalignment and mechanical overshoot in equipment control.
Smart Images
Figure CN123035319A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of control technology, specifically relating to an intelligent central control and audio-visual linkage control system for digital exhibition halls. Background Technology
[0002] Digital exhibition halls rely on a central control system to coordinate audio-visual playback and the actions of environmental equipment during daily operation. Existing conventional digital exhibition hall central control architectures generally employ a script-based timing control scheme using programmable logic controllers (PLCs). In this scheme, the audio / video source server and the PLC are connected to a local area network switch, operating on two independent communication links. During exhibit rehearsals, the central control programmer presets a fixed time delay list in the PLC based on the audio / video program's timeline. When the program starts, the PLC's built-in timer begins counting. Upon reaching the preset time point, the PLC sends control commands to environmental equipment such as lighting controllers and mechanical exhibit motor drivers. Simultaneously, the audio / video source server sends the audio / video media stream to the decoding terminal for independent playback. By setting a fixed lead time for the PLC to send commands, the system aligns the actions of the environmental equipment with the key frames of the audio / video footage in terms of timing.
[0003] The aforementioned independent dual-link control scheme based on fixed-time delay has a core flaw: when network packet loss, decoding frame rate fluctuations, or manual jump operations occur at the audio / video playback end, the preset fixed delay cannot dynamically adapt to the actual presentation progress of the media stream, leading to timing misalignment in multi-source heterogeneous devices. Because the timing of the programmable logic controller (PLC) is physically isolated from the actual decoding and presentation of audio / video data packets, if an anomaly occurs in the audio / video media stream causing a shift in the actual presentation time, the PLC will still issue control commands according to the original time nodes. This results in a disconnect between lighting changes, mechanical device actions, and the audio / video images, producing an unavoidable cumulative timing error. Summary of the Invention
[0004] The purpose of this invention is to provide an intelligent central control and audio-visual linkage control system for digital exhibition halls, which can effectively solve the problems mentioned in the background art.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: The intelligent central control and audio-visual linkage control system for digital exhibition halls includes a streaming media parsing gateway, a high-precision clock synchronization server, a linkage mapping and scheduling server, and device execution terminal nodes; The high-precision clock synchronization server uses a precise time protocol to construct a master-slave clock topology and distributes the synchronization clock source to the streaming media parsing gateway and the device execution terminal node. The streaming media parsing gateway extracts the presentation timestamp from the audio and video data packets being transmitted in real time; The linkage mapping scheduling server has a built-in time axis linkage mapping table, which binds the numerical range of the presentation timestamp with the status parameters of the environmental device. The linkage mapping scheduling server compares the presentation timestamp with the time axis linkage mapping table in real time. When the presentation timestamp enters the preset numerical range, an environmental device control message carrying the target execution timestamp is generated. The device execution terminal node receives the environmental device control message, extracts the target execution timestamp, calculates the difference between the target execution timestamp and the local precise time protocol clock, performs local timer compensation based on the difference, and drives the environmental device to perform the corresponding state switching action when the local precise time protocol clock reaches the target execution timestamp.
[0006] Preferably, the high-precision clock synchronization server is deployed on the underlying control local area network, and the high-precision clock synchronization server serves as the grandmaster clock node of the precision time protocol. The streaming media parsing gateway and the device execution terminal node are both configured as ordinary clock nodes. The high-precision clock synchronization server periodically broadcasts precise time protocol synchronization messages and follow-up messages in the underlying control LAN. The streaming media parsing gateway and the device execution terminal node calculate the link delay deviation based on the precise time protocol synchronization messages and follow-up messages, and calibrate the clock offset of the local precise time protocol clock based on the link delay deviation.
[0007] Preferably, the streaming media parsing gateway is connected between the audio / video source server and the switch, and the streaming media parsing gateway performs deep packet parsing on the transport stream data packets sent by the audio / video source server; For video data packets, the streaming media parsing gateway decapsulates the program mapping table segment and the basic stream segment to obtain the video presentation timestamp; For audio data packets, the streaming media parsing gateway decapsulates and repackages the basic stream to obtain the audio presentation timestamp; The streaming media parsing gateway encapsulates the video presentation timestamp and the audio presentation timestamp obtained from the parsing into a custom event notification message, and sends the custom event notification message to the linkage mapping scheduling server.
[0008] Preferably, the linkage mapping scheduling server internally constructs a memory circular buffer based on time axis rotation as the time axis linkage mapping table, and each slot of the memory circular buffer corresponds to a numerical range of the presentation timestamp; After receiving the presentation timestamp, the linkage mapping scheduling server uses the value of the presentation timestamp as an index pointer to perform address matching in the memory circular buffer. When a corresponding slot is matched, the linkage mapping scheduling server reads the status parameters of the environmental device and the target execution timestamp pre-stored in the slot, and assembles the status parameters and the target execution timestamp according to a preset binary protocol format to generate the environmental device control message.
[0009] Preferably, the device execution terminal node is equipped with a hardware timer chip and a microprocessor. After receiving the environmental device control message, the microprocessor strips the message header fields to extract the target execution timestamp and device address code. The microprocessor reads the current count value of the hardware timer chip as the current local precise time protocol clock, and performs a subtraction operation between the current local precise time protocol clock and the target execution timestamp to generate the difference; The microprocessor converts the difference into the number of clock cycles of the hardware timer chip and writes the number of clock cycles into the compare register of the hardware timer chip.
[0010] Preferably, the hardware timer chip integrates automatic reload logic and an interrupt controller. When the current count value of the hardware timer chip increases to be equal to the number of clock cycles in the comparison register, the interrupt controller outputs a hardware interrupt level signal to the microprocessor. The microprocessor responds to the hardware interrupt level signal, addresses the local peripheral bus according to the device address code, and outputs a pulse width modulation waveform or a high-low level toggle signal to the environmental device driver circuit connected to the local peripheral bus to perform the state switching action.
[0011] Preferably, both the streaming media parsing gateway and the device execution terminal node are equipped with a network physical layer chip that supports hardware tagging of the Precision Time Protocol. When the network physical layer chip receives the start delimiter of the physical layer medium access control frame of the precise time protocol synchronization message, it captures the current snapshot value of the nanosecond-level counter inside the network physical layer chip and writes the current snapshot value into the reserved field of the precise time protocol synchronization message. The streaming media parsing gateway and the software process of the ordinaryclock node in the device execution terminal node directly read the current snapshot value in the reserved field to calculate the link delay deviation.
[0012] Preferably, the streaming media parsing gateway internally maintains a presentation timestamp continuity state machine, and the streaming media parsing gateway compares the currently received video presentation timestamp or audio presentation timestamp with the historical presentation timestamp of the previous parsing cycle by difference. When the difference exceeds the preset single-frame duration threshold or the difference is negative, the continuous state machine switches to a jump abnormal state. In the abnormal jump state, the streaming media parsing gateway marks the currently received video presentation timestamp or audio presentation timestamp as a resynchronization anchor timestamp, and sends it after replacing the regular timestamp field in the custom event notification message with the resynchronization anchor timestamp.
[0013] Preferably, each slot of the memory ring buffer stores a multi-device linkage descriptor, which includes status parameters of multiple environmental devices and device priority identifiers; When the linkage mapping scheduling server matches the status parameters of multiple environmental devices with the same target execution timestamp within a single numerical range, the linkage mapping scheduling server reorders the hardware bus time slice occupied by the status parameters of the environmental devices according to the device priority identifier, and arranges the status parameters of the environmental devices corresponding to the high priority identifier at the front end of the data payload of the environmental device control message.
[0014] Preferably, a mechanical limit status register is connected to the local peripheral bus of the device execution terminal node, and the mechanical limit status register collects the physical limit switch signals of the corresponding environmental devices in real time. Before the microprocessor outputs the pulse width modulation waveform or the high-low level toggling signal to the environmental device driving circuit, the microprocessor reads the current level state in the mechanical limit status register; When the current level state indicates that the corresponding environmental device is at a physical limit position, the microprocessor shields the hardware interrupt level signal and stops outputting drive signals to the environmental device drive circuit.
[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention constructs a master-slave clock topology using a high-precision clock synchronization server and a precise time protocol. A streaming media parsing gateway extracts the presentation timestamps from audio and video data packets in real time. A linkage mapping and scheduling server binds the numerical range of the presentation timestamps to environmental device status parameters based on a timeline linkage mapping table, generating an environmental device control message carrying the target execution timestamp. The device execution terminal node extracts the target execution timestamp and calculates the difference with its local precise time protocol clock for local timer compensation. This scheme changes the control trigger source of the environmental device from the independent time node of the programmable logic controller to the presentation timestamp within the audio and video data packets. This allows the actions of the environmental device to directly follow the actual presentation progress of the audio and video media stream, eliminating the cumulative timing errors caused by independent dual links and achieving coordinated action of multi-source heterogeneous devices under a unified clock reference.
[0016] 2. The network physical layer chip captures a nanosecond-level counter snapshot value when it receives the start delimiter of the media access control frame of the precise time protocol synchronization message, reducing the calculation error of link delay deviation; the streaming media parsing gateway internally maintains a timestamp continuity state machine, marking the resynchronization anchor timestamp when the difference exceeds the single frame duration threshold or is negative, adapting to scenarios where media stream jumps abnormally; the linkage mapping scheduling server reorders the hardware bus occupancy time slices of the status parameters in the multi-device linkage descriptor according to the device priority identifier, avoiding bus data congestion; the microprocessor of the device execution terminal node reads the current level state of the mechanical limit status register before outputting the drive signal, and shields the hardware interrupt level signal at the physical limit position, preventing mechanical overshoot damage to environmental equipment. Attached Figure Description
[0017] Figure 1 This is a flowchart illustrating the overall execution of the intelligent central control and audio-visual linkage control system for digital exhibition halls according to the present invention. Figure 2 This is a flowchart of the high-precision clock synchronization and hardware marking calibration process of the present invention; Figure 3 This is a flowchart of the streaming media parsing and redirection exception handling process of the present invention; Figure 4 This is a flowchart of the time axis mapping addressing and multi-device priority scheduling process of the present invention; Figure 5 This is a flowchart illustrating the local timer compensation and hardware interrupt triggering process of the present invention. Figure 6 This is a flowchart of the mechanical limit protection and drive signal output of the present invention. Detailed Implementation
[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0019] Please refer to Figure 1This embodiment provides an intelligent central control and audio-visual linkage control system for a digital exhibition hall, deployed within a dedicated underlying control LAN of the digital exhibition hall. The system includes a streaming media parsing gateway, a high-precision clock synchronization server, a linkage mapping and scheduling server, and device execution terminal nodes. The high-precision clock synchronization server uses a precise time protocol to construct a master-slave clock topology. During the construction of this topology, the high-precision clock synchronization server acts as the sole clock reference source within the entire control LAN, distributing synchronization clock sources to all connected streaming media parsing gateways and device execution terminal nodes, ensuring that the local clocks of all nodes are under the same time reference and eliminating clock discrepancies between different nodes. The streaming media parsing gateway is connected in series between the audio-visual source server and the core switch within the exhibition hall. It captures audio-visual transmission stream data packets sent by the audio-visual source server in real time, performs protocol parsing on the captured data packets, and extracts the presentation timestamp from the transmitting audio-visual data packets. The presentation timestamp is an absolute time value embedded during the audio-visual encoding process, used to identify the corresponding audio-visual frame that should be rendered after decoding. This time value is completely bound to the timeline of the audio-visual program and is updated synchronously with the real-time transmission of the audio-visual stream. The linkage mapping scheduling server has a built-in timeline linkage mapping table. Based on the complete timeline of the audio-visual program, this table binds the numerical range of the presentation timestamps to the status parameters of various environmental devices within the exhibition hall. Each numerical range corresponds to a specific time segment in the audio-visual program, and all parameters for the state switching actions that the corresponding environmental devices need to perform within each time segment are pre-stored in the timeline linkage mapping table. The linkage mapping scheduling server receives the presentation timestamps sent by the streaming media parsing gateway in real time and compares them with the preset numerical ranges in the timeline linkage mapping table. When a presentation timestamp enters a certain preset numerical range, the linkage mapping scheduling server generates a corresponding environmental device control message. This message carries the target execution timestamp bound to that numerical range. The target execution timestamp is the absolute time point at which the environmental device performs the corresponding state switching action. This time point maintains a preset synchronization offset with the presentation timestamps of the audio-visual frames, ensuring that the actions of the environmental devices are perfectly aligned with the rendering of the audio-visual images. The device execution terminal nodes are deployed near the various environmental devices in the exhibition hall, and maintain communication connections with the linkage mapping and scheduling server and the high-precision clock synchronization server through the underlying control LAN.The device execution terminal node receives the environmental device control message issued by the linkage mapping scheduling server, performs protocol parsing on the message, extracts the target execution timestamp carried in the message, and reads the current value of the local precise time protocol clock after it has been calibrated by the precise time protocol. It calculates the difference between the target execution timestamp and the current value of the local precise time protocol clock, and compensates the trigger time of the local timer based on the calculated difference to eliminate the link delay deviation and the timekeeping deviation of the local clock generated during message transmission. When the value of the local precise time protocol clock reaches the target execution timestamp, it drives the corresponding connected environmental device to perform the state switching action corresponding to the state parameters.
[0020] Specifically, the basic binding structure of the time axis linkage mapping table used in this embodiment is shown in Table 1: Table 1. Basic Binding Structure of Time Axis Linkage Mapping Table The basic binding structure of the timeline linkage mapping table presented in Table 1 corresponds to continuous numerical intervals on the audio and video program timeline. The length of each numerical interval can be configured according to the control precision requirements of the exhibit. The minimum granularity of the numerical interval is consistent with the single frame duration of the audio and video stream, ensuring that the presentation timestamp corresponding to each audio and video frame can be matched with the corresponding binding entry. The value of the target execution timestamp does not exceed the upper limit of the corresponding presentation timestamp value interval, ensuring that the state switching action is prepared before the presentation time of the corresponding audio and video frame and is precisely triggered at the target execution time.
[0021] In this embodiment, a high-precision clock synchronization server provides a unified clock reference for all system nodes, directly binding the control trigger source of the environmental devices to the presentation timestamp inside the audio and video data packets. This ensures that the execution of the environmental devices' actions completely follows the actual presentation progress of the audio and video media stream, avoiding timing misalignment between the two independent links of audio and video playback and device control, eliminating the cumulative timing error caused by the fixed delay scheme, and achieving time synchronization between audio and video playback and the actions of multiple environmental devices.
[0022] In a preferred embodiment, reference Figure 2The high-precision clock synchronization server is deployed at the core node of the underlying control LAN. The underlying control LAN adopts a full gigabit wired Ethernet architecture, and all network forwarding devices within the LAN support the transparent clock transmission mode of the precision time protocol, ensuring that the transmission latency jitter of synchronization messages is controlled within the nanosecond range. As the grandmaster clock node of the precision time protocol, the high-precision clock synchronization server uses a high-stability clock signal provided by a temperature-controlled crystal oscillator as its local clock source, with a clock frequency stability of no less than ±0.1ppm, providing the sole master clock reference for the entire LAN. The streaming media parsing gateway and all device execution terminal nodes are configured as ordinary clock nodes of the precision time protocol. All ordinary clock nodes calibrate their local clocks using the clock source of the grandmaster clock node as the sole reference. Furthermore, the high-precision clock synchronization server broadcasts precise time protocol synchronization messages and follow-up messages at a preset fixed period in the underlying control LAN. The broadcast period of the synchronization message can be configured to 125ms, 250ms, 500ms, or 1s. In this embodiment, the broadcast period of the synchronization message is set to 125ms to increase the frequency of local clock calibration and reduce the accumulation of clock timing deviations. The synchronization message carries the estimated transmission timestamp of the grandmaster clock node when sending the synchronization message. The follow-up message is sent immediately after the synchronization message is sent, and carries the precise transmission timestamp of the grandmaster clock node when it actually sends the frame start delimiter of the synchronization message at the physical layer. After receiving the synchronization and follow messages, the ordinaryclock node in the streaming media parsing gateway and device execution terminal node calculates the link delay deviation between the local clock and the grandmaster clock based on the timestamp information in the synchronization and follow messages and the precise reception timestamp when the local clock receives the synchronization message. It then calibrates the clock offset of the local precise time protocol clock based on the calculated link delay deviation, so that the local clock keeps perfectly synchronized with the grandmaster clock.
[0023] Specifically, the clock offset between the local clock and the grandmaster clock Calculated using the following formula: in, This is the precise timestamp of the frame start delimiter when the grandmaster node sends the synchronization message at the physical layer; this value is carried by the follow-up message. This is the precise timestamp of the ordinaryclock node when it receives the start-of-frame delimiter of the synchronization message at the physical layer. The precise transmission timestamp when the ordinaryclock node sends the start-of-frame delimiter of the delay request message at the physical layer; This is the precise timestamp of the frame start delimiter received by the grandmaster node at the physical layer when it receives the delay request message. This value is carried in the delay response message returned by the master clock node.
[0024] Furthermore, the link transmission delay between the master clock node and the ordinary clock node Calculated using the following formula: The physical meaning of each parameter in the formula is consistent with that of the parameters in the clock offset calculation formula above.
[0025] After the Ordinaryclock node completes the calculation of the clock offset and link transmission delay, it calibrates the local precision time protocol clock. The calibrated local clock value is calculated using the following formula: in, This is the original count value of the local clock of the ordinaryclock node, which is updated in real time by the local hardware clock counter; This is the calibrated local precision time protocol clock value, which is synchronized with the reference time of the grandmaster master clock.
[0026] Preferably, both the streaming media parsing gateway and the device execution terminal node are equipped with a network physical layer chip that supports hardware marking for the Precision Time Protocol (PTP). The PTP chip communicates with the corresponding microprocessor via a media-independent interface (MIDI). This MDI can be a gigabit MDI or a simplified gigabit MDI, ensuring bandwidth and real-time performance for data transmission. Upon receiving the start delimiter of the physical layer media access control frame of the PTP synchronization message, the PTP chip immediately triggers its internal nanosecond-level counter snapshot capture action. It captures the current snapshot value of the internal nanosecond-level counter and writes it into the reserved field of the PTP synchronization message. This process eliminates time jitter and delay caused by software interrupts, without requiring intervention from the microprocessor's software process. The ordinaryclock node software processes in the streaming media parsing gateway and the device execution terminal node do not need to perform software estimation of the message reception time; they directly read the current snapshot value from the reserved field of the synchronization message as the precise reception timestamp. This participates in the calculation of link delay deviation and clock offset, significantly reducing the error in timestamp capture. Similarly, when the ordinary clock node sends a delay request message, the network physical layer chip captures the current snapshot value of its internal nanosecond-level counter at the start delimiter of the medium access control frame of the sent message, using it as the accurate transmission timestamp. Write this information into the corresponding field of the delayed request message to ensure the accuracy of the sending timestamp.
[0027] Specifically, the correspondence between the hardware tagging fields of the precise time protocol message in this embodiment is shown in Table 2: Table 2. Correspondence Table of Hardware Marking Fields for Precision Time Protocol Messages The hardware tagging field correspondence defined in Table 2 clarifies the timestamp capture trigger point and storage location for different types of precise time protocol messages. All timestamp captures are completed at the time of frame start delimiter transmission in the physical layer, avoiding delay jitter caused by protocol processing in the data link layer, network layer, and transport layer, ensuring that the timestamp capture accuracy reaches the nanosecond level, and providing a high-precision time reference for clock synchronization of the entire system.
[0028] In this embodiment, a master-slave clock topology is constructed by the grandmaster node and the ordinaryclock node, combined with the hardware timestamp marking mechanism of the physical layer chip, to achieve nanosecond-level clock synchronization of all nodes in the entire system. This eliminates clock timing deviations between different nodes and delay jitter in the message transmission link, providing a unified time reference for the accurate comparison of audio and video presentation timestamps and device execution timestamps, and ensuring the time synchronization accuracy of environmental device state switching actions and audio and video image presentation.
[0029] In a preferred embodiment, reference Figure 3The streaming media parsing gateway adopts a serial deployment architecture, connecting the audio / video source server and the core switch of the exhibition hall. The gateway's two network interfaces are configured as a LAN port and a WAN port, respectively. The WAN port is directly connected to the network output port of the audio / video source server, and the LAN port is directly connected to the uplink port of the core switch. All audio / video transmission stream data packets sent by the audio / video source server must be bidirectionally forwarded through the streaming media parsing gateway to ensure that it can capture complete audio / video transmission stream data and parse the presentation timestamp in each data packet without omission. The gateway performs deep packet parsing on the transmission stream data packets sent by the audio / video source server. The transmission stream is encapsulated in MPEG-2TS format, and each data packet has a fixed length of 188 bytes, including a 4-byte transmission stream header and 184 bytes of payload data. For transport stream data packets containing video data, the streaming media parsing gateway first parses the transport stream packet header to obtain the packet identifier. Based on the packet identifier, it matches the corresponding program mapping table segment. After parsing the program mapping table segment, it obtains the packet identifier and stream type information corresponding to the video primary stream. Then, it decapsulates the video primary stream segment, extracts the video access units (VIDs) packaged within the video primary stream, and parses the video rendering timestamp from the header fields of the VIDs. The video rendering timestamp is a 33-bit unsigned integer, based on a 90kHz system clock frequency, corresponding to the decoded rendering time of the video frame. For transport stream data packets containing audio data, the streaming media parsing gateway obtains the packet identifier corresponding to the audio primary stream through the program mapping table segment. It decapsulates the packaged primary stream corresponding to the audio primary stream, extracts the audio rendering timestamp from the header of the packaged primary stream. The audio rendering timestamp is also based on a 90kHz system clock frequency, corresponding to the decoded playback time of the audio frame.
[0030] Specifically, the nominal step value of the video and audio rendering timestamps obtained by the streaming media parsing gateway is determined by the nominal frame rate and sampling rate of the audio and video streams. The nominal rendering timestamp step value corresponding to a single video frame is calculated using the following formula: in, The nominal frame rate of the video stream, measured in frames per second; This is the nominal step value of the timestamp corresponding to a single frame of video, in units of 90kHz clock cycles.
[0031] Correspondingly, single frame duration threshold Calculated using the following formula: in, In this embodiment, the preset threshold coefficient is used. The value is 1.5, which is used to determine whether there is an anomaly in the continuity of the presented timestamps.
[0032] The streaming media parsing gateway synchronizes and aligns the obtained video and audio presentation timestamps, then encapsulates them into a custom event notification message. This message is then sent in real-time to the linkage mapping and scheduling server via the underlying control LAN. The custom event notification message is encapsulated using the User Datagram Protocol (UDP). The message header includes a message type identifier, audio / video stream identifiers, sequence number, and checksum fields. The message payload contains the aligned video and audio presentation timestamps, along with a timestamp status identifier, ensuring the linkage mapping and scheduling server can quickly parse and obtain the real-time presentation progress of the current audio and video streams.
[0033] Preferably, the streaming media parsing gateway internally maintains a presentation timestamp continuity state machine. This state machine includes two stable states: a normal operation state and a transition exception state. The default initial state of the state machine is the normal operation state. In the normal operation state, the streaming media parsing gateway compares the currently received and parsed video or audio presentation timestamp with the historical presentation timestamps stored in the previous parsing cycle. The difference in presentation timestamps is calculated using the following formula: in, The video or audio rendering timestamp obtained for the current parsing cycle; The historical presentation timestamp stored for the previous parsing cycle; This represents the difference in presentation timestamps between two consecutive parsing cycles.
[0034] When calculated Exceeding the preset single frame duration threshold ,or A negative value indicates that the audio / video stream has experienced playback progress jumps, frame loss due to packet loss, or decoding anomalies. In this case, the continuity state machine switches from the normal operating state to the jump-abnormal state. In the jump-abnormal state, the streaming media parsing gateway marks the currently received video or audio presentation timestamp as the resynchronization anchor timestamp, and the correction value of the resynchronization anchor timestamp... Calculated using the following formula: in, The link propagation delay for custom event notification messages transmitted from the streaming media parsing gateway to the linkage mapping and scheduling server is updated in real time using the link delay calculation results of the precise time protocol. This is the corrected resynchronization anchor timestamp. The streaming media parsing gateway replaces the regular timestamp field in the custom event notification message with the resynchronization anchor timestamp, and sets the timestamp status flag field to an abnormal jump flag before sending it to the linkage mapping scheduling server. This allows the linkage mapping scheduling server to immediately adjust the timeline comparison benchmark based on the resynchronization anchor timestamp, avoiding control timing misalignment caused by audio and video stream jumps. When the difference in presentation timestamps within three consecutive parsing cycles... When all values are within the single frame duration threshold and are positive, the timestamp continuity state machine switches from the abnormal state back to the normal operating state, restoring the normal timestamp parsing and reporting process.
[0035] Specifically, the correspondence between MPEG-2TS stream packet parsing and rendering timestamp extraction in this embodiment is shown in Table 3: Table 3. Correspondence between MPEG-2 TS stream packet parsing and rendering timestamp extraction The TS stream packet parsing process defined in Table 3 clarifies the parsing level and the extraction position of the presentation timestamp for different types of transport stream data packets. The streaming media parsing gateway parses each transport stream data packet according to the parsing rules in the table, ensuring that the presentation timestamp corresponding to each frame in the audio and video stream can be extracted without omission, providing real-time and accurate audio and video presentation progress data for time axis comparison of the linkage mapping and scheduling server.
[0036] In this embodiment, a serially deployed streaming media parsing gateway performs deep packet parsing on the audio and video transmission streams, directly extracting the presentation timestamp from the encapsulation structure of the audio and video data packets. This ensures that the obtained timestamps are completely bound to the actual presentation progress of the audio and video streams. A presentation timestamp continuity state machine monitors the continuity of timestamps in real time, quickly triggering a resynchronization mechanism when playback jumps or anomalies occur in the audio and video streams. This allows the linkage control process to dynamically adapt to the actual playback progress of the audio and video streams, avoiding the shortcomings of fixed-sequence schemes that cannot cope with changes in playback progress.
[0037] In a preferred embodiment, reference Figures 4 to 6The linkage mapping scheduling server internally constructs a time-axis-based circular memory buffer as a time-axis linkage mapping table. This circular memory buffer is deployed in the kernel-mode memory space of the linkage mapping scheduling server to avoid latency caused by memory copying between user mode and kernel mode, ensuring real-time address matching. Each slot in the circular memory buffer corresponds to a value range of a presentation timestamp. All slots are arranged consecutively in time-axis order, forming a ring-shaped storage structure. The storage space of each slot is fixed and can store a preset number of device linkage descriptors. After receiving the presentation timestamp sent by the streaming media parsing gateway, the linkage mapping scheduling server uses the presentation timestamp value as an index pointer to perform address matching in the circular memory buffer. The index value for address matching is calculated using the following formula: in, The received presentation timestamp value; The length of the timestamp value range corresponding to each slot, in 90kHz clock cycles; The total number of slots in the memory circular buffer. mod is the modulo operation; This is the index value of the corresponding slot in the memory circular buffer.
[0038] The linkage mapping scheduling server directly accesses the corresponding slot in the memory circular buffer based on the calculated index value. When a valid device linkage descriptor is stored in the matched slot, it indicates that the current presentation timestamp has entered the preset value range. At this time, the linkage mapping scheduling server reads the pre-stored environmental device status parameters and the target execution timestamp from the slot, and assembles the read status parameters and target execution timestamp according to the preset binary protocol format to generate an environmental device control message. The total length of the environmental device control message is calculated using the following formula: in, Fixed header length for environmental equipment control messages; The number of valid device linkage descriptors in the current slot; The length in bytes of the i-th device linkage descriptor; The total byte length of the assembled environmental equipment control message.
[0039] Preferably, the device linkage descriptor stored in each slot of the memory circular buffer is a multi-device linkage descriptor. The multi-device linkage descriptor includes the status parameters of multiple environmental devices, device address codes, target execution timestamps, and device priority identifiers. The device priority identifier is an 8-bit unsigned integer; the smaller the value, the higher the priority. It can be configured according to the action response speed and synchronization accuracy requirements of the environmental devices. When the linkage mapping scheduling server matches the status parameters of multiple environmental devices with the same target execution timestamp within a single presentation timestamp value range, it indicates that multiple environmental devices need to perform state switching actions at the same time. At this time, the linkage mapping scheduling server reorders the hardware bus time slice occupied by the status parameters of the environmental devices according to the device priority identifier. The weight value of the sorting is calculated using the following formula: in, This is the numerical value representing the device priority identifier for the corresponding environmental equipment; The length of the executable time window corresponding to the target execution timestamp, in milliseconds; The ranking weights of the corresponding environmental devices are used, with smaller weights ranking higher. The linkage mapping and scheduling server rearranges the status parameters of the environmental devices according to their weights from smallest to largest, placing the status parameters of the environmental devices corresponding to high-priority identifiers at the front of the data payload of the environmental device control messages. This ensures that the device execution terminal nodes prioritize receiving and processing control commands from high-priority devices, avoiding hardware bus data congestion caused by multiple devices sending control commands simultaneously. This ensures that devices requiring high synchronization accuracy can complete their actions precisely at the target execution timestamp.
[0040] Furthermore, the device execution terminal node is equipped with a hardware timer chip and a microprocessor. The hardware timer chip is a 32-bit auto-incrementing timer, internally integrating auto-reload logic and an interrupt controller. The clock input source of the hardware timer chip is a local clock signal calibrated with a precise time protocol, ensuring that the timer's count value is completely synchronized with the local precise time protocol clock. After receiving the environmental device control message from the linkage mapping scheduling server, the microprocessor first performs link layer and network layer parsing on the message, stripping the message header fields, and extracting the target execution timestamp, device address code, and corresponding environmental device status parameters carried in the message. The microprocessor reads the current count value of the hardware timer chip through the serial peripheral interface, converts the current count value into the value of the current local precise time protocol clock, and performs a subtraction operation between the current local precise time protocol clock value and the target execution timestamp to generate the difference between the two. The formula for calculating the difference is: in, The target execution timestamp extracted from the message; The value of the current local precision time protocol clock; The difference between the target timestamp and the local clock.
[0041] when When the value is positive, it indicates that the target execution time point has not yet been reached. The microprocessor converts the calculated difference into the number of clock cycles of the hardware timer chip. The conversion formula is as follows: in, This refers to the operating frequency of the hardware timer chip, measured in Hertz (Hz). This represents the number of clock cycles that the hardware timer chip needs to count.
[0042] The microprocessor calculates the reload value of the compare register based on the current count value of the hardware timer chip. The calculation formula is as follows: in, This is the current count value of the hardware timer chip; The overload value is written to the compare register. The microprocessor writes the calculated overload value into the compare register of the hardware timer chip to complete the compensation configuration of the local timer and eliminate timing errors caused by message transmission delay and local clock timing deviation.
[0043] The hardware timer chip's counter continuously increments at a preset operating frequency. When the counter's current count value equals the overload value in the compare register, the internal interrupt controller immediately outputs a hardware interrupt level signal to the microprocessor. This interrupt signal is a non-maskable interrupt, ensuring the microprocessor can respond immediately without interference from other software processes. After responding to the hardware interrupt level signal, the microprocessor immediately stops the currently executing low-priority task, reads the address code and status parameters of the corresponding device from memory, and addresses the local peripheral bus based on the device address code. The local peripheral bus can be an RS485 bus, a controller area network bus, or a direct input / output bus. The microprocessor outputs a pulse width modulation waveform or a high / low level toggle signal to the corresponding environmental device driver circuit connected to the local peripheral bus, driving the environmental device to perform a state switching action corresponding to the status parameters.
[0044] Preferably, a mechanical limit status register is connected to the local peripheral bus of the device execution terminal node. The mechanical limit status register is a 16-bit parallel input register, with each input channel corresponding to a physical limit switch signal of an environmental device. The physical limit switches include a forward limit switch and a reverse limit switch, respectively installed at the two extreme positions of the mechanical movement structure of the environmental device. When the mechanical movement structure reaches its extreme position, the corresponding limit switch is triggered, outputting a corresponding level signal. The mechanical limit status register acquires the physical limit switch signal of the corresponding environmental device in real time and latches the acquired level status into an internal register. The microprocessor can read the level status in the register in real time via the local peripheral bus. Before the microprocessor outputs a pulse width modulation waveform or a high / low level toggle signal to the environmental device drive circuit, the microprocessor first reads the current level status in the mechanical limit status register, analyzes the current level status, and determines whether the corresponding environmental device is at its physical limit position. When the current voltage level indicates that the corresponding environmental device is at its physical limit position, and the upcoming state switching action will drive the mechanical moving structure to continue moving towards that limit position, the microprocessor immediately blocks the corresponding hardware interrupt voltage level signal, stops outputting drive signals to the environmental device drive circuit, and generates a fault alarm message to report to the linkage mapping and scheduling server to prevent the mechanical moving structure of the environmental device from overshooting and avoid damage to the mechanical structure.
[0045] Specifically, the correspondence between environmental device status parameters and drive signals in this embodiment is shown in Table 4: Table 4 Correspondence between Environmental Equipment Status Parameters and Drive Signals The correspondence between environmental device status parameters and drive signals defined in Table 4 clarifies the drive signal types and configuration methods corresponding to the status parameters of different types of environmental devices. The device execution terminal node converts the received status parameters into corresponding drive signals according to the correspondence in the table, ensuring that the environmental devices can accurately execute preset state switching actions. Meanwhile, the device priority identifiers configured in the table can be adjusted according to the synchronization accuracy requirements of the devices, providing a configuration basis for the bus time slice sequencing of multi-device linkage.
[0046] In this embodiment, a timeline linkage mapping table built based on a memory circular buffer enables fast addressing and matching of presentation timestamps, improving the real-time performance of control message generation. A device priority sorting mechanism avoids bus congestion when multiple devices trigger simultaneously, ensuring the synchronization accuracy of high-priority devices. An interrupt triggering mechanism using a hardware timer enables precise timing control of environmental device state switching actions. A real-time monitoring and protection mechanism using mechanical limit status registers prevents mechanical overshoot damage to environmental devices, improving the stability and reliability of system operation.
Claims
1. A digital exhibition hall intelligent central control and audio-visual linkage control system, characterized in that, This includes a streaming media parsing gateway, a high-precision clock synchronization server, a linkage mapping and scheduling server, and device execution terminal nodes; The high-precision clock synchronization server uses a precise time protocol to construct a master-slave clock topology and distributes the synchronization clock source to the streaming media parsing gateway and the device execution terminal node. The streaming media parsing gateway extracts the presentation timestamp from the audio and video data packets being transmitted in real time; The linkage mapping scheduling server has a built-in time axis linkage mapping table, which binds the numerical range of the presentation timestamp with the status parameters of the environmental device. The linkage mapping scheduling server compares the presentation timestamp with the time axis linkage mapping table in real time. When the presentation timestamp enters the preset numerical range, an environmental device control message carrying the target execution timestamp is generated. The device execution terminal node receives the environmental device control message, extracts the target execution timestamp, calculates the difference between the target execution timestamp and the local precise time protocol clock, performs local timer compensation based on the difference, and drives the environmental device to perform the corresponding state switching action when the local precise time protocol clock reaches the target execution timestamp.
2. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 1, characterized in that, The high-precision clock synchronization server is deployed on the underlying control local area network. The high-precision clock synchronization server serves as the grandmaster clock node of the precision time protocol. The streaming media parsing gateway and the device execution terminal node are both configured as ordinary clock nodes. The high-precision clock synchronization server periodically broadcasts precise time protocol synchronization messages and follow-up messages in the underlying control LAN. The streaming media parsing gateway and the device execution terminal node calculate the link delay deviation based on the precise time protocol synchronization messages and follow-up messages, and calibrate the clock offset of the local precise time protocol clock based on the link delay deviation.
3. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 1, characterized in that, The streaming media parsing gateway is connected between the audio / video source server and the switch. The streaming media parsing gateway performs deep packet parsing on the data packets of the transmission stream sent by the audio / video source server. For video data packets, the streaming media parsing gateway decapsulates the program mapping table segment and the basic stream segment to obtain the video presentation timestamp; For audio data packets, the streaming media parsing gateway decapsulates and repackages the basic stream to obtain the audio presentation timestamp; The streaming media parsing gateway encapsulates the video presentation timestamp and the audio presentation timestamp obtained from the parsing into a custom event notification message, and sends the custom event notification message to the linkage mapping scheduling server.
4. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 1, characterized in that, The linkage mapping scheduling server internally constructs a memory circular buffer based on time axis rotation as the time axis linkage mapping table, and each slot of the memory circular buffer corresponds to a numerical range of the presentation timestamp. After receiving the presentation timestamp, the linkage mapping scheduling server uses the value of the presentation timestamp as an index pointer to perform address matching in the memory circular buffer. When a corresponding slot is matched, the linkage mapping scheduling server reads the status parameters of the environmental device and the target execution timestamp pre-stored in the slot, and assembles the status parameters and the target execution timestamp according to a preset binary protocol format to generate the environmental device control message.
5. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 1, characterized in that, The device execution terminal node is equipped with a hardware timer chip and a microprocessor. After receiving the environmental device control message, the microprocessor strips the message header fields to extract the target execution timestamp and device address code. The microprocessor reads the current count value of the hardware timer chip as the current local precise time protocol clock, and performs a subtraction operation between the current local precise time protocol clock and the target execution timestamp to generate the difference; The microprocessor converts the difference into the number of clock cycles of the hardware timer chip and writes the number of clock cycles into the compare register of the hardware timer chip.
6. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 5, characterized in that, The hardware timer chip integrates automatic reload logic and an interrupt controller. When the current count value of the hardware timer chip increases to be equal to the number of clock cycles in the comparison register, the interrupt controller outputs a hardware interrupt level signal to the microprocessor. The microprocessor responds to the hardware interrupt level signal, addresses the local peripheral bus according to the device address code, and outputs a pulse width modulation waveform or a high-low level toggle signal to the environmental device driver circuit connected to the local peripheral bus.
7. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 2, characterized in that, Both the streaming media parsing gateway and the device execution terminal node are equipped with a network physical layer chip that supports hardware tagging of the precise time protocol. When the network physical layer chip receives the start delimiter of the physical layer medium access control frame of the precise time protocol synchronization message, it captures the current snapshot value of the nanosecond-level counter inside the network physical layer chip and writes the current snapshot value into the reserved field of the precise time protocol synchronization message. The streaming media parsing gateway and the ordinaryclock node software process in the device execution terminal node read the current snapshot value in the reserved field to calculate the link delay deviation.
8. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 3, characterized in that, The streaming media parsing gateway internally maintains a presentation timestamp continuity state machine, and the streaming media parsing gateway compares the currently received video presentation timestamp or audio presentation timestamp with the historical presentation timestamp of the previous parsing cycle by difference. When the difference exceeds the preset single-frame duration threshold or the difference is negative, the continuous state machine switches to the jump abnormal state. In the abnormal jump state, the streaming media parsing gateway marks the currently received video presentation timestamp or audio presentation timestamp as a resynchronization anchor timestamp, and sends it after replacing the regular timestamp field in the custom event notification message with the resynchronization anchor timestamp.
9. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 4, characterized in that, Each slot of the memory ring buffer stores a multi-device linkage descriptor, which includes the status parameters of multiple environmental devices and device priority identifiers. When the linkage mapping scheduling server matches the status parameters of multiple environmental devices with the same target execution timestamp within a single numerical range, the linkage mapping scheduling server reorders the hardware bus time slice occupied by the status parameters of the environmental devices according to the device priority identifier, and arranges the status parameters of the environmental devices corresponding to the high priority identifier at the front end of the data payload of the environmental device control message.
10. The intelligent central control and audio-visual linkage control system for digital exhibition halls according to claim 6, characterized in that, The device execution terminal node has a mechanical limit status register connected to its local peripheral bus. The mechanical limit status register collects physical limit switch signals of the corresponding environmental devices in real time. Before the microprocessor outputs the pulse width modulation waveform or the high-low level toggling signal to the environmental device driving circuit, the microprocessor reads the current level state in the mechanical limit status register; When the current level state indicates that the corresponding environmental device is at a physical limit position, the microprocessor shields the hardware interrupt level signal and stops outputting drive signals to the environmental device drive circuit.