A method, apparatus, and device for real-time embedding of semantic metadata
By using FPGA analysis and edge multimodal large model analysis, combined with audio and video adaptive frame extraction and main link bypass separation architecture, the problems of real-time semantic information synchronization and latency conflict in existing technologies are solved. Low-cost, zero-latency semantic metadata generation and synchronization are achieved, supporting plug-and-play and reducing hardware modification costs.
Patent Information
- Application Number
- CN202611610099.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies cannot output dynamic semantic information in real time, resulting in high costs for manual monitoring and a high risk of missed judgments. Furthermore, AI semantic analysis suffers from latency conflicts and low resource utilization, making it impossible to achieve latency-free conversion and real-time semantic reasoning in live streaming scenarios. Private data packets cannot be used plug-and-play, hardware modification costs are high, and audio and video timing misalignment leads to a decrease in recognition accuracy.
By analyzing audio and video parameters using FPGA, combining macroblock differences and audio energy to determine adaptive frame extraction, edge multimodal large model analysis, adopting a main link and AI bypass separation architecture, dynamic latency compensation, and standard encapsulation of metadata, audio and video source binding and real-time embedding of semantic data are achieved.
It enables low-latency real-time generation of semantic metadata in live streaming scenarios, reducing the cost of manual monitoring, avoiding semantic tag gaps and latency conflicts, supporting plug-and-play functionality, reducing hardware modification costs, and ensuring audio-visual synchronization and recognition accuracy.
Smart Images

Figure CN123123604A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, and device for real-time embedding of semantic metadata. Background Technology
[0002] Traditional interface conversion equipment only transmits raw audio and video pixel data, failing to simultaneously output dynamic semantic information such as characters, scene types, speaker emotions, presence of inappropriate content, and content summaries. Live broadcast directors and online reviewers must rely on manual monitoring to assess content, which is costly, carries a high risk of missed judgments, and lacks machine-assisted real-time content prompts. Standard auxiliary data channel resources are underutilized. The SDI vertical blanking VANC area and HDMI manufacturer-specific InfoFrame information frames are reserved for standard auxiliary data transmission, but existing conversion equipment only uses them to transmit fixed static parameters such as HDR and timecode, failing to utilize these standardized channels to carry AI-generated dynamic structured semantic tags in real time. This results in idle channel resources and a lack of standardized real-time semantic embedding implementation solutions.
[0003] Content tag generation is delayed, resulting in low efficiency in post-production material retrieval and archiving. Existing semantic recognition methods all rely on offline, pre-production analysis, requiring AI tagging to be initiated only after video recording is complete. The massive amounts of material generated from live streams and outdoor shooting lack real-time synchronized content tags, necessitating manual keyword annotation by staff during media asset archiving and segment retrieval. This process is time-consuming and inefficient, leading to extremely low material reuse rates. Real-time semantic analysis conflicts with the main video feed. Embedding AI inference directly into the main video thread increases video transmission latency due to the time required for multimodal, large-model inference, compromising the low real-time requirements of live streaming. Existing solutions lack a decoupled architecture between the main transmission link and AI analysis bypass, preventing simultaneous real-time semantic inference while maintaining video transitions. When multiple live streams and high-density screen switching cause a surge in AI computing power, existing offline inference solutions lack tiered computing power control and semantic interpolation compensation mechanisms, leading to issues such as keyframe loss, semantic tag gaps, and missed identification of risky content. Furthermore, the absence of a lightweight pre-risk screening mechanism compresses compliance detection capabilities during high computing power usage, failing to prioritize the identification of illegal content.
[0004] Furthermore, existing AI semantic analysis outputs are mostly proprietary custom data packets, not encapsulated according to the SMPTE ST 291 and HDMIVendor Specific InfoFrame universal protocols. Downstream recorders, broadcast consoles, and media asset systems require the separate development of proprietary parsing plugins, resulting in high costs for hardware modification and software adaptation, and making it impossible to read semantic tags in a plug-and-play manner. Existing frame extraction and audio segmentation mechanisms lack hardware-based 1ms precision unified PTS clock anchoring logic, leading to timing misalignment between keyframe images and corresponding audio segments. This causes a decrease in the accuracy of multimodal inference results for face recognition, speech recognition, and emotion analysis, and makes it impossible to accurately bind the speech context corresponding to a single frame. Summary of the Invention
[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A real-time embedding method for semantic metadata includes: acquiring SDI / HDMI video streams and PCM audio; parsing basic audio and video parameters using FPGA; determining scene switching by combining 16×16 macroblock MAD, HSV Bass distance, and 2-frame verification; identifying sound abrupt changes based on audio energy indicators after 300ms noise floor calibration; achieving adaptive frame extraction based on content importance scoring; extracting 500ms of synchronized audio and adding a 1ms precision hardware PTS timestamp to complete source binding; preprocessing the extracted frames and sending them into an edge multimodal large model to complete face recognition, scene classification, ASR, emotion and violation risk detection; outputting structured JSON semantic data; compressing the data using gzip / CBOR, with a single frame payload ≤512Byte; exceeding this limit and truncating according to priority; segmenting excessively long data with fragment identifiers; pre-screening risk levels using a risk sentinel model with ≤5M parameters; and using SMPTE ST 291 and HDMI Vendor... The SpecificInfoFrame standard encapsulates metadata and adopts a main link and AI bypass separation architecture. It dynamically compensates for latency by combining baseline latency, inference time, and queue depth calculations, and relies on frame number + PTS dual matching to bind semantics. The task queue has a maximum limit of 64, and only low-importance ordinary tasks are discarded when the queue is overloaded. If the target frame expires, the tag can be reissued within 3 frames. A four-level smooth degradation mechanism is set up, caching 8 historical semantics for exponential smooth interpolation, and interpolation is prohibited for risk fields. Risk tasks are promoted to the highest priority P0 to preempt computing power. High-risk scenarios are first written with temporary risk tags, and the tags are updated or revoked after the main model reviews them. Metadata is written to the VANC area of the 12th row of the SDI hidden area and the HDMI Vendor Specific InfoFrame area without destroying the original video bitstream. The device panel displays the input format, recognized scene, person, risk level, and metadata injection status in real time. Downstream devices that support SMPTE ST 291 and HDMI extended frames can directly parse it to form video interface conversion information that embeds multimodal semantic metadata in real time.
[0006] A real-time embedding apparatus for semantic metadata, the apparatus being used to execute executable instructions to perform the real-time embedding method for the aforementioned semantic metadata.
[0007] A real-time embedding system for semantic metadata, wherein the system is used to execute executable instructions to perform the aforementioned real-time embedding method for semantic metadata.
[0008] Its beneficial effects are as follows: For professional video conversion scenarios such as SDI and HDMI, a collaborative architecture of FPGA and edge multimodal large model is built. Adaptive frame extraction is first achieved through dual judgment of macroblock color difference and audio energy, and audio-video source alignment is achieved with a 500ms center window and 1ms hardware PTS. A risk sentinel with 5 million parameters is used for rapid screening, and the main model outputs full-dimensional structured semantics in parallel. A decoupled design of the main video link and AI bypass is adopted, and semantically accurate frame binding is achieved through dynamic latency calculation using multi-factor weighting. This is combined with 8-frame buffer exponential interpolation and four-level smooth degradation to ensure semantic coherence. Risk tasks are discarded in a graded manner according to importance in the task queue, and metadata is written to the blanking auxiliary area in accordance with SMPTE ST 291 and HDMI standards, with the device panel synchronously displaying the status. In case of overload, layered load reduction is implemented. High-risk images are first issued temporary risk labels, which are then corrected after review by the main model. The output stream can be parsed by standard downstream devices without modification, achieving real-time generation and embedding of standardized semantic metadata while transcoding without destroying the original video quality or increasing the main line latency.
[0009] This application aims to ensure low latency for live streaming using a decoupled architecture, ensuring that fluctuations in AI inference do not affect the real-time output of the main video, thus meeting the low-latency broadcasting requirements of broadcasting. It employs a layered frame extraction + dynamic latency algorithm to balance computational power consumption with key semantic capture, effectively avoiding issues such as missed detections at scene boundaries and semantic image misalignment. Millisecond-level risk warnings are achieved through sentinel pre-positioning + P0 high-priority scheduling, multi-level verification reduces false alarms, and model self-iterative optimization is supported.
[0010] Furthermore, a four-level smooth degradation and interpolation completion mechanism is employed to avoid semantic tag gaps during computing power congestion, ensuring that compliance risk detection capabilities remain consistently up-to-date. Based on universal standards for embedding metadata, no custom hardware plugins are required, and existing broadcast control and media asset equipment can directly read the data, significantly reducing deployment and adaptation costs. It only occupies auxiliary data channels, maintaining complete lossless original video pixel and bitrate quality, and the local panel simplifies on-site maintenance and troubleshooting. Attached Figure Description
[0011] Figure 1 A flowchart illustrating a real-time embedding method for semantic metadata provided in an embodiment of the present invention; Figure 2 A product-side system block diagram provided for an embodiment of the present invention; Figure 3A flowchart of a technical method provided in an embodiment of the present invention; Figure 4 A diagram illustrating an asynchronous analysis and time-aligned back-injection mechanism provided in this embodiment of the invention; Figure 5 This is a diagram illustrating an exception handling and degradation strategy mechanism provided in an embodiment of the present invention. Detailed Implementation
[0012] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention. In one embodiment, this application also proposes a real-time embedding method for semantic metadata.
[0013] In this application embodiment, a real-time embedding method for semantic metadata is provided, such as... Figure 1 As shown: The S101 acquires SDI / HDMI video streams and PCM audio, and the FPGA analyzes the basic audio and video parameters. It combines 16×16 macroblock MAD, HSV Bass distance plus 2-frame verification to determine scene switching. After 300ms noise floor calibration, it identifies sound changes based on audio energy indicators. It achieves adaptive frame extraction based on content importance scoring, extracts 500ms of synchronous audio and adds a 1ms precision hardware PTS timestamp to complete the same source binding.
[0014] In one implementation, SDI and HDMI input video streams and PCM audio data are acquired. The FPGA analyzes the resolution, frame rate, color format, and audio timing parameters. Scene switching is jointly determined using the 16×16 macroblock MAD mean and HSV histogram Bach distance, and a continuous two-frame verification mechanism is added to suppress false triggering of camera zoom and flash. The FPGA controls the SDI and HDMI interface chips to receive video streams of the corresponding standards, and simultaneously extracts PCM audio data from the video signals. After completing signal deserialization and protocol parsing, the video resolution, frame rate, color format, and audio timing synchronization information are output, providing a unified timing reference and parameter basis for subsequent keyframe extraction and audio-video homogeneous binding. For example, if the device receives a 3G-SDI input signal, the FPGA deserializes the signal through the interface chip, obtaining a video resolution of 1920×1080, a frame rate of 25 frames per second, and a color format of YUV422. Simultaneously, PCM audio data with a sampling rate of 48 kHz is extracted from the signal, completing the basic parameter identification and timing reference alignment of the input signal.
[0015] The FPGA performs lightweight difference analysis on two adjacent video frames, employing a dual-indicator judgment mechanism of brightness difference and color distribution difference to identify scene transitions, balancing hardware computational efficiency and scene judgment accuracy. First, brightness dimension difference calculation is performed. Each frame is divided into 16×16 uniform macroblocks. The absolute difference of the mean Y component of corresponding macroblocks in adjacent frames is calculated one by one. The average of the calculation results for all macroblocks is then taken as the quantified value of the brightness difference for the entire frame, reflecting the overall degree of change in brightness content between frames.
[0016] Color dimension difference calculation is performed synchronously. The two images are converted to the HSV color space, and the color histograms of each channel are statistically analyzed. The distribution difference between the two histograms is calculated using the Bach distance algorithm to obtain a quantized value of the color distribution difference, reflecting the overall degree of change in color composition between frames. The system sets judgment thresholds for both indicators: the default value for the brightness difference threshold is 12, corresponding to the quantization range of the Y component from 0 to 255; the default value for the color Bach distance threshold is 0.35. Only when both indicators simultaneously exceed their respective preset thresholds is a scene switch initially determined for the current frame, avoiding misjudgments caused by random fluctuations in a single indicator.
[0017] To suppress false triggers caused by slow camera zooms and momentary white flashes, the system introduces a two-frame confirmation mechanism. A single frame meeting both threshold criteria serves only as a preliminary judgment and does not directly trigger a frame-up operation. Only when two consecutive frames simultaneously meet both threshold criteria is a scene-switching frame-up operation formally triggered, and the corresponding frame is marked as a high-priority analysis task. Secondary temporal verification filters out single-frame noise and momentary image fluctuations. For example, during a studio shot slowly zooming in from a wide shot to a close-up of the presenter, the absolute difference of the macroblock mean in adjacent frames may fluctuate slightly, but not reaching the brightness difference threshold, and the color histogram Bach distance is also within the threshold; therefore, the system does not trigger a scene switch. When the director performs a hard cut, switching the image directly from the studio to an outdoor interview, if the absolute difference of the macroblock mean and the histogram Bach distance in two consecutive frames simultaneously exceed the corresponding thresholds, the system formally determines that a scene switch has occurred, immediately extracts the current frame, and marks it as a high-priority analysis task.
[0018] Using a 300ms silent noise floor as a benchmark, audio abrupt events are identified based on three quantitative indicators: RMS amplification exceeding 6dB, instantaneous peak amplitude -18dBFS, and short-term energy slope of 3dB / 20ms. The FPGA performs continuous feature extraction on the input PCM audio data using a fixed-duration sliding window. The window length is set to 20 milliseconds, sliding sequentially across the audio stream with a fixed step size to achieve uninterrupted monitoring throughout the entire time period. Within each window, two core acoustic features are calculated in parallel: root mean square energy (RMS energy), which is the root mean square value of the amplitudes of all sampling points within the window, reflecting the average loudness level of the audio during that time period; and peak amplitude, which is the maximum absolute amplitude of the sampling points within the window, reflecting the instantaneous highest volume during that time period. These two features together constitute a temporal description sequence of audio energy, providing a data foundation for subsequent abrupt event detection. For example, for a PCM audio stream with a sampling rate of 48 kHz, each 20-millisecond window contains a total of 960 audio sampling points. The FPGA completes the energy accumulation and peak retrieval of all sampling points within each window period, calculates the corresponding root mean square energy value and peak amplitude, and outputs continuous audio feature timing data as the window slides, ensuring the real-time and continuous nature of the detection.
[0019] The system introduces a long-cycle noise floor tracking mechanism, setting a 300-millisecond sliding window for noise floor analysis. It continuously tracks the lowest root-mean-square (RMS) energy value within this window, using it as the noise floor benchmark for the current environment. This benchmark is dynamically updated based on changes in the acoustic environment, replacing a fixed threshold scheme and adapting to different background noise levels. All audio abrupt changes are pre-checked; only when the current window's RMS energy increases by more than 8 dB relative to the noise floor benchmark does the system proceed to the subsequent significant change assessment process. This filters out small, random fluctuations in ambient background noise at their source, reducing the probability of false triggers. For example, during a break in program recording when the environment is quiet and in standby mode, the system continuously tracks the lowest energy point within the scrolling 300-millisecond window and sets this value as the current environment's noise floor benchmark. If only weak equipment noise exists, and the RMS energy fluctuates slightly but increases by less than 8 dB relative to the noise floor, the system classifies it as normal ambient noise and does not trigger keyframe extraction.
[0020] The system employs three independent judgment indicators, covering different types of audio abrupt changes across three dimensions: energy jump, instantaneous intensity, and rate of change. Meeting any one of these indicators is sufficient to classify a significant audio change event. The first indicator is the energy increase judgment for adjacent windows. It calculates the root mean square energy difference between two adjacent sliding windows. When the energy increase exceeds 6 dB, it is considered a step energy jump, corresponding to scenarios such as sudden speech initiation or sudden loud noises. The second indicator is the instantaneous peak amplitude judgment. When the peak amplitude within a single sliding window exceeds -18 dB of full scale, it is considered a sudden strong sound signal, corresponding to sudden loud noises such as applause or impacts. The third indicator is the energy envelope slope judgment. It calculates the rate of increase of the energy envelope formed by multiple consecutive windows. When the energy growth slope corresponding to each 20-millisecond window exceeds 3 dB, it is considered a rapid and sustained energy increase, corresponding to scenarios such as gradually increasing volume or a gradual rise in the pitch of a human voice. These three indicators complement each other, ensuring that different forms of audio changes are accurately captured.
[0021] For example, if a host suddenly starts speaking from a silence, and at the instant the speech begins, the root mean square energy increase between two adjacent 20-millisecond windows reaches 7 dB, while the current energy increase relative to the ambient noise exceeds 8 dB, simultaneously meeting both the noise floor verification condition and the energy increase judgment standard, the system immediately determines this as a significant audio change event. It then simultaneously extracts the corresponding video keyframe and sends it to the multimodal analysis module, ensuring that the video and audio data at the start of the speech are collected and analyzed synchronously.
[0022] The system dynamically adjusts the sampling interval by integrating three strategies: fixed-period frame sampling, event-triggered frame sampling, and adaptive frame sampling based on multi-factor content importance scoring. When importance suddenly increases, 1-2 frames are back-sampled from the DDR circular cache to avoid missed detections. The system integrates these three strategies to extract keyframes, constructing a hierarchical sampling system with a stable baseline, abrupt change response, and dynamic optimization. Fixed-period frame sampling serves as the basic sampling layer, setting a default sampling interval to ensure a minimum semantic update frequency, guaranteeing continuous semantic data output even in long-term static scenarios. Event-triggered frame sampling, as the abrupt change response layer, is directly triggered by scene switching detection and significant audio change detection results, accurately capturing semantic abrupt change nodes. Adaptive frame sampling, as the dynamic optimization layer, continuously adjusts the sampling density based on the real-time semantic value of the content, tilting computing resources towards high-value scenarios. The three strategies operate in parallel, with event triggering and adaptive adjustment both taking effect on top of the fixed-period strategy, jointly covering three types of content change scenarios: stable, gradual, and abrupt.
[0023] Content importance scoring is the core calculation basis for adaptive frame extraction. It employs a multi-factor linear weighted model to quantify the semantic value of a single frame, outputting a normalized importance score. The scoring model incorporates five core influencing factors, each corresponding to the intensity of semantic change in different dimensions. The first factor is scene change intensity, derived from the joint calculation results of inter-frame brightness and color differences, reflecting the overall scene switching and change level. The second factor is target motion amplitude, derived from inter-frame motion vector statistics, reflecting the intensity of motion of objects and the camera within the frame. The third factor is face change degree, derived from the number of face detection results and pose change statistics, reflecting the appearance and change of the main subject. The fourth factor is audio saliency, derived from the calculation results of audio energy mutation amplitude, reflecting the intensity of semantic change in the sound dimension. The fifth factor is risk prior indication, derived from the risk level output by the risk sentinel model, reflecting the compliance sensitivity of the content.
[0024] Each of the five factors is assigned a corresponding weight, with default weights of 0.25, 0.20, 0.20, 0.15, and 0.20 respectively. After normalization of all factors, a weighted sum is performed to obtain a content importance score ranging from zero to one. A higher score indicates higher semantic density and greater analytical value at that moment, corresponding to a higher sampling frequency. For example, in a static landscape scene, the intensity of scene change is extremely low, there are no moving targets or faces, the audio energy is stable and there are no risk warnings, and the values of all five factors are low, resulting in a weighted importance score below 0.35, placing it in the low-value range. However, when the scene transitions to an interview scene accompanied by audio content, the intensity of scene change jumps, the face factor significantly increases, and the audio salience simultaneously strengthens, causing the weighted importance score to rise above 0.8, entering the high-value range.
[0025] The system sets two levels of scoring thresholds (high and low) and dynamically switches between three sampling densities based on a basic frame sampling interval. The high threshold defaults to 0.75, the low threshold to 0.35, and the basic frame sampling interval to 25 frames. When the content importance score is greater than or equal to the high threshold, the system enters dense sampling mode, adjusting the frame sampling interval to 1 to 3 frames, significantly increasing the sampling density and fully capturing the semantic details of high-value scenes. When the score is between the high and low thresholds, the system maintains the basic frame sampling interval to balance sampling density and computational power consumption. When the score is below the low threshold, the system enters sparse sampling mode, extending the frame sampling interval to 2 to 4 times the basic interval, reducing the computational power consumption of low-value static scenes. The sampling interval is continuously and smoothly adjusted according to the score, avoiding frequent jumps in levels.
[0026] For example, in static landscape shots, where the content importance score is low, the system performs sparse sampling at intervals of one frame every 25 frames, reducing the AI inference load while ensuring semantic continuity. When the scene suddenly cuts to a scene of human interaction accompanied by a sudden change in voice content, the content importance score jumps to a high threshold range. The system immediately switches to a dense sampling mode, extracting one frame every two frames, significantly increasing the sampling frequency and fully capturing high-value semantic information such as human expressions and voice content.
[0027] To avoid missing sampling at important semantic starting points, the system employs a DDR circular buffer for boundary backtracking and supplementary sampling. The DDR circular buffer continuously caches the raw video frame data of the most recent few frames, with a buffer depth covering the number of frames corresponding to the maximum frame extraction interval, providing data support for backtracking operations. When a content importance score is detected to jump from a low to a high range, and the time elapsed since the last frame extraction exceeds the newly set frame extraction interval, a semantic boundary sampling gap is identified. In this case, in addition to extracting the current frame, the system backtracks and extracts 1 to 2 of the most recent frames from the DDR circular buffer as supplementary sampling frames, sending them to the analysis queue to fill the sampling gap at the important semantic starting stage, ensuring complete coverage of semantic changes.
[0028] For example, in a static landscape scene, the system samples sparsely at 25-frame intervals. If 12 frames have passed since the last frame boost, and the scene suddenly transitions to a character interaction scene, causing the importance score to jump to the high threshold range, the new frame sampling interval is adjusted to 2 frames. The current frame, which is 12 frames from the last boost, far exceeds the new interval requirement. In addition to extracting the current frame, the system simultaneously retrieves the most recent frame from the circular buffer as a supplementary frame, covering the semantic boundary fragments at the moment of scene transition and avoiding missing crucial initial shots of characters appearing.
[0029] A 500ms audio window is extracted centered on the hardware capture time of the frame. The start and end indices of the audio samples are calculated using the sampling rate, and a unified 1ms precision hardware PTS timestamp is used. The audio and video are then packaged and bound together, including the frame_id, audio and video start and end timestamps, sampling rate, and bit depth fields to achieve audio-video homogeneity. The system uses the hardware capture time of each key video frame as the time anchor point and employs a centrally symmetrical approach to extract audio segments, extending forward and backward by 250 milliseconds to form an audio window of 500 milliseconds. This window length is adapted to the input requirements of multimodal model speech analysis and emotion recognition, while placing the video frame time at the midpoint of the audio segment's time, ensuring the highest accuracy in the temporal correspondence between the video and audio. The audio window covers different video periods at different video frame rates. At 25 frames per second, the 500ms window covers approximately 12.5 video periods; at 50 frames per second, it covers approximately 25 video periods, fully carrying the speech context information before and after the keyframe. For example, using the hardware capture time of a key frame in a 25 frames per second format as the alignment reference, the system uses that time as the midpoint to extract 250 milliseconds of historical audio before and 250 milliseconds of subsequent audio after, forming a complete audio segment with a total length of 500 milliseconds. This fully covers the speech context corresponding to that frame, providing complete data for subsequent speech recognition and audio-video joint analysis.
[0030] For the captured audio window, the system converts the time-dimensional window boundaries into start and end indices in the sampling point dimension based on the sampling rate of the current input audio. During the calculation, the starting index uses a floor function, and the ending index uses an up function, ensuring that the sampling point sequence completely covers the entire 500-millisecond time window and avoiding audio information loss caused by boundary sample truncation. The index calculation is entirely based on the product of the absolute time of the hardware capture moment and the sampling rate, ensuring that each audio sample corresponds to a precise time position. For example, considering an audio sampling rate of 48 kHz, 250 milliseconds corresponds to 12,000 sampling points. Therefore, taking the sampling point corresponding to the keyframe moment as the center, the system takes the 12,000th sampling point forward as the start of the window and the 12,000th sampling point backward as the end of the window. The entire audio segment contains 24,000 sampling points, completely corresponding to the 500-millisecond duration. The calculation process uses floor functions to ensure complete coverage of boundary samples.
[0031] The system assigns a monotonic clock stamp based on the same hardware clock source to each video frame and its corresponding audio segment, with a timestamp accuracy of 1 millisecond. Each video frame corresponds to a single capture time timestamp, while each audio segment corresponds to two timestamps: a start time and an end time. All timestamps are anchored to the hardware capture time of the keyframe, ensuring that their time bases are completely identical. Using a hardware monotonic clock avoids software-level time disturbances such as system time callbacks and time zone changes, guaranteeing the reliability of audio-video synchronization at the clock source level. For example, the system assigns a 1-millisecond precision hardware timestamp to the corresponding capture time of the video frame, while simultaneously assigning a start timestamp and an end timestamp to the corresponding 500-millisecond audio segment. All three types of timestamps originate from the same hardware clock domain, and the video timestamp is precisely located at the midpoint between the two audio timestamps, achieving strict time alignment.
[0032] After time alignment, the system packages video frame data, audio segment data, and accompanying metadata into a standard analysis data packet. This packet carries fields including frame identifier, video timestamp, audio start and end timestamps, sampling rate, and bit depth. Before being sent to the AI analysis module, the data packet undergoes a source-matching check to confirm that the video timestamp is at the point in the audio time window, all timestamps are continuous and monotonic, and parameters match. Only after passing the check is the packet sent to the inference queue. This entire encapsulation mechanism ensures that the image and audio data received by the multimodal model are strictly aligned to the same source, eliminating analysis errors caused by timing misalignment. For example, the final encapsulated standard data packet contains complete metadata such as frame number, video timestamp, audio start and end timestamps, 48 kHz sampling rate, and 16-bit depth, as well as the corresponding original image and audio data. After passing the source-matching check, the data packet is sent to the subsequent multimodal analysis module, where the model can perform joint inference tasks such as face recognition, speech recognition, and emotion analysis based on the strictly aligned audio and video data.
[0033] S102 extracts frames, preprocesses them, and sends them into the edge multimodal large model to complete face recognition, scene classification, ASR, emotion and violation risk detection, and outputs structured JSON semantic data. After gzip / CBOR compression, the single frame payload is ≤512 bytes. If the limit is exceeded, it is truncated according to priority. Long data is segmented and carries fragmentation identifiers. A risk sentinel model with ≤5M parameters is used to pre-screen the risk level.
[0034] In one implementation, during image preprocessing, the system performs uniform scaling and color conversion operations on key image frames obtained through adaptive frame extraction, providing consistent input data for the multimodal model. At the algorithm level, a proportional scaling strategy is employed, maintaining the original aspect ratio throughout the process. Areas smaller than the target resolution after scaling are filled with zero pixels to avoid image stretching and distortion that could reduce recognition accuracy. The system offers two input resolution options: a default standard resolution of 1280x720, which automatically downgrades to a lower resolution of 640x360 when insufficient hardware computing power or inference queue backlog is detected, balancing inference speed and recognition performance by reducing the amount of input data. The color conversion stage supports both RGB888 and YUV420 output formats to match the input requirements of different models. For example, the system extracts keyframes from 25 frames per second of 1080P high-definition video input via 3G-SDI in the studio and scales it proportionally to a standard size of 1280x720. The original image aspect ratio matches the target resolution, eliminating the need for additional zero-pixel padding. The color format is converted to the model-compatible RGB888 format, completing image preprocessing. When the task queue occupancy rate exceeds 80% during peak live streaming periods, the system automatically reduces the input resolution to 640x360, decreasing the inference time per frame and alleviating computational pressure.
[0035] In the audio preprocessing stage, the system performs multi-channel downmixing and sampling rate standardization on the synchronously captured 500ms PCM audio. At the algorithm level, the input multi-channel audio is downmixed using an energy-weighted method, resulting in a unified output of either mono or stereo audio, eliminating the impact of channel number differences on the model input. Two sampling rate standards are set: a standard mode of 48kHz to ensure the accuracy of speech recognition and emotion analysis; and an automatic switch to a 16kHz low-sampling mode when computational power is limited, reducing the amount of audio data processing. All output audio uses a uniform 16-bit depth to ensure data format consistency. For example, if the input signal carries 48kHz stereo audio, the system downmixes the left and right channels into mono audio with equal weights, maintaining a 16-bit bit depth, and extracts the corresponding 500ms segment for audio preprocessing. When NPU utilization consistently exceeds 90%, the system automatically switches to the 16kHz low-sampling mode to reduce the computational overhead of audio inference. The preprocessed image and audio data are simultaneously fed into the edge multimodal large model as input material for multimodal parsing.
[0036] The edge multimodal large-scale model employs a multi-task parallel inference architecture, simultaneously executing five types of semantic parsing tasks and compliance screening tasks to achieve full-dimensional semantic extraction of audio and video content. The five parsing tasks are face recognition, scene classification, object detection, speech recognition, and sentiment analysis, with the ability to screen for illegal content simultaneously overlaid. The model input consists of preprocessed image frames and corresponding audio segments, and the output is complete structured semantic information. Fields include frame number, audio / video timestamp, person name and confidence level, scene type and confidence level, object list, speech recognition text, sentiment category and confidence level, risk level, risk label, content summary, importance score, model version, and inference time. All output fields are integrated into a standardized UTF-8 text format, providing data support for subsequent metadata encapsulation and scheduling strategy adjustments.
[0037] The algorithm is deeply integrated with business logic. The output importance score is directly fed back to the frame extraction scheduling module for dynamically adjusting the frame extraction interval. The risk level field is synchronously input into the degradation scheduling module to determine task priority and computing power allocation. The remaining semantic fields serve as the core content of metadata and are ultimately embedded into the video stream auxiliary channel. For example, after a news live broadcast and its corresponding audio are fed into the model, the inference outputs that the person is host A with a confidence level of 0.93, the scene is a news studio with a confidence level of 0.91, the emotion is serious with a confidence level of 0.88, the risk level is normal, and the audio text is the broadcast content. At the same time, it outputs supporting fields such as frame number and inference time. All fields are integrated into structured text data and synchronously transmitted to the metadata encapsulation module and the scheduling module.
[0038] The system performs compression processing on the generated complete structured semantic text, adapting to the bandwidth limitations of the video auxiliary data channel, and supports two compression algorithms. The first is a general lossless compression algorithm, suitable for text compression in general scenarios; the second is a binary structured compression algorithm, specifically optimized for structured data, with a higher compression ratio, suitable for scenarios with sufficient computing power and strict requirements on payload length.
[0039] The system sets the maximum length of the compressed semantic payload per frame to no more than 512 bytes, matching the single-frame data carrying capacity of the blanking region and the information frame. When the compressed data exceeds the length limit, lower-priority fields are truncated sequentially according to the priority order of people, scenes, risks, and voice, prioritizing the transmission of core semantic information. If the length limit is still exceeded after truncating the core fields, the complete semantic data is split into multiple independent data packets, each carrying a fragment number and a total fragment count identifier, and distributed across auxiliary channels for multiple consecutive frames for transmission. Downstream devices can reassemble the complete semantic data based on the fragment identifiers, avoiding information loss. For example, in a live broadcast of a large event, multiple people, multiple types of objects, and multiple voice texts are identified. The length of the compressed complete structured text exceeds 512 bytes. The system first removes secondary fields such as fine-grained object lists. If the remaining content still exceeds the length limit, the complete semantic data is then split into three data packets, each carrying a corresponding fragment number and a total fragment count, and transmitted sequentially through auxiliary channels for three consecutive frames. Downstream devices receive all fragments and then reassemble them to restore the complete semantic information.
[0040] Before the main multimodal model performs formal inference, the system sets up a lightweight risk sentinel model to construct a two-stage risk identification architecture. The sentinel model has fewer than 5 million parameters, making it an ultra-lightweight detection model. It performs rapid initial screening for compliance risks such as illegal content, sensitive identifiers, and abnormal behavior, with inference latency significantly lower than the main model. The model input consists of keyframe thumbnails and corresponding audio clips, and outputs three levels of risk warnings: no risk, suspected risk, and high risk.
[0041] The algorithm and business scheduling logic are deeply integrated. Risk alert results are directly used as the basis for task priority determination and transmitted to the task scheduling unit. When the detection result is not risk-free, the analysis task of the corresponding frame is immediately promoted to the highest priority queue, preempting NPU computing resources. If necessary, ordinary tasks can be paused to ensure that the end-to-end processing latency of risk tasks does not exceed 150 milliseconds. This mechanism achieves priority response to risky content without increasing excessive computing power, meeting the rigid compliance requirements of broadcast control scenarios. For example, if a violation mark suddenly appears in the live broadcast, the sentinel model quickly determines it to be a high-risk event after receiving the thumbnail and audio, and immediately pushes a high-risk alert to the scheduling unit. The scheduling unit directly promotes the inference task of that frame to the highest priority, pauses the currently processing ordinary scene analysis task, and prioritizes the scheduling of NPU resources to complete the full risk review of that frame, ensuring that the processing latency of risk tasks is controlled within 150 milliseconds.
[0042] S103 encapsulates metadata according to SMPTE ST 291 and HDMI Vendor Specific InfoFrame standards. It adopts a main link and AI bypass separation architecture, and dynamically compensates for latency by combining baseline latency, inference time, and queue depth calculation. It relies on frame number + PTS dual matching to bind semantics. The task queue has a maximum limit of 64. If the queue is overloaded, only low-importance ordinary tasks are discarded. If the target frame expires, the tag can be reissued within 3 frames.
[0043] In one implementation, semantic metadata is encapsulated according to the SMPTE ST 291 and HDMI Vendor Specific InfoFrame protocols. A decoupled architecture is adopted, bypassing the main video transmission link and AI asynchronous analysis. Dynamically compensated latency is calculated using the formula: D_act = D_fix + ceil((T_infer - T_target) / T_frame) + Q_depth, where D_fix is the baseline fixed latency frame count, T_frame is the duration of a single video frame, T_infer is the current task inference time, T_target is the target inference budget, and Q_depth is the queue depth compensation term. The target inference budget is set to 120ms, and the dynamic latency cap is 30 frames. 25 / 50 / 60fps correspond to 15 / 8 / 10 frames of baseline fixed latency, respectively. The system performs standardized encapsulation processing according to the output interface type, converting the structured semantic data output by the AI module into data packets conforming to industry-standard protocols, ensuring that downstream compatible devices can directly recognize and read the data without additional protocol adaptation development.
[0044] For SDI output scenarios, the system assembles ANC and VANC data packets according to the SMPTE ST 291 standard. A complete data packet contains five protocol fields: data identifier, secondary data identifier, data count, user data word, and checksum. The data identifier and secondary data identifier jointly mark the service type and data attribution of the data packet, used by downstream devices to distinguish different types of auxiliary data; the data count field records the byte length of the payload; the user data word area carries compressed and encoded structured semantic content; and the checksum field is used for data integrity verification to prevent semantic data anomalies caused by bit errors during transmission. The encapsulation process strictly follows the byte order, word length, and alignment rules specified in the standard, ensuring that all professional video devices supporting this standard can correctly parse and extract the semantic content.
[0045] For HDMI output scenarios, the system performs encapsulation according to the manufacturer's proprietary information frame standard. Data packets are sequentially written with four categories of content: header, IEEE unique identifier, payload, and checksum. The header identifies the type and length of the information frame; the IEEE unique identifier marks the manufacturer affiliation of metadata, distinguishing extended information from different manufacturers; the payload area carries compressed semantic data; and the checksum is used to verify the integrity of the entire frame. The encapsulation format fully complies with the HDMI protocol specification and can be directly read and parsed by downstream display, recording, and directing equipment that supports manufacturer-specific extended information frames.
[0046] For example, when the device operates in SDI-to-HDMI mode, the FPGA simultaneously receives the same structured semantic data output from the AI module and encapsulates it in parallel according to two different protocol standards. On the SDI side, the agreed-upon data identifier, secondary data identifier, data count value, semantic payload, and checksum are sequentially filled according to the SMPTE ST 291 standard to generate a complete VANC data packet. On the HDMI side, a manufacturer-specific information frame is generated simultaneously, and the standard packet header, the corresponding IEEE organization unique identifier, the semantic payload, and the checksum are written into it. Both types of data packets are filled with complete protocol fields and the checksum is calculated. After output, they can be directly parsed and read by professional broadcast control and recording equipment with the corresponding interface to extract the embedded semantic metadata.
[0047] The system adopts an architecture design that completely decouples the main video transmission link from the AI asynchronous analysis bypass, ensuring that the low-latency real-time conversion of the main video stream is unaffected by fluctuations in AI inference time, while achieving precise time alignment between semantic data and video frames. The main link is responsible for the real-time conversion and output of the video interface throughout, with stable transmission latency that does not change with AI load, ensuring the low-latency requirements of live streaming scenarios. Keyframes and corresponding audio segments are written to the task cache queue asynchronously. The AI module performs semantic inference in parallel through multiple computing units. After inference, it returns semantic results carrying the original frame number and hardware timestamp. Finally, the FPGA, based on the dynamic latency calculation results, aligns the semantic data and writes it into the auxiliary data area of the corresponding target output frame.
[0048] The actual injection latency frame count is quantitatively calculated using a standardized formula: D_act = D_fix + ceil((T_infer - T_target) / T_frame) + Q_depth, where D_act is the final output total actual injection latency frame count, D_fix is the baseline fixed latency frame count, T_infer is the actual inference time for the current task, T_target is the preset target inference budget, T_frame is the duration of a single video frame at the current frame rate, Q_depth is the queue depth compensation frame count, and ceil is the rounding rule. The entire algorithm uses a three-level latency term stacking to simultaneously cover three types of latency sources: baseline inference time, single-frame inference fluctuations, and queue backlog, achieving dynamic adaptation of semantic injection timing.
[0049] The first item is the baseline fixed latency D_fix, which is a preset base latency value for different video frame rates, used to cover the time consumed by AI inference, data transmission, and encapsulation under normal load. Different frame rates correspond to different baseline values: 15 frames per second (fps) for 25 fps, corresponding to approximately 600 milliseconds; 8 frames per second (fps) for 50 fps, corresponding to approximately 160 milliseconds; and 10 frames per second (fps) for 60 fps, corresponding to approximately 167 milliseconds. The baseline value can be automatically calibrated based on long-term measured average AI inference latency to adapt to hardware platforms with different computing power configurations.
[0050] The second term is the inference time deviation compensation term, corresponding to the ceil((T_infer-T_target) / T_frame) part in the formula, used to offset the alignment deviation caused by fluctuations in single-frame inference time. The calculation first calculates the difference between the actual inference time of the current task and the target inference budget, then divides the difference by the duration of a single frame, and finally rounds the result up to obtain the number of frames to be compensated. The target inference budget is set to 120 milliseconds by default, serving as a baseline reference for normal inference time. When the actual inference time is higher than the target budget, positive compensation frames are added; when it is lower than the budget, no negative compensation is performed, maintaining the baseline latency lower limit to avoid premature writing of semantic data. The single-frame duration is calculated from the current video frame rate; the higher the frame rate, the shorter the single-frame duration, and the more compensation frames are required for the same time deviation.
[0051] The third item is the queue depth compensation item Q_depth, which is used to offset the additional waiting delay caused by task queue backlog. Based on the current depth of the task cache queue, a corresponding number of compensation frames are added. The deeper the queue is occupied, the greater the compensation amount, reserving sufficient queue waiting time to avoid misalignment caused by the target frame being output when the semantic result is returned.
[0052] The system sets a dynamic latency cap of 30 frames. The actual injection latency smoothly increases between the baseline fixed latency and the maximum latency cap, avoiding semantic label jumps caused by sudden latency changes and ensuring the temporal continuity of the output semantics. For example, in a live signal processing scenario with 25 frames per second, the system's default baseline fixed latency D_fix is 15 frames, corresponding to a duration of 600 milliseconds. During a certain period, the live content enters a high-value, intensive scene, increasing the AI task load. The single-frame inference time T_infer reaches 180 milliseconds, while the preset target inference budget T_target is 120 milliseconds, resulting in a difference of 60 milliseconds. At 25 frames per second, the single-frame duration T_frame is 40 milliseconds. Dividing 60 milliseconds by 40 milliseconds yields 1.5, which, after rounding up, represents a 3-frame inference time deviation compensation. Adding the current queue depth compensation Q_depth, which is 2 frames, the total actual injection latency is calculated to be 15 + 3 + 2, totaling 20 frames. This value does not exceed the maximum latency limit of 30 frames, which ensures that the semantic results are stably aligned with the video frame and that there will be no premature or late writing.
[0053] By binding semantics through a dual condition of output frame number and hardware PTS timestamp, the matching verification error is controlled within 1ms. Each AI inference task carries the original source frame number and source hardware timestamp. The system calculates the target output frame number based on the actual injection delay. The FPGA maintains a mapping table between the output frame counter and the hardware timestamp. When the output frame number is equal to the target frame number, or the difference between the output timestamp and the target timestamp is no more than 1 millisecond, either condition is met, and the frame is determined to be an aligned frame. The corresponding semantic metadata is then written to the auxiliary data area of that frame.
[0054] For example, a keyframe source number is frame 1200, and the source timestamp is the corresponding hardware clock value. After dynamic delay calculation, the target output frame number is frame 1220. When the output frame counter reaches frame 1220, the system verifies that the corresponding hardware timestamp difference is 0.3 milliseconds, which meets the accuracy requirement of less than 1 millisecond. Then, the semantic metadata of the frame is written into the SDI blanking area and the HDMI information frame area.
[0055] The task cache queue employs a hybrid FIFO + priority scheduling mechanism, with a basic limit of 64 and a high-performance mode that expands to 128. When the queue occupancy exceeds 80%, only low-importance ordinary tasks are discarded, while risk-related tasks are not discarded. The system configures a task cache queue for asynchronous analysis links, adopting a hybrid scheduling mechanism that combines first-in-first-out and priority scheduling. It also sets tiered capacity and overload discard strategies to prioritize the execution of high-value semantic analysis tasks when computing power load fluctuates.
[0056] The basic queue scheduling logic follows a first-in, first-out (FIFO) rule. Tasks of the same priority are allocated computing resources strictly according to their enqueue order, avoiding starvation caused by low-priority tasks waiting indefinitely. On top of this, a priority-based hierarchical scheduling capability is added, allowing high-priority tasks to skip earlier ordinary tasks in the queue and enter the inference unit first, balancing scheduling fairness with business priority requirements. Two capacity configuration levels are available: in basic operation mode, the queue can hold a maximum of 64 tasks, adapting to the load capacity of conventional computing hardware; in high-performance operation mode, the queue capacity is expanded to a maximum of 128, capable of handling higher-concurrency analysis tasks and adapting to multi-input or high-density frame-skipping scenarios.
[0057] When the queue occupancy rate exceeds 80%, the system automatically triggers an overload discarding mechanism. This mechanism releases queue space by selectively discarding low-value tasks, maintaining the stable operation of the analysis pipeline. The discarding process follows an algorithmic logic of hierarchical filtering, value-based sorting, and sequential elimination, ensuring that high-priority tasks are unaffected throughout the process. The first step involves hierarchical filtering by task type. The system divides all tasks in the queue into two categories: ordinary tasks and protected tasks. All tasks with non-ordinary risk levels are classified as protected tasks and are not included in the discarding candidate pool, ensuring that compliance risk-related analysis tasks are always retained. Other routine content analysis tasks without risk markers are placed in the ordinary task pool as discardable candidates.
[0058] The second step is to prioritize tasks based on their importance. All tasks in the general task pool are sorted in ascending order according to their content importance score, with tasks having lower scores ranked higher and prioritized for discarding. The content importance score is derived from a content importance assessment model, calculated by considering factors such as scene changes, character proportions, and audio salience. Lower scores indicate lower semantic value. The third step is to perform successive elimination. The system discards tasks sequentially, starting with the top-ranked task. After each task is discarded, the current queue occupancy rate is calculated in real time until it falls below 80%, at which point discarding stops. This aims to retain as many effective analysis tasks as possible and avoid semantic information gaps caused by excessive discarding. This scheduling algorithm is deeply integrated with the business logic, and queue overload status is synchronously fed back to the front-end frame extraction module. When the queue triggers discarding multiple times consecutively, the frame extraction module automatically reduces the frame extraction density for general scenes, decreasing the number of general tasks entering the queue and reducing queue pressure from the source. However, the rules for risk-triggered frame extraction and high-value content frame extraction remain unchanged, achieving a differentiated scheduling effect that tilts computing resources towards high-value and high-risk content.
[0059] In large-scale event live streaming scenarios, frequent scene transitions and intensive character interactions, coupled with numerous stage transitions and long-distance empty shots, continuously increase the workload of AI analysis tasks. In the basic mode, the number of tasks in the queue gradually increases, eventually reaching 85% occupancy, triggering an overload drop mechanism. The system first performs tiered filtering on all tasks in the queue. All tasks containing main characters or marked with potential risks are classified as protected tasks and excluded from the drop candidate range. The remaining ordinary tasks, such as static long-distance empty shots and transitions without characters, enter the candidate pool. The system sorts these tasks in ascending order according to their content importance score, with pure empty shots having the lowest importance score and ranking first in the drop sequence. The system then sequentially drops the lowest-scoring static empty shot tasks. After the drop-off, the queue occupancy rate drops back to 77%, returning to a safe operating range. Throughout the drop-off process, all high-priority tasks containing character interviews, scene transitions, and potential risks are retained. The overall queue operation remains within a stable load range, ensuring the integrity of core semantic analysis while avoiding task loss and network congestion caused by queue overflow.
[0060] If the target output frame has already been output, a 3-frame compensation window is opened for writing to the nearest frame. The frame exceeding the window is marked as expired, and a summary tag is resent to the next frame. When the AI queue is backlogged or inference times out, old keyframes can be discarded, the frame extraction frequency can be reduced, a lightweight model can be switched, and only risk detection can be retained, ensuring continuous output of the main video link throughout the process. The system is designed with a layered compensation and graded handling mechanism to address abnormal scenarios that may occur in the asynchronous inference link, such as expired target output frames and severe computing power overload. In normal expiration scenarios, a fixed compensation window is used to achieve writing to the nearest frame and resending the summary. In severe overload scenarios, a multi-level computing power contraction strategy is executed according to gradients, without interfering with the real-time conversion output of the main video link, ensuring the stability of system operation.
[0061] The target output frame expiration determination and compensation writing algorithm is as follows. After the semantic result is returned, the system first performs target frame status verification, determines the expiration level based on the frame number difference, and executes different writing strategies accordingly.
[0062] The first step is to determine the outflow of the target frame. After each AI inference task is completed, the returned result carries the original source frame number and the calculated target output frame number. The FPGA maintains an output frame counter in real time, continuously recording the sequence number of the video frames that have been output. The system compares the target output frame number with the current output frame counter value. When the output frame counter value is greater than the target output frame number, it is determined that the target output frame that was originally written has already flown out, and semantic metadata can no longer be written to that frame.
[0063] The second step involves writing the data within a 3-frame compensation window. After determining that the target frame has been output, the system calculates the absolute difference between the current output frame number and the target frame number. If the difference is less than or equal to 3 frames, it is determined that the data is within the compensation window. The system then directly writes the semantic metadata to the currently processed output frame, which is the next frame closest to the target frame. During the writing process, a hardware timestamp corresponding to the original target frame is simultaneously included. Downstream devices can use this timestamp information to calibrate the actual frame time corresponding to the semantic data, eliminating timing errors caused by write position offsets.
[0064] The third step is to perform super-window expiration resending. When the sequence number difference between the current output frame and the target frame exceeds 3 frames, the task is determined to have exceeded the compensation window and is marked as an expired task. The system no longer writes complete structured semantic data to the video stream. Instead, it extracts the core summary fields from the original semantic results, retaining only the scene category and risk level information, and writes the simplified summary label to the first available output frame. This mechanism avoids expired data consuming a large amount of auxiliary channel bandwidth while ensuring that core semantic information is not completely lost.
[0065] For example, in a sudden live broadcast scenario where the content changes rapidly, the AI computing power is instantly saturated, significantly increasing the time required for single-frame inference. A keyframe source numbered 1200 was originally scheduled to be written to frame 1220 after dynamic delay calculation. When the inference result returned, the FPGA output frame counter had already reached frame 1224, with a difference of 4 frames between the current frame and the target frame, exceeding the 3-frame compensation window range. The system marked this task as expired, extracted the studio scene and normal risk level summary information, and wrote them into the auxiliary data area of the subsequent frame 1225, ensuring that the core semantics could be obtained by downstream devices.
[0066] The multi-level handling strategy under severe computing power overload is as follows: when the AI task queue is severely backlogged, inference times out continuously, and the conventional discarding mechanism can no longer stabilize the queue load, the system executes the four-level handling strategy in order from light to heavy, releasing computing power resources step by step to ensure the continuous and stable output of the main video conversion link throughout the process.
[0067] The first level of handling involves cleaning up old tasks. The trigger condition is a queue occupancy rate consistently above 90% and two consecutive inference timeouts. The system selects and discards several ordinary keyframe tasks from the tail of the task queue that were enqueued earliest, retaining only the latest keyframe task corresponding to the most recent scene transition. This operation quickly releases queue storage space while ensuring that the semantic information of the latest scene can still be fully analyzed, preventing situations where scene semantics remain outdated for extended periods.
[0068] The second level of intervention involves reducing the frame extraction frequency. If the queue occupancy rate still does not return to a safe range after the first level of intervention, the system will adjust the process from the source of task generation, lowering the base interval parameter for adaptive frame extraction while increasing the trigger threshold for dense sampling mode, thereby reducing the overall number of keyframes extracted in normal steady-state scenarios. The rules for triggering frame extraction due to risk events remain unchanged. Abnormal scenarios such as video violations and audio abrupt changes can still trigger keyframe extraction and analysis in real time, ensuring that the content compliance monitoring capability is not weakened.
[0069] The third level of handling involves switching to a lightweight inference model. If the queue occupancy rate remains above 95%, the system will switch the primary multimodal inference model to a lightweight version with fewer parameters, reducing the inference time per task and increasing the task processing throughput per unit time. The model switch adopts a smooth transition mechanism: tasks already queued before the switch continue to be processed using the original model, while newly queued tasks use the lightweight model, avoiding load fluctuations and result gaps during the model switch.
[0070] The fourth level of handling safeguards core capabilities. If the queue backlog continues to worsen, the system enters a minimum computing power guarantee mode, suspending common semantic analysis tasks such as face recognition, fine-grained scene classification, speech recognition, and sentiment analysis. Only the risk detection links between the risk sentinel model and the main model are maintained to fully guarantee content compliance screening capabilities. The suspended semantic fields are filled in transitionally based on historical valid records using a semantic continuity interpolation mechanism to avoid gaps in the output semantic tags.
[0071] Throughout the entire process, the main video transmission link operated independently. Core functions such as video interface conversion, color space conversion, timing synchronization, and basic technical metadata pass-through were unaffected by AI-side load fluctuations, maintaining uninterrupted and stable output. For example, in the aforementioned sudden scenario, the queue backlog persisted. The system first executed the first-level processing, discarding the three earliest static empty shot tasks in the queue and retaining only the latest character appearance scene frames. If the queue load did not decrease after this processing, the second-level processing was executed, increasing the basic frame extraction interval from 25 frames to 50 frames to reduce the number of tasks generated in normal scenes. If the backlog worsened further, the system executed the third-level processing, switching to a lightweight multimodal model, reducing single-frame inference time from 180 milliseconds to 90 milliseconds, doubling the processing speed. If the load remained high under extreme conditions, the system entered the fourth-level backup mode, suspending all normal semantic analysis and retaining only risk detection capabilities. Other semantic fields were filled by historical interpolation, while the main video conversion link maintained normal output without any interruption or image quality degradation.
[0072] S104 sets up a four-level smooth degradation mechanism, caches 8 historical semantics for exponential smooth interpolation, and prohibits interpolation of risk fields; risk tasks are promoted to P0 with the highest priority to preempt computing power; high-risk scenarios are first written with temporary risk labels, and the labels are updated or revoked after the main model reviews them.
[0073] In one implementation, based on a four-level smooth hierarchical degradation architecture, an eight-line historical semantic circular cache, and risk-specific scheduling logic, exponential smooth interpolation, P0 highest priority preemption, and sentinel pre-screening dual-stage verification are used to achieve continuous semantic completion, risk computing power priority, and temporary risk label correction, respectively. The system establishes a four-level computing power smooth degradation system, dividing different operating levels according to NPU load, task queue occupancy, and inference timeout status, ranging from level zero to level three from normal to severe. Level zero is the normal mode, performing full multimodal analysis, with frame extraction intervals running according to an adaptive strategy, outputting complete semantic fields. Level one is a mild degradation, triggered by inference timeout or queue occupancy exceeding 60%. In this case, the output of long text and fine-grained object lists for speech recognition is disabled, retaining only the core fields of person, scene, emotion, and risk, while the upper limit of the frame extraction interval is adjusted to twice the basic interval. Level 2 represents a moderate downgrade, triggered when queue occupancy exceeds 80% or NPU utilization exceeds 90%. In this case, a lightweight inference model is switched to, and the output fields are further compressed into four items: person, scene, risk level, and importance score. Semantic interpolation is simultaneously enabled to fill in missing content. Level 3 represents a severe downgrade, triggered when queue occupancy exceeds 95% or three consecutive inference timeouts occur. In this case, only risk detection and scene category recognition capabilities are retained; all other semantic fields are filled by the semantic interpolation module.
[0074] For example, in routine news live streaming scenarios, the system operates in Level 0 normal mode, outputting all semantic fields in full. When the live stream enters peak periods with multi-camera switching, the AI workload increases, and the queue occupancy rate rises to 65%. The system then enters Level 1 mild degradation, disabling object list and long text speech recognition output to ensure the normal output of core semantics. If the workload continues to increase and the queue occupancy rate exceeds 85%, the system enters Level 2 moderate degradation, switching to a lightweight model and enabling interpolation to supplement fields. If a momentary task congestion causes the queue occupancy rate to exceed 95%, the system enters Level 3 severe degradation, ensuring only risk detection and scene category recognition are maintained, while other fields rely on historical interpolation to avoid gaps in semantic labels.
[0075] Interpolation compensation is performed using a cache of 8 historical semantic records. Slow-changing fields for people and scenes are updated using an exponential fusion with a smoothing coefficient of 0.6. Risk level fields are prohibited from interpolating to generate normal states. Degradation switching is configured with a 500ms hysteresis and a 2-level step-down anti-jitter suppression that frequently switch between each other. The system maintains a circular cache of 8 valid semantic records on the programmable logic side. It cyclically stores the most recently generated complete inference results in order of source frame number. Each record contains complete semantic fields, corresponding confidence levels, source frame numbers, and generation timestamps. When a new valid inference result is generated, it overwrites the oldest record in the cache in chronological order, always keeping the cache containing the latest 8 valid semantic data, providing a continuous temporal reference for interpolation calculations. For example, in a steady-state news live broadcast scenario, the frame extraction interval is 25 frames, and a valid inference record is generated every 25 frames. The cache sequentially stores 8 semantic results from different times, covering a time span of approximately 200 frames, which is sufficient to support interpolation completion calculations at different intervals.
[0076] When the current output frame lacks a corresponding AI inference result, or when the system is in degraded mode resulting in missing semantic fields, semantic interpolation compensation is automatically triggered. The interpolation calculation uses the distance between the source frame numbers as the weighting basis, selecting the two most recent valid records in the cache that are closest in time to the current frame. These records are then weighted and fused together with the confidence values of their respective fields to generate transitional semantic content. Valid records closer to the current frame receive a higher weight in the fusion calculation, ensuring the interpolation result aligns with the temporal trend. For example, if two valid records correspond to frames 1200 and 1225 respectively, and the current frame to be completed is 1210, the system calculates the weight ratio of the two records based on the frame number distance, and fuses them with the confidence values of their respective fields to obtain the transitional semantics for frame 1210. This ensures that even frames without inference results can output coherent semantic tags without any gaps or gaps.
[0077] For semantic fields like people and scenes that change slowly, the system uses an exponential smoothing algorithm to perform fusion updates, further mitigating label jumps caused by fluctuations in single results. The algorithm adjusts the fusion ratio of new results and historical values using a smoothing coefficient, with a default value of 0.6. This means the contribution of the new inference result is 0.6, while the contribution of the smoothed value at the previous moment was 0.4. Each time a new valid inference result is obtained, the smoothed output value of the corresponding field is updated according to this ratio, allowing semantic labels to transition naturally with content changes and avoiding abrupt jumps. For example, in a static studio scene, the main subject is always host A. A certain inference result temporarily identifies host B due to a change in the shooting angle, and the corresponding confidence level is low. During exponential smoothing calculation, after fusing the weight of the new result (0.6) with the weight of the historical value (0.4), the output still retains the judgment result of host A, preventing character label jumps caused by single recognition fluctuations and ensuring the long-term stability of the output semantics.
[0078] For fields like risk levels that change rapidly and directly impact content compliance, the system strictly prohibits generating safe or normal results through interpolation. During interpolation, the risk field always retains the result from the most recent actual detection. The risk level and corresponding label are only updated when a new valid inference judgment is obtained. This rule fundamentally prevents interpolation from masking the true risk, avoiding missed risk detections and ensuring the rigor of content compliance monitoring. For example, if the previous valid detection determined the risk level to be "suspected risk," and subsequent frames are in an interpolation completion state, the risk field in these frames will always use the "suspected risk" judgment result and will not be modified to a normal state due to interpolation. The risk level will only be updated to normal after a new inference result is returned and confirms no risk, ensuring that the risk signal is not diluted or eliminated by the interpolation mechanism.
[0079] The system operates in four levels based on computing unit load, queue utilization, and inference timeout status, progressively reducing computing power overhead from Level 0 (normal mode) to Level 3 (severe degradation mode). Each level has a specific trigger threshold: mild degradation is triggered by an inference timeout or a queue utilization exceeding 60%; moderate degradation is triggered by a queue utilization exceeding 80% or a computing unit utilization exceeding 90%; and severe degradation is triggered by a queue utilization exceeding 95% or three consecutive inference timeouts.
[0080] To avoid unnecessary level switching triggered by momentary load fluctuations, the system configures a 500-millisecond hysteresis window for all downgrade level changes. When a load metric is detected to reach the trigger threshold for a higher downgrade, a mode switch is not immediately executed. Instead, a timer window is started, continuously monitoring the metric status within the 500-millisecond window. Only if the metric remains above the threshold throughout the window is the downgrade / upgrade officially executed; if the metric falls below the threshold within the window, it is considered a momentary fluctuation, and no level change is triggered. Similarly, when the load declines and needs to exit downgrade mode, a 500-millisecond hysteresis check is performed to confirm that the load is continuously stable before reverting to the previous running level.
[0081] For example, in a live streaming scenario, a sudden peak in tasks occurs, causing the queue occupancy rate to briefly rise to 62%, and then drop back to 55% after two frames. Since the peak duration is less than 500 milliseconds, the hysteresis mechanism determines it as a momentary load fluctuation, and the system always remains in the zero-level normal mode. It will not briefly enter a slight degradation and then immediately switch back, thus avoiding repeated mode switching that could affect the consistency of semantic output.
[0082] Building upon the hysteresis mechanism, the system additionally employs a two-tiered anti-jitter rule. Both upgrades and downgrade callbacks proceed in a step-by-step manner, prohibiting jumps across levels. When the load increases, it can only proceed sequentially from level zero to level one, two, and three, increasing by only one level at a time. Similarly, when the load decreases, it decreases step-by-step from higher levels, decreasing by only one level at a time. After each level switch, a stabilization period of at least the corresponding single step is required to confirm a stable state before proceeding to the next level. This rule further mitigates mode jitter caused by load oscillations, ensuring a smooth transition between system operating levels. For example, if the system is currently in level zero normal mode, and the load continues to rise beyond the severe downgrade threshold, the system will not immediately jump to level three severe downgrade. Instead, it will first enter level one mild downgrade, stabilize for a period, and then gradually increase to level two and three. Similarly, when the load decreases, it decreases step-by-step, avoiding repeated jumps between high and low levels and ensuring the continuity and predictability of the semantic output strategy.
[0083] The entire interpolation and anti-shake mechanism is deeply integrated with the business logic. The interpolation results are directly used for metadata encapsulation and writing into the output video stream. The degradation level status is synchronously fed back to the frame extraction scheduling and task queue module, forming a closed-loop control of computing load and semantic output, taking into account both semantic continuity and compliance security in computing power fluctuation scenarios.
[0084] The system employs a four-level degradation strategy (Level 0 to Level 3) based on NPU load, queue usage, and inference timeout. Each level progressively simplifies output fields and switches to a lightweight model. Severe degradation retains only risk and major scenario identification, with missing fields filled by interpolation. A lightweight risk sentinel model is placed before the main multimodal analysis link, constructing a two-stage risk protection system combining rapid pre-screening and priority computing power scheduling. This achieves low-latency response to risky content without significantly increasing computing power overhead, and remains unaffected by degradation modes throughout the process, ensuring the rigid requirements of content compliance detection are met.
[0085] The Risk Sentinel model is an ultra-lightweight multimodal detection model with a total parameter count kept below 5 million. It is specifically designed for rapid initial screening of three types of compliance risks: illegal content, sensitive identifiers, and abnormal behavior. The model consists of five layers: an input preprocessing layer, a visual feature extraction branch, an audio feature extraction branch, a multimodal feature fusion layer, and a risk classification output layer. The input preprocessing layer unifies the input specifications for images and audio, completing data format conversion and size standardization. The visual feature extraction branch uses depthwise separable convolutions, comprising four convolutional stages, progressively downsampling to extract visual features from the image; its parameters account for 60% of the total model size. The audio feature extraction branch uses a combination of one-dimensional convolution and temporal pooling to extract temporal and frequency domain features from the audio; its parameters account for 20% of the total model size. The multimodal feature fusion layer uses a weighted concatenation method to fuse visual and audio features into a unified multimodal feature vector. The risk classification output layer connects to a fully connected classifier, outputting a three-level risk assessment result.
[0086] Model training was completed in two phases. The first phase was branch pre-training, where the visual branch was pre-trained on a public image violation detection dataset, and the audio branch was pre-trained on an abnormal sound detection dataset, with the network weights of the two branches initialized respectively. The second phase was joint fine-tuning, using a dedicated dataset for broadcast control scenarios for end-to-end fine-tuning. The dataset contained three categories of positive samples: violation content, sensitive markers, and abnormal behavior, as well as negative samples from various normal scenarios such as news studios, outdoor interviews, and variety show recordings. All samples were labeled with corresponding level-three risk tags. The training process used the cross-entropy loss function, with an initial learning rate set to 1 / 100,000, a batch size of 32, and 20 training epochs. The final model parameters were controlled to be within 5 million, and the single-frame inference latency did not exceed 20 milliseconds.
[0087] The model input data consists of fixed-size thumbnails and audio clips. Image input is a thumbnail scaled proportionally from the keyframes, with a uniform resolution of 224x224, RGB color format, and original aspect ratio maintained; insufficient areas are filled with zero pixels. Audio input is a 500ms PCM clip bound to the keyframes, uniformly downsampled to 16kHz mono, and maintaining a 16-bit bit depth. The model output provides a three-level risk warning: no risk, suspected risk, and high risk, simultaneously outputting the corresponding confidence level for each risk category. Detection results are directly transmitted to the task scheduling unit, serving as the core basis for task priority allocation and achieving deep integration between algorithm results and business scheduling logic. For example, in a steady-state scenario of daily news live streaming, the system extracts a keyframe every 25 frames. After the keyframe is generated, a corresponding thumbnail and audio clip are simultaneously generated and fed into the risk sentinel model. The model completes inference within 10 milliseconds and outputs a no-risk judgment result with a confidence level of 0.96. After the result is transmitted to the task scheduling unit, the task enters the regular inference queue according to normal priority, waiting for the main multimodal model to process it in sequence.
[0088] The system task scheduling architecture employs multi-level priority queues, with tasks carrying risk warnings assigned to the highest priority queue (P0), which has a higher priority than all ordinary analysis tasks. When the risk sentinel outputs a non-risk-free detection result, the task scheduler immediately transfers the analysis task for the corresponding keyframe from the ordinary queue to the highest priority queue (P0). Each time the computing power scheduling unit allocates NPU computing resources, it prioritizes searching the P0 queue. If there are pending tasks in the P0 queue, computing power resources are allocated to them first. When a high-risk task appears in the P0 queue and computing power resources are insufficient, the scheduler can temporarily pause the currently processing ordinary scene analysis task and prioritize allocating computing power resources to the risk task, ensuring that the total end-to-end latency from enqueuing to completing full review for the risk task does not exceed 150 milliseconds.
[0089] For example, during a live stream, the camera suddenly cuts to an outdoor scene with a potentially inappropriate tag. The risk sentinel model, upon receiving the corresponding thumbnail and audio, quickly identifies it as high-risk with a confidence level of 0.89. Upon receiving the high-risk alert, the scheduler immediately moves the analysis task for that frame to the highest priority queue (P0). At this time, the NPU is processing the analysis of a regular static empty shot. The scheduler temporarily pauses the inference process of this regular task, prioritizing all available computing power to process the high-risk frame. Ultimately, the total time from sentinel detection to the main model completing a full review of the risk task is 120 milliseconds, meeting the latency requirement of less than 150 milliseconds, ensuring that risky content can be identified and addressed promptly.
[0090] In the system's four-level smooth degradation system, the risk detection link is a core backup capability, unaffected by any degradation level, maintaining complete detection capabilities throughout. Regardless of whether the system is running in normal mode or level three severe degradation mode, P0 priority risk tasks always execute the complete detection process, including full risk label identification, confidence calculation, and violation category determination, without any reduction in output fields due to degradation. When the task queue implements an overload drop policy, all tasks carrying risk warnings are included in the protected task scope and not included in the drop candidate pool; only ordinary tasks without risk labels are dropped according to importance. During degradation level switching, the risk detection link continues to run without interruption or degradation of detection capabilities due to mode switching.
[0091] For example, during peak live streaming periods, the workload surges, and the system operates in a level 2 moderate downgrade state. The main inference model has been switched to a lightweight version, and ordinary tasks only output four core fields: person, scene, risk level, and importance score, while the queue occupancy rate remains consistently high. If a frame of content appears that appears to be in violation, the risk sentinel identifies it as a potential risk, and the corresponding task immediately enters the P0 queue. A complete violation content identification and detailed classification analysis process is then performed, outputting full risk details including risk category, risk location, and confidence level. If the queue occupancy rate further increases, triggering the discard mechanism, this risky task remains protected and will not be discarded. Multiple ordinary static empty shot tasks in the queue are discarded sequentially according to their importance, from lowest to highest. The entire risk detection chain remains unaffected by the moderate downgrade, maintaining full identification capabilities and meeting the stringent compliance requirements of broadcast control scenarios.
[0092] The risk sentinel model with ≤5M parameters outputs a three-level risk warning. Abnormal tasks are elevated to the P0 queue to preempt NPU computing power. In high-risk scenarios, temporary risk metadata with a provisional identifier is written first. After the main model reviews the data, the risk details are updated, or false alarm labels are removed within 1-2 frames, and false alarms are statistically analyzed for threshold self-learning. The P0 risk task detection chain is retained throughout all degradation modes without being discarded. When the risk sentinel model outputs a high-risk judgment result, and the main multimodal model has not yet returned a review conclusion, the temporary metadata writing process is immediately triggered to prioritize the timeliness of risk warnings.
[0093] The temporarily written risk metadata includes three core components: risk level, risk source marker, and temporary effective status marker. The risk level is synchronized with the high-risk or suspected risk assessment output by the sentinel; the risk source marker is the sentinel model; and the temporary effective status marker is true. It also includes the corresponding source frame number and hardware timestamp for accurate matching of subsequent verification results. This mechanism is deeply integrated with the metadata encapsulation module, allowing temporary metadata to be directly integrated into the standard encapsulation process and written into the auxiliary data area of the output video stream. This ensures that downstream devices can obtain risk warning signals immediately, preventing delays in risk warnings due to main model inference latency.
[0094] For example, if a suspected violation pattern caused by light and shadow distortion suddenly appears in the live broadcast, the risk sentinel model will complete the detection within ten milliseconds and output a high-risk judgment. At this time, the main multimodal model is still performing a full risk review, which is expected to take eighty milliseconds. The system immediately triggers temporary metadata writing, encapsulating the high-risk level, sentinel source marker, and temporary effective status marker into the auxiliary data area of the current output frame. Downstream broadcast control equipment can immediately receive the risk warning signal and prepare for handling in advance, with the overall warning latency controlled within twenty milliseconds.
[0095] After the main model completes a full risk review, it matches the corresponding temporary metadata based on the source frame number. The process is divided into two paths based on the review result: risk confirmation and false alarm removal. The first path is risk confirmation and formal implementation. If the main model confirms a compliance risk, the temporary effective status is updated to invalid, and the formal effective status is updated to effective. Simultaneously, complete risk details are added, including the specific risk category, the risk's location in the frame, the confidence level output by the main model, and detailed risk tags. The updated formal risk metadata continues to be transmitted with subsequent video frames, providing downstream devices with accurate risk assessment criteria. For example, if a violation marker appears in an outdoor live stream, the risk sentinel first identifies it as a high-risk risk and writes it into temporary metadata. 120 milliseconds later, the main model completes a full review, confirming that the marker is a sensitive violation marker with a corresponding confidence level of 0.94. The system then matches the temporary metadata of the corresponding source frame, removes the temporary effective marker, updates it to formal effective status, and adds details such as the sensitive marker category, marker frame coordinates, and confidence level. Subsequent output frames carry complete formal risk metadata for accurate location and handling by downstream audit systems.
[0096] The second path is the issuance of a false alarm clearance command. If the main model's review determines that the sentinel result is a false alarm and there is no corresponding compliance risk, a risk clearance command is immediately issued. The system removes the temporary risk marker within the next one to two frames, restoring the risk level to normal. Simultaneously, it records information such as the scene category, triggering reason, and sentinel output confidence level of this false alarm, storing it in the false alarm statistics database. For example, in a studio scene, rapidly flickering stage lights create a light and shadow pattern that is mistakenly judged as a high-risk event by the risk sentinel, and temporary metadata is written to the current output frame. After the main model completes a refined review, it determines that the image is a normal lighting effect with no violations, constituting a false alarm. The system then issues a risk clearance command, removing the temporary risk marker in the next output frame, restoring the risk level to normal, and storing the false alarm case caused by this light flickering in the false alarm statistics database, labeling the scene category as a studio lighting scene.
[0097] The system establishes a false alarm statistics and dynamic threshold adjustment mechanism. Based on accumulated false alarm sample data, it automatically optimizes the judgment threshold of the risk sentinel model, continuously reducing the probability of false triggers in similar scenarios. False alarm data is categorized and statistically analyzed along three dimensions: scenario category, triggering factor, and sentinel confidence interval. Scenario categories include sub-types such as studios, outdoor scenes, variety shows, and sports events. Triggering factors include common false alarm causes such as light and shadow flicker, camera movement, and image noise. When the cumulative number of false alarms in a specific scenario reaches a set threshold, or the false alarm rate exceeds a preset proportion, the system automatically raises the judgment threshold for the corresponding risk category of that scenario, increasing the detection threshold and reducing the detection sensitivity in that scenario. The adjustment process adopts a step-by-step, small-incremental approach, with each adjustment not exceeding 5% of the original threshold to avoid the risk of missed detections due to large fluctuations in the threshold.
[0098] The system updates the threshold every 100 valid false positive samples, or performs full-scene threshold calibration every 24 hours. The updated threshold directly affects the classification output layer of the risk sentinel model, enabling continuous iterative optimization of detection performance. For example, within a week, 23 high-risk false positives were triggered in a studio scene due to stage lighting flicker, exceeding the false positive rate of 8%, thus triggering a threshold adjustment. The system automatically increased the threshold for visual risk categories in the studio scene by 4%, appropriately raising the detection threshold. After the threshold update, the number of false positives in similar lighting flickering scenes decreased by approximately 60%, while the detection rate of normal risk content did not decrease significantly, effectively reducing the probability of false triggers while maintaining recall.
[0099] S105, metadata is written to the VANC area in the 12th row of the SDI blanking area and the HDMI Vendor Specific InfoFrame area without destroying the original video bitstream; the device panel displays the input format, recognized scene, person, risk level, and metadata injection status in real time. Downstream devices that support SMPTE ST 291 and HDMI extended frames can directly parse it to form video interface conversion information with real-time embedded multimodal semantic metadata.
[0100] In one implementation, based on SMPTE ST 291, HDMI Vendor Specific InfoFrame standard auxiliary channels, device local display modules, and downstream protocol compatibility specifications, the system employs SDI vertical blanking area writing, HDMI intra-frame embedding, real-time panel reporting, and standard stream output to achieve semantic metadata carrying, real-time status visualization, and cross-device parsing compatibility. The system relies on two types of industry-standard auxiliary data channels, coupled with local status display and downstream protocol compatibility, to complete the entire chain of semantic metadata from generation to carrying, presentation, and parsing through four corresponding implementation methods. These four implementation methods correspond to four scenarios: SDI signal transmission, HDMI signal transmission, device local maintenance, and cross-device service adaptation. The entire process adheres to publicly available and universal protocol specifications, without customizing proprietary data formats, ensuring standard compatibility and interoperability at every stage of the entire chain.
[0101] The semantic metadata carried under SDI signals follows the SMPTE ST 291 standard and is implemented by writing it into the vertical blanking area of the video signal. This process avoids touching the effective pixel area of the image and does not alter the quality or latency of the original main video stream. During the metadata encapsulation phase, the system assembles complete auxiliary data packets according to the standard protocol format. Each data packet contains five parts in sequence: data identifier, secondary data identifier, data count, user data word, and checksum. The data identifier and secondary data identifier are used to mark the service type and data ownership of the data packet, allowing downstream devices to distinguish auxiliary data for different purposes. The data count field records the byte length of the payload and is used by the receiving end to verify data integrity. The user data word area carries compressed and encoded structured semantic content and is the carrier of the actual semantic information. The checksum field is used for bit error checking during transmission to prevent semantic data anomalies caused by signal transmission interference. The encapsulation process strictly follows the byte order, word length, and alignment rules specified in the standard, ensuring that all professional video devices supporting this standard can parse it correctly.
[0102] The write location is selected as the VANC region in the twelfth row of the vertical blanking area. This region belongs to the auxiliary data space of the video signal and does not carry effective image content itself. The write operation will not affect the normal video image. When the video frame output scans to the corresponding row, the FPGA writes the encapsulated semantic metadata data packet into the data range of that row according to the standard timing, and outputs it synchronously with the main video stream. For example, the device is working in SDI to SDI conversion mode, and the input is a news live broadcast signal in 3G-SDI format. The structured semantic data generated by the AI module includes people, scenes, risk levels, voice and text, etc., and the total length after compression is 420 bytes. The FPGA encapsulates the semantic data into a complete VANC data packet according to the SMPTE ST 291 standard, and fills in the agreed data identifier, secondary data identifier, data count value, semantic payload and checksum in sequence. When the video frame scans to the twelfth row of the vertical blanking area, the FPGA writes the data packet into that row according to the standard timing. In the final output SDI signal, the effective image pixels are completely consistent with the original input, and the semantic data only exists in the blanking auxiliary channel. Downstream professional recording and broadcasting equipment that supports the SMPTE ST 291 standard can directly extract the complete semantic metadata from the signal.
[0103] The semantic metadata carried under HDMI signals follows the vendor-specific information frame specifications in the HDMI protocol, implemented by embedding information frame areas into each video frame, without modifying the effective pixel data. During the metadata encapsulation phase, the system assembles data packets according to the standard format of vendor-specific information frames. Each data packet contains four parts in sequence: a header, an IEEE unique identifier, a payload, and a checksum. The header identifies the type and total length of the information frame, allowing the receiver to quickly identify the frame type and perform preliminary verification. The IEEE unique identifier marks the vendor affiliation of the metadata, distinguishing extended information defined by different vendors and avoiding conflicts between extended data from different vendors. The payload area carries compressed structured semantic data, maintaining consistency with the semantic data of the SDI channel. The checksum is used for integrity verification of the entire frame data, preventing data errors during transmission. The encapsulation format fully complies with the HDMI protocol specifications and can be directly read by all downstream devices that support vendor-specific extended information frames.
[0104] The information frame is written to the transmission interval corresponding to each video frame and is transmitted synchronously with each video signal. Within the blanking interval of the HDMI output, the FPGA sends information frame data packets according to standard timing, maintaining strict timing synchronization with the main video link to ensure the time alignment of semantic data with the corresponding video frame. For example, when the device operates in SDI to HDMI conversion mode, semantic metadata needs to be embedded into the HDMI output signal. The FPGA receives the same structured semantic data output by the AI module and synchronously performs encapsulation according to the HDMI manufacturer's proprietary information frame standard. It writes the standard packet header, the corresponding IEEE organization unique identifier, the semantic payload, and the checksum to generate a complete information frame data packet. During the blanking phase of each HDMI video output frame, the FPGA embeds the information frame into the output stream according to the protocol timing. In the final output HDMI signal, the image quality, color, and latency are completely consistent with the converted main video signal; the semantic data exists only within the information frame extension channel. Downstream display devices and control consoles that support reading manufacturer-specific information frames can directly parse the embedded semantic metadata.
[0105] The system uses a local display unit on the front panel of the device to visualize the semantic analysis status and metadata injection status, providing intuitive operational status feedback to on-site maintenance and operation personnel. The panel display includes five core information categories: current input signal format, current recognition scene type, current person information in the frame, current content risk level, and current metadata injection status. The displayed data directly originates from the output results of the semantic analysis module and the status feedback from the metadata encapsulation module, maintaining synchronous updates with the semantic metadata embedded in the output video stream, eliminating display delays and data discrepancies.
[0106] The display update mechanism is synchronized with the keyframe analysis rhythm, updating the panel display content synchronously after each effective semantic analysis is completed. When the system is in degraded mode or temporary risk warning state, the panel will synchronously display the corresponding status marker, allowing operators to intuitively grasp the current system operation mode and risk status. For example, in a news studio live broadcast scenario, the device receives a 3G-SDI input signal, which is converted to HDMI output. The device's front panel synchronously updates to show that the current input format is 3G-SDI, the identified scene is a news studio, the person is host A, the risk level is normal, and the metadata injection status is injected. The director and maintenance personnel do not need to connect additional parsing equipment; they can directly and quickly confirm that the semantic analysis function is operating normally and the metadata injection link is stable through the panel. When suspected illegal content appears on the screen, triggering a temporary risk warning, the risk level field of the panel will be synchronously updated to a high-risk status and marked with a temporary marker, allowing operators to be aware of the anomaly immediately.
[0107] The system's output video stream fully complies with the industry-standard interfaces. All downstream devices supporting the corresponding standard protocols can directly read and parse the embedded semantic metadata without hardware modifications or protocol adaptation, achieving cross-device business compatibility. In SDI output scenarios, all downstream devices supporting the SMPTE ST 291 standard, including broadcast control servers, intelligent media asset systems, content review platforms, and professional recording devices, can directly extract structured semantic metadata from the vertical blanking region of the signal, which can be directly used for business scenarios such as director assistance, content review, material archiving, and rapid retrieval. In HDMI output scenarios, all downstream devices supporting the reading of manufacturer-specific information frames, including intelligent display terminals, director control consoles, and live streaming devices, can directly parse semantic metadata from the information frames, enabling functions such as real-time risk warnings and content information display.
[0108] The entire output system does not define proprietary data formats. The naming, structure, and encoding of semantic fields all follow general structured specifications. Downstream devices only need to adapt to standard protocols to complete data reading, significantly reducing integration costs and adaptation difficulties. For example, the downstream deployed intelligent media asset system itself supports SMPTE ST 291 standard parsing. After connecting to the SDI signal output by this device, it can directly extract scene, character, and risk tags for each frame, automatically completing material classification and content tagging, eliminating the need for offline AI analysis of recorded video files. The downstream deployed director control console supports reading vendor-specific information frames. After connecting to the HDMI signal output by this device, it can directly retrieve real-time risk level information. When high-risk content is detected, it automatically triggers alarm prompts, assisting directors in making quick decision-making. Neither type of downstream device requires customized development; they can obtain complete semantic metadata capabilities simply through standard protocol interfaces.
[0109] According to standard protocols, encapsulated semantic metadata is written to the VANC area of the 12th line of the SDI blanking region and the HDMI VendorSpecific InfoFrame area. This only occupies auxiliary data space and does not modify the effective video pixels or damage the original video stream. The system adopts a hardware architecture of full pass-through transmission of the main video stream plus embedded auxiliary data channels. The FPGA accurately writes the encapsulated standardized semantic metadata data packets into the non-effective screen area specified by the corresponding interface protocol, based on the output interface type. The writing operation is performed synchronously with the main video scanning clock at the hardware logic level, avoiding the effective screen pixel area throughout, ensuring that the structure, image quality, and latency of the original video stream are not affected by additional factors. The writing under SDI signals follows the SMPTE ST 291 standard, and the writing position is selected as the VANC area of the 12th line of the vertical blanking region. This area belongs to the standard auxiliary data space of the SDI signal, does not carry effective screen content itself, and is a common auxiliary data embedding position in the broadcasting industry.
[0110] The FPGA internally maintains independent line counters and pixel counters to track the current scan position of the output video in real time. When the line counter reaches the 12th line, the VANC write logic is triggered, pausing the original blanking level output of that line and writing the encapsulated semantic metadata data packet sequentially according to the standard word length and byte order. After the data packet is written, the scan output of subsequent lines returns to normal level, and the pixel data of all valid image lines is transmitted directly without modification. For example, the device receives a native 1080P 25fps 3G-SDI live broadcast signal and outputs an SDI signal of the same format. The structured semantic data generated by the AI module is 410 bytes long after compression and is encapsulated into a complete VANC data packet according to the SMPTE ST 291 standard. The FPGA's line counter scans line by line, and when it reaches the 12th line of the vertical blanking area, the data packet is written to the data range of that line according to the standard timing. The valid image pixel data from the 13th to the 1125th lines are all directly transmitted without any modification. In the final output SDI signal, the resolution, frame rate, color space, and pixel value of the effective image are completely consistent with the original input. The semantic data only exists in the blanking auxiliary channel of the 12th line. Downstream professional video equipment can normally parse the image and extract complete semantic information from the blanking area.
[0111] The writing process under HDMI signals follows the vendor-specific information frame specifications in the HDMI protocol, with the writing location being the information frame transmission interval corresponding to each video frame. This interval is located in the blanking transmission phase of each frame and does not belong to the transmission period of valid pixel data, thus not occupying the transmission bandwidth of the valid frame. The FPGA precisely locates the blanking transmission window for each frame in the frame-level timing control of HDMI output. After the valid pixel data transmission is completed, the vendor-specific information frame data packet is sent according to the timing specified in the protocol. The data packet sequentially includes a header, an IEEE organization unique identifier, a semantic payload, and a checksum. After the information frame is sent, the valid pixel transmission phase of the next frame begins. The entire process does not modify the value or transmission order of any valid pixel data.
[0112] For example, the device converts a 1080P 25fps signal input via SDI to HDMI 1.4 format output. The same structured semantic data is encapsulated according to the HDMI manufacturer's proprietary information frame standard, with the total length adapted to the upper limit of the information frame's payload. The FPGA inserts the information frame data packet according to the HDMI protocol timing during the blanking interval after each frame's effective pixels have been output. The RGB data, color depth, and frame rate of the effective pixels are all consistent with the converted main video parameters, without any modifications. In the final output HDMI signal, the display device can output a complete picture normally, with image quality completely consistent with the converted main stream. Downstream devices that support information frame reading can parse the embedded semantic metadata from each information frame.
[0113] The entire writing process employs hardware parallel timing control. The main video pixel path and the auxiliary data writing path are independent and parallel hardware logics within the FPGA, eliminating serial processing dependencies. Valid pixel data undergoes only physical layer conversion and clock synchronization from input to output, without additional algorithmic processing or buffering delays, resulting in transmission latency identical to that of ordinary interface conversion devices. Auxiliary data writing is performed only during blanking intervals or non-valid pixel transmission periods, without preempting valid pixel transmission bandwidth or altering the values, order, or transmission timing of valid pixels. All writing operations strictly adhere to the industry standard protocol of the corresponding interface, preventing the introduction of non-standard signal distortions, and preserving the integrity of the original video stream's structure, parameters, and quality indicators.
[0114] For example, the device performs SDI to HDMI format conversion while embedding semantic metadata. Under test conditions, the end-to-end transmission latency of the original SDI signal is 1.2 milliseconds. After the device performs the conversion and embeds the metadata, the end-to-end latency of the HDMI output is 1.3 milliseconds. The increase comes only from the physical layer overhead of the interface conversion itself; the metadata writing does not introduce additional latency. Using a professional video analyzer to test the image quality parameters of the output HDMI signal, the peak signal-to-noise ratio, color accuracy, contrast ratio, and other indicators are completely consistent with the output in pure conversion mode, with no image quality loss. When downstream display devices play the output signal, the image shows no superimposed characters, color blocks, or distortion. The semantic data exists entirely within the auxiliary channel and does not interfere with normal image display.
[0115] The device utilizes a local display unit on its front panel to output five categories of real-time monitoring information: input format, recognized scene, personnel information, risk level, and metadata injection status. This local display unit serves as the core status entry point for on-site operations and maintenance, outputting all five types of operational monitoring information. All displayed data is directly derived from the native output of the internal semantic analysis and metadata encapsulation links, maintaining synchronization with the semantic metadata embedded in the output video stream. Operations and maintenance personnel can intuitively grasp the operational status, semantic analysis results, and metadata injection status of the entire device chain without needing external specialized analysis equipment.
[0116] Input format monitoring information originates from the physical and protocol layer parsing results of the FPGA interface parsing module, covering core signal indicators such as interface type, resolution, frame rate, color space, and audio sampling parameters. The update mechanism employs an event-triggered mode, refreshing the display in real-time when a signal is connected, disconnected, or its format changes, while maintaining the current value under steady-state conditions. For example, when a device receives an SDI input signal, the FPGA interface parsing module identifies the signal specification as 3G-SDI, resolution 1920×1080, frame rate 25fps, color space YUV422, and audio sampling rate 48kHz from the input serial stream. The panel synchronously updates to display the current input format as 3G-SDI. Maintenance personnel can directly confirm the normal signal link connection and that the format parameters meet the broadcast system's expectations through the panel, without the need for additional waveform monitors or signal analyzers.
[0117] The scene monitoring information is derived from the scene classification inference output of a multimodal large model, along with the corresponding classification confidence level. The update mechanism is strictly linked to the keyframe analysis rhythm; the scene display is refreshed synchronously after each valid semantic inference is completed. When the system enters degraded mode and enables semantic interpolation, an interpolation status indicator is simultaneously marked to distinguish between real-time inference results and smoothly generated results. For example, in a news studio live broadcast scenario, the AI module performs scene classification inference on the extracted keyframes, outputting the scene type as "news studio" with a confidence level of 0.91. The panel synchronously updates the scene field to "news studio." When the system enters moderate degraded mode, the inference frequency decreases, and adjacent frame interpolation is enabled, an interpolation marker is simultaneously displayed next to the scene field, indicating that the current scene value is generated by weighted average of historical valid records and is not a real-time inference output.
[0118] The person information monitoring data comes from the face recognition inference output of a multimodal large model, including the person's name and recognition confidence level. The update mechanism is synchronized with keyframe inference, prioritizing the display of the main person with the largest screen area in multi-person scenes. To avoid label jumps caused by single recognition errors, the person display results are processed using an exponential smoothing algorithm, with a default smoothing coefficient of 0.6, meaning the new inference result has a weight of 0.6, and historical smoothing values have a weight of 0.4. For example, in a studio scene, if host A is in the center of the frame, the AI module performs face recognition inference, outputting "host A" with a confidence level of 0.93. The panel synchronously updates the person information field to "host A". When a slow camera zoom causes a temporary deviation in single recognition, the exponential smoothing algorithm integrates historical stable values with the new result to prevent frequent changes in person labels, maintaining display continuity and stability.
[0119] Risk level monitoring information originates from the two-tiered risk assessment output of the risk sentinel model and the main model, covering three categories: risk level, sub-risk tags, and status markers. The update mechanism employs a priority-triggered mode, immediately refreshing the display when the risk status changes. Temporary alerts generated by the initial sentinel screening are marked with a temporary status marker, which is updated to the official status after the main model's review. For example, under normal broadcast conditions, both the risk sentinel and the main model output a "no risk" assessment, and the panel displays a "normal" risk level. When suspected violations appear on screen, the lightweight risk sentinel first outputs a "high-risk" assessment, and the panel immediately updates the risk level to "high-risk" and marks it with a temporary marker, indicating it's the initial sentinel screening result. After the main model completes a full review confirming the risk, the panel removes the temporary marker, displays the official "high-risk" status, and adds the corresponding sub-risk tags; if the main model's review determines it's a false alarm, the display immediately reverts to the normal status.
[0120] Metadata injection status monitoring information comes from the status feedback of the FPGA metadata encapsulation and writing module, including three categories: injection link connectivity, data packet integrity verification results, and the currently active output interface type. The update mechanism is frame-level real-time verification. After each frame is written, verification and validation are automatically performed. If the verification passes, the system remains in normal status; if an anomaly occurs, it is immediately refreshed to an alarm status. For example, in SDI output mode, the FPGA completes metadata data packet encapsulation and blanking area writing according to the SMPTE ST 291 standard for each frame. If the verification and calculation match correctly, the injection link is running normally. The panel synchronously displays the metadata injection status as injected. When anomalies such as data packet verification failure or write position offset occur, the panel immediately updates to an injection anomaly status, allowing maintenance personnel to detect metadata link failures immediately without waiting for feedback from downstream devices.
[0121] The display updates of the five types of monitoring information are strictly synchronized with the internal processing links, with no additional caching delays. Input format and injection status information are updated in real-time at the frame level, while the semantic information of scene, person, and risk is consistent with the keyframe inference rhythm. All displayed data and the semantic metadata embedded in the output video stream come from the same data source, ensuring complete consistency between the panel display content and the actual embedded metadata, with no display deviation or time difference. This entire display mechanism eliminates the need for on-site directors and maintenance personnel to access the backend review system or connect external professional metadata parsing equipment. They can quickly confirm the normality of the signal link, the stable operation of the semantic analysis function, and the effectiveness of the metadata injection link simply through the device's front panel, significantly reducing the complexity of on-site maintenance and the cost of troubleshooting.
[0122] The output video stream is compatible with SMPTE ST 291 and HDMI Vendor Specific InfoFrame protocols. Downstream devices can directly read and parse the embedded semantic metadata, ultimately forming real-time embedded information of multimodal semantic metadata during the video interface conversion process. Semantic metadata uses the encapsulation format and transmission channel specified by the standard protocol in both types of interfaces. In the SDI output scenario, metadata strictly follows the SMPTE ST 291 standard, encapsulating into ANC and VANC data packets. Standard-defined data identifiers and sub-data identifiers are used as service type differentiation fields, and data counts, user data words, and checksums are all generated and verified according to standard specifications. In the HDMI output scenario, metadata is encapsulated into vendor-specific information frames, using a standard IEEE organization unique identifier as the attribution identifier. The packet header, payload, and checksum fully comply with the HDMI protocol specifications.
[0123] The payload content adopts a universal structured encoding format, with the basic encoding being UTF-8 format structured text. The compression algorithm uses universal lossless compression or binary structured encoding, allowing downstream devices to recover the complete semantic fields simply by performing a standard decompression process. Semantic data exceeding the single-frame payload limit uses a standard fragmented transmission mechanism. Each fragment carries a fragment sequence number and a total fragment number field. Downstream devices can reconstruct the complete semantics by concatenating the fragments according to the sequence number, without requiring dedicated decoding logic. For example, a professional recorder natively supporting the SMPTE ST 291 standard, after connecting to the SDI signal output by this device, can directly record the semantic metadata embedded in the blanking region and the video footage synchronously into the source file without upgrading firmware or adding a parsing module. The recorded footage can be directly read into the post-production system using the corresponding semantic tags, without the need for additional offline AI analysis.
[0124] Downstream devices under the SDI interface mainly cover the entire broadcasting production and broadcasting chain, including media asset management systems, content review platforms, broadcast control servers, professional recording equipment, etc. All types of equipment can directly implement corresponding business functions based on standard parsing capabilities. The intelligent media asset management system can directly parse the structured semantic data of each frame from the vertical blanking zone of the signal, extracting core fields such as people, scenes, risk levels, and voice / text, and automatically mapping them to the system's own tagging system to complete automatic material classification and content tagging. For long semantic content transmitted in segments, the media asset management system performs recombination verification according to the segment number and the total number of segments to obtain complete semantic information before performing tag mapping, avoiding information loss caused by segmentation. For example, news live broadcast materials are synchronously entered into the intelligent media asset management system. The system parses the semantic data of each frame in real time, automatically extracts tags such as host A, news studio, and normal, automatically classifies the material into the news program material library, and establishes an index of people, scenes, and content keywords. During subsequent material retrieval, staff can directly enter the person's name or scene type to locate the corresponding segment, without needing to manually review and tag each segment.
[0125] The content moderation platform can achieve real-time compliance monitoring of live streaming content based on the parsed risk level and risk tags. When the platform detects a risk level that is not normal, it automatically triggers the corresponding level of alarm response, including real-time pop-up prompts, automatic keyframe capture, and automatic screen recording and retention of abnormal segments. At the same time, it synchronizes the risk time point and risk category to the review ticket system. For example, when multiple live streaming signals are simultaneously connected to the content moderation platform, the platform parses the semantic metadata embedded in each signal in real time, and only silently records under normal circumstances. When a signal shows a high-risk tag, the platform immediately pops up a red alarm on the review interface, automatically captures the current screen and retains a segment of ten seconds before and after it, and generates a review ticket to be pushed to the reviewers. Reviewers can directly locate the abnormal time point for review without having to monitor every signal throughout the entire process.
[0126] Downstream devices using the HDMI interface primarily cover front-end monitoring and control scenarios, including broadcast control consoles, professional monitors, and live streaming equipment. These devices can directly read semantic data from information frames and overlay it into their own display and control logic. The broadcast control console can parse real-time semantic data from the manufacturer-specific information frames of the HDMI signal, extracting fields such as risk level, scene type, and personnel information, and overlaying them onto the status display area of the broadcast monitoring interface. When the risk level increases, interface alarms and audio prompts are automatically triggered to assist broadcast personnel in making quick broadcast control decisions. For example, when the studio broadcast control console is connected to the HDMI output signal of this device, the sidebar of the monitoring interface permanently displays the scene, personnel, and risk level information of the current frame. When suspected illegal content appears in the frame, triggering a high-risk warning, a red alarm bar immediately pops up at the top of the interface, accompanied by an audio prompt. Broadcast personnel can immediately become aware of the anomaly and perform a broadcast switch operation without waiting for manual reporting from back-end review personnel.
[0127] Professional display terminals can overlay semantic information onto the OSD display layer of the screen. Users can choose to enable or disable the semantic information display. When enabled, users can view information such as people, scenes, and risk status in the current scene without affecting the main screen viewing. For example, when the semantic OSD function is enabled on a studio monitor, the lower right corner of the screen displays the current scene and people information in real time, automatically highlighting it in red when an abnormal risk status occurs. On-site staff do not need additional metadata parsing equipment; they can grasp the semantic status of the content simply through the monitor.
[0128] The entire solution maintains standard consistency across the entire chain from data generation to business application, ensuring that the semantic data parsed by downstream devices is fully aligned with the embedded end in terms of timing, content, and completeness. Regarding timing consistency, semantic metadata achieves precise alignment with the corresponding video frames through a dynamic latency compensation algorithm and a dual-condition binding mechanism of frame number timestamps, controlling timing errors to within 1 millisecond. Each piece of semantic data parsed by downstream devices strictly corresponds to the currently output video frame, preventing semantic and visual misalignment and ensuring the accuracy of time-sensitive services such as director alerts and content review.
[0129] Regarding content consistency, all semantic fields adopt common naming rules and data formats, allowing downstream devices to directly map them to their own business field systems without requiring additional semantic conversion. The checksum mechanism follows the calculation rules of standard protocols, enabling downstream devices to verify data integrity through standard check logic and ensuring that there are no semantic deviations caused by bit errors during transmission.
[0130] Regarding transmission quality consistency, metadata embedding only occupies the standard auxiliary data channel, does not modify valid pixel data, does not occupy the main screen transmission bandwidth, and does not add additional transmission latency. The video processing link of downstream devices does not require any adjustments, and the image quality, frame rate, and latency of the main screen are completely consistent with the signal output by ordinary interface conversion devices. The original video transmission quality will not be affected by the embedding of semantic metadata.
[0131] like Figure 2 As shown, Figure 2 This is a system block diagram of the product, illustrating the hardware system architecture of the integrated intelligent video interface conversion device. The input side features two independent signal inputs: one is an SDI input interface with an SDI interface chip, and the other is an HDMI input interface with an HDMI interface chip. Both types of input signals are uniformly connected to the FPGA main control board.
[0132] The FPGA main control board, as the core scheduling unit, connects to six functional modules: DDR frame buffer, AI analysis module NPU / edge computing unit, gigabit network management interface, control display, SDI output interface, and HDMI output interface. The AI analysis module interacts bidirectionally with the FPGA main control board, while the other modules receive control commands and data scheduling from the FPGA main control board, collectively completing the entire process of video interface conversion, real-time semantic analysis, and metadata embedding.
[0133] like Figure 3 As shown, Figure 3This is a flowchart of the technical method, illustrating the complete processing chain of the real-time semantic metadata embedding method. The input video stream first enters the FPGA protocol parsing stage. After signal deserialization and basic parameter identification, it is split into two parallel processing branches: one performs keyframe extraction, and the other performs audio PCM extraction.
[0134] Two data streams, aligned to the same source, are simultaneously fed into the multimodal AI analysis module. After completing joint semantic reasoning of audio and video, standardized structured semantic data is generated, encoded in JSON or TLV format. This structured semantic data then enters the FPGA metadata encapsulation stage, matching the corresponding encapsulation path based on the output interface type: SDI output scenarios execute the SMPTE ST 291 standard encapsulation, while HDMI output scenarios execute the HDMI InfoFrame standard encapsulation. After both encapsulation processes are completed, the final output video stream, embedding semantic metadata, is output.
[0135] like Figure 4 As shown, Figure 4 This diagram illustrates the asynchronous analysis and time-aligned back-injection mechanism, showcasing the asynchronous inference and precise frame semantic alignment mechanism that decouples the main video link from the AI analysis bypass. The audio and video data output from the keyframe extraction module and the audio segment extraction module are first sent to the asynchronous task cache queue, which employs a hybrid architecture of circular caching and first-in-first-out (FIFO).
[0136] The task scheduling module retrieves tasks from the queue according to priority and assigns them to the AI asynchronous analysis thread pool for semantic inference. The semantic results after inference are entered into the semantic result output queue. The timestamp / frame number alignment module, in conjunction with the main video output frame counter and the PTS hardware clock, performs target output frame matching determination on the semantic results: if the target output frame has been reached, the semantic data is sent to the metadata injection encapsulation module, and the video stream is output after writing in ST291 or HDMI InfoFrame format; if the target frame has not been reached, the data is sent to the delay buffer waiting area, waiting for subsequent frames to cycle through matching and verification to ensure that the semantic data is accurately aligned with the video timeline.
[0137] like Figure 5 As shown, Figure 5 This diagram illustrates the anomaly handling and degradation strategy mechanism, showcasing the tiered handling and degradation mechanisms for abnormal scenarios such as task queue overload and insufficient computing power. When an asynchronous task enters the queue, a queue length threshold is first determined. If the queue length does not exceed the preset threshold, the task proceeds normally into the AI analysis process; otherwise, the anomaly handling strategy is triggered.
[0138] The system categorizes anomalies into four handling paths: In scenarios with queue backlog, the oldest tasks are discarded, retaining only the most recent keyframes; in scenarios with AI inference timeouts, a lightweight inference model is switched to; in scenarios with high computational load, the frame extraction frequency is reduced to decrease the number of tasks generated; in risk-priority scenarios, only risk detection tasks are retained, and general semantic analysis is suspended. After these four handling strategies are executed, the system continues asynchronous analysis, ensuring continuous and stable output of the main video stream throughout, preventing AI-side anomalies from affecting the normal transmission of the main video stream.
[0139] This application also provides a real-time embedding device for semantic metadata, including: The audio and video synchronous acquisition frame extraction module is used to receive SDI and HDMI video streams and PCM audio. The FPGA parses the audio and video timing parameters, and uses 16×16 macroblock MAD, HSV Bass distance and 2-frame verification to identify scene switching. It captures audio mutations with a 300ms silent baseline and multi-dimensional audio energy threshold. It adaptively adjusts the sampling interval based on multi-factor importance scoring, extracts 500ms of synchronous audio and marks 1ms hardware PTS to complete audio and video homogeneous alignment. The multimodal semantic reasoning module is used to perform face recognition, scene classification, ASR speech, emotion, and multi-dimensional detection of violation risks on preprocessed frames and synchronous audio. It outputs UTF-8 structured JSON, uses gzip / CBOR compression and constrains the single frame payload to ≤512 bytes. If the limit is exceeded, it is truncated according to field priority, long messages are fragmented and carry segmentation identifiers, and a lightweight sentinel model with ≤5M parameters is used to screen the risk level in advance. The metadata encapsulation timing binding module is used to encapsulate semantic packets in accordance with the SMPTE ST 291 and HDMI Vendor SpecificInfoFrame standards. It adopts a decoupled architecture between the main transmission link and AI inference bypass, and dynamically calculates compensation latency by combining the baseline latency, inference time, and queue depth. It completes accurate semantic binding through frame number + PTS dual verification. The task queue limit is 64, and only low-weight ordinary tasks are discarded under high load. Expired frames open a 3-frame compensation window to reissue semantic tags. The graded and downgraded risk scheduling module is used to build a smooth downgrade system of computing power from Level 0 to Level 3. It caches 8 historical semantics and uses exponential interpolation with α=0.6 to fill in the missing content, and prohibits interpolation of risk fields. Risk tasks are assigned to P0 with the highest priority to preempt NPU computing power. High-risk scenes are written to temporary risk metadata first, and the temporary risk mark is updated or revoked after the main model reviews it. The semantic stream output presentation module is used to write encapsulated metadata into the VANC and HDMI VendorSpecific InfoFrame auxiliary channels on line 12 of SDI without altering the original video bitstream. The device panel synchronously displays five types of real-time information: input specifications, scene, person, risk, and injection status. It is compatible with downstream devices of the corresponding standards for direct parsing and outputs an integrated video conversion stream with real-time semantics.
[0140] A computing device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the device is triggered to execute a real-time embedding method for any semantic metadata.
[0141] The methods and / or embodiments in this application can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by a processing unit, it performs the functions defined in the methods of this application.
[0142] It should be noted that the computer-readable medium described in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0143] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application.
Claims
1. A real-time embedding method for semantic metadata, characterized in that, include: Acquire SDI / HDMI video streams and PCM audio, and use FPGA to analyze basic audio and video parameters; Combining 16×16 macroblock MAD, HSV Bass distance plus 2-frame verification to determine scene switching, after 300ms noise floor calibration, sound abrupt changes are identified based on audio energy index, adaptive frame extraction is achieved based on content importance score, 500ms of synchronized audio is extracted and a 1ms precision hardware PTS timestamp is added to complete the same source binding. The extracted frames are preprocessed and fed into the edge multimodal large model to complete face recognition, scene classification, ASR, emotion and violation risk detection, and output structured JSON semantic data. After gzip / CBOR compression, the single frame payload is ≤512 bytes. If the limit is exceeded, it is truncated according to priority. The excessively long data is segmented and carries fragmentation identifiers. The risk sentinel model with ≤5M parameters pre-screens the risk level. Metadata is encapsulated according to SMPTE ST 291 and HDMI Vendor Specific InfoFrame standards. It adopts a main link and AI bypass separation architecture, and dynamically compensates for latency by combining baseline latency, inference time, and queue depth calculation. It relies on frame number + PTS dual matching to bind semantics. The task queue has a maximum limit of 64. If the queue is overloaded, only low-importance ordinary tasks are discarded. If the target frame expires, the tag can be resent within 3 frames. A four-level smooth degradation mechanism is set up, 8 historical semantic records are cached for exponential smooth interpolation, and interpolation is prohibited for risk fields; risk tasks are promoted to P0 with the highest priority to preempt computing power, and temporary risk labels are written to high-risk scenarios first, and the labels are updated or revoked after the main model reviews them. Metadata is written to the VANC area in the 12th row of the SDI blanking area and the HDMI Vendor Specific InfoFrame area without destroying the original video bitstream; the device panel displays the input format, recognized scene, person, risk level, and metadata injection status in real time. Downstream devices that support SMPTE ST 291 and HDMI extended frames can directly parse it to form video interface conversion information with multimodal semantic metadata embedded in real time.
2. The real-time embedding method for semantic metadata according to claim 1, characterized in that, Acquire SDI / HDMI video streams and PCM audio; FPGA analyzes basic audio and video parameters; combines 16×16 macroblock MAD, HSV Bass distance, and 2-frame checksum to determine scene switching; after 300ms noise floor calibration, identify sudden sound changes based on audio energy indicators; implement adaptive frame extraction based on content importance scoring; extract 500ms of synchronized audio and add a 1ms precision hardware PTS timestamp to complete source binding, including: The system acquires SDI and HDMI input video streams and PCM audio data, and uses FPGA to analyze resolution, frame rate, color format, and audio timing parameters. It uses 16×16 macroblock MAD mean and HSV histogram Bach distance to jointly determine scene switching, and adds a continuous 2-frame verification mechanism to suppress lens push-pull and flash white false triggers. Based on a 300ms silent noise floor, audio mutation events are identified using three quantitative indicators: RMS amplification of over 6dB, instantaneous peak value of -18dBFS, and short-term energy slope of 3dB / 20ms. The sampling interval is dynamically adjusted by integrating three-layer strategies: fixed-period frame sampling, event-triggered frame sampling, and multi-factor content importance scoring adaptive frame sampling. When the importance suddenly increases, 1-2 frames are backtracked from the DDR circular cache to avoid missed detection. A 500ms audio window is captured centered on the frame hardware capture time. The start and end indices of the audio samples are calculated by converting the sampling rate. A hardware PTS timestamp with 1ms precision is uniformly marked. The audio and video are packaged with frame_id, audio and video start and end timestamps, sampling rate, and bit depth fields to complete the audio and video source binding.
3. The real-time embedding method for semantic metadata according to claim 1, characterized in that, Metadata is encapsulated according to SMPTE ST 291 and HDMI Vendor Specific InfoFrame standards. The main link and AI bypass are separated into an architecture. Dynamic latency compensation is calculated by combining baseline latency, inference time, and queue depth. Semantics are bound by frame number + PTS dual matching. The task queue has a maximum capacity of 64. If the queue is overloaded, only low-importance, regular tasks will be discarded. Expired target frames can be re-tagged within 3 frames, including: Semantic metadata is encapsulated according to SMPTE ST 291 and HDMI Vendor Specific InfoFrame protocols. A decoupled architecture is adopted for the main video transmission link and AI asynchronous analysis bypass. Dynamic compensation latency is calculated according to D_act=D_fix+ceil((T_infer-T_target) / T_frame)+Q_depth, where D_act is the total number of actual back-injection latency frames in the final output, D_fix is the baseline fixed latency frame number, T_infer is the actual inference time of the current task, T_target is the preset target inference budget, T_frame is the duration of a single frame of video at the current frame rate, Q_depth is the queue depth compensation frame number, and ceil is the rounding up operation rule. By binding semantics with both output frame number and hardware PTS timestamp, the matching and verification error is controlled within 1ms. The task cache queue adopts a FIFO + priority hybrid scheduling, with a basic limit of 64 and a high-performance mode that expands to 128. When the queue occupancy exceeds 80%, only low-importance ordinary tasks are discarded, while risk-related tasks are not discarded. If the target output frame has already been sent out, a 3-frame compensation window is opened to write to the nearest frame, the super window is marked as expired and a summary tag is sent to the next frame; when the AI queue is backed up or inference times out, old keyframes can be discarded, the frame extraction frequency can be reduced, a lightweight model can be switched, and only risk detection can be retained to ensure continuous output of the main video link throughout the process.
4. The real-time embedding method for semantic metadata according to claim 1, characterized in that, A four-level smooth degradation mechanism is set up, caching 8 historical semantic records for exponential smooth interpolation, and interpolation is prohibited for risk fields; risky tasks are promoted to P0 with the highest priority to preempt computing power, and high-risk scenarios are first written with temporary risk labels, which are updated or revoked after review by the main model, including: Based on a four-level smooth hierarchical degradation architecture, eight historical semantic ring caches, and risk-specific scheduling logic, exponential smooth interpolation, P0 highest priority preemption, and sentinel pre-screening two-stage review are used to achieve semantic continuous completion, risk computing power priority, and temporary risk label correction, respectively. Interpolation compensation is carried out based on the cache of 8 historical semantic records. The characters and scenes with slow changes are updated by exponential fusion with a smoothing coefficient of 0.
6. The risk level field is prohibited from interpolation to generate a normal state. The downgrade switching configuration is configured to switch frequently between 500ms hysteresis and level 2 step anti-shake suppression. Based on NPU load, queue usage, and inference timeout, the degradation is divided into four levels: Level 0 to Level 3. Each level simplifies the output fields and switches to a lightweight model. The severe degradation only retains the risk and major scene recognition, and the missing fields are filled by interpolation. The risk sentinel model with ≤5M parameters outputs a three-level risk warning. Abnormal tasks are promoted to the P0 queue to seize NPU computing power. In high-risk scenarios, temporary risk metadata with provisional identifier is written first. After the main model reviews, the risk details are updated or false alarm labels are removed within 1 to 2 frames, and false alarms are counted for threshold self-learning. The P0 risk task detection link is retained throughout the entire process without being discarded under various degradation modes.
5. The real-time embedding method for semantic metadata according to claim 4, characterized in that, Metadata is written to the VANC area (line 12) of the SDI blanking region and the HDMI Vendor Specific InfoFrame area without disrupting the original video stream. The device panel displays the input format, identified scene, person, risk level, and metadata injection status in real time. Downstream devices supporting SMPTE ST291 and HDMI extended frames can directly parse this data, forming multimodal semantic metadata embedded in real-time video interface conversion information, including: Based on SMPTE ST 291, HDMI Vendor Specific InfoFrame standard auxiliary channel, device local display module, and downstream protocol compatibility specifications, semantic metadata carrying, real-time status visualization, and cross-device parsing compatibility are achieved by using SDI vertical blanking area writing, HDMI intra-frame embedding, panel real-time reporting, and standard stream output, respectively. According to the standard protocol, the encapsulated semantic metadata is written to the VANC area of the 12th line of SDI blanking and the HDMI VendorSpecific InfoFrame area, which only occupies auxiliary data space, does not modify the effective pixels of the video, and does not destroy the original video bitstream; Based on the local display unit on the front panel of the device, it outputs five types of real-time monitoring information: input format, recognition scene, personnel information, risk level, and metadata injection status. The output video stream is compatible with SMPTE ST 291 and HDMI Vendor Specific InfoFrame protocols. Downstream devices can directly read and parse the embedded semantic metadata, ultimately forming real-time embedded information of multimodal semantic metadata during the video interface conversion process.
6. A real-time embedding device for semantic metadata, characterized in that, The apparatus is used to execute executable instructions to perform the real-time embedding method of semantic metadata as described in any one of claims 1 to 5.
7. A real-time embedding system for semantic metadata, characterized in that, The system is used to execute executable instructions to perform the real-time embedding method of semantic metadata as described in any one of claims 1 to 5.
8. An electronic device, characterized in that, include: First processor; And a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the real-time embedding method of semantic metadata according to any one of claims 1 to 5 by executing the executable instructions.
9. A computing device, the device comprising a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein, When the computer program instructions are executed by the processor, the device is triggered to execute the real-time embedding method of semantic metadata as described in any one of claims 1 to 5.
