Real-time roi extraction and signal processing system of two-way video stream integrated with ai vision

Abstract

The present application relates to the technical field of video signal processing and multi-source data synchronization, in particular to a two-way video stream real-time ROI extraction and signal processing system integrated with AI vision, aiming to solve the problem that when two independent video streams are identified by AI vision, the time-consuming calculation fluctuates due to the complexity of the target in the frame and the fluctuation of hardware scheduling, resulting in time sequence deviation of the two signals and asynchronization of the same physical event feature points, affecting signal alignment and comprehensive information extraction, through a video acquisition unit to obtain two video streams, an AI vision identification unit to adopt a lightweight convolutional neural network and a bidirectional long short-term memory network to detect and identify ROIs in real time and output features and identification signals, a time sequence deviation compensation unit to monitor the time-consuming calculation fluctuation and generate a smooth time sequence offset compensation deviation against the ring effect, and a signal processing unit to generate a comprehensive signal matrix through dynamic time warping, Gaussian kernel smoothing and attention fusion, and realize feature point alignment in combination with phase error feedback closed loop and the Hungarian algorithm.

Description

Real-time ROI capture and signal processing system for two video streams integrating AI vision Technical Field

[0001] This invention relates to the field of video signal processing and multi-source data synchronization technology, specifically to a system for real-time ROI capture and signal processing of two video streams integrated with AI vision. Background Technology

[0002] Video signal processing and multi-source data synchronization is an important technology, specifically applied to the precise correlation processing of physical events between two independent video streams. The core is to restore the complete information of physical events through ROI extraction and signal synchronization processing, adapting to the core requirements of multi-source video data fusion in monitoring and detection scenarios. During AI visual ROI recognition of two independent video streams, computation time fluctuates due to the complexity of the target within the frame and hardware scheduling factors, leading to asynchronous processing progress between the two signals. Consequently, the feature points of the same physical event in the two signals exhibit temporal deviations, affecting signal alignment and comprehensive information extraction, and failing to accurately restore the full picture of the physical event. To solve this technical problem, we provide a real-time ROI extraction and signal processing system for two video streams integrating AI vision. Summary of the Invention

[0003] The purpose of this invention is to provide a real-time ROI capture and signal processing system for two video streams integrated with AI vision, in order to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, a real-time ROI acquisition and signal processing system integrating AI vision for two video streams is provided, including:

[0005] The video acquisition unit is used to acquire two video streams from two independent video sources in real time and transmit the two video streams to the AI ​​visual recognition unit.

[0006] The AI ​​visual recognition unit is used to perform real-time ROI detection and recognition on each video frame in the two video streams, and outputs the feature signal of the ROI region and the corresponding recognition result signal.

[0007] The signal processing unit is used to align the feature signals of the ROI region with the corresponding recognition result signals to generate a comprehensive signal of the physical event;

[0008] The timing deviation compensation unit is used to monitor the computation time fluctuation of the AI ​​visual recognition unit in real time, and to compensate for the timing deviation in the two signal processing based on the computation time fluctuation. This includes recording the start timestamp when the AI ​​visual recognition unit starts ROI recognition for each video frame, recording the end timestamp when the recognition is completed, calculating the time difference between the start timestamp and the end timestamp as the actual computation time value of the video frame, and setting a benchmark computation time value as a reference. The timing offset is determined by comparing the deviation between the actual computation time value and the benchmark computation time value, and the timing offset is sent to the signal processing unit in real time. During the signal alignment process, the signal processing unit adjusts the phase position of the feature signal according to the timing offset, so that the feature points of the same physical event in the two signal processing results remain synchronized.

[0009] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0010] This invention acquires two independent video streams in real time through a video acquisition unit. The AI ​​visual recognition unit uses a lightweight convolutional neural network and a bidirectional long short-term memory network to efficiently complete ROI detection and recognition, outputting a signal vector that fuses position and color features, eliminating redundant areas to ensure real-time performance. The timing deviation compensation unit uses a sliding time window and exponential weighted smoothing to accurately calculate the time-consuming fluctuation value, and combines the fluctuation change rate to generate a smooth timing offset to resist ringing effects, dynamically compensating for the timing deviation caused by the time-consuming fluctuation. The signal processing unit uses a dynamic time warping algorithm to match key waveform inflection points, generates a comprehensive signal matrix through Gaussian kernel smoothing and attention fusion, and then ensures high-precision synchronization of the two signals through phase calibration and feedback closed-loop optimization. Finally, the Hungarian algorithm and backtracking mechanism are used to achieve complete alignment of feature points, effectively restoring the full picture of the physical event and improving the accuracy and real-time performance of the two video stream signal processing. Attached Figure Description

[0011] Figure 1 is an overall block diagram of the present invention.

[0012] The meanings of the labels in the diagram are as follows:

[0013] 1. Video acquisition unit; 2. AI visual recognition unit; 3. Signal processing unit; 4. Timing deviation compensation unit. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] This invention provides a real-time ROI capture and signal processing system for two video streams integrated with AI vision, as shown in Figure 1, including:

[0016] The video acquisition unit 1 is used to acquire two video streams from two independent video sources in real time and transmit the two video streams to the AI ​​visual recognition unit 2.

[0017] AI visual recognition unit 2 is used to perform real-time ROI detection and recognition on each video frame in the two video streams, and output the feature signal of the ROI area and the corresponding recognition result signal.

[0018] The signal processing unit 3 is used to align the feature signals of the ROI region and the corresponding recognition result signals to generate a comprehensive signal of the physical event;

[0019] The timing deviation compensation unit 4 is used to monitor the fluctuation value of the calculation time of the AI ​​visual recognition unit 2 in real time, and to compensate for the timing deviation in the two signal processing based on the fluctuation value of the calculation time. This includes recording the start timestamp when the AI ​​visual recognition unit 2 starts the ROI recognition of each video frame, recording the end timestamp when the recognition is completed, calculating the time difference between the start timestamp and the end timestamp as the actual calculation time value of the video frame, and setting a benchmark calculation time value as a reference. The timing offset is determined by comparing the deviation between the actual calculation time value and the benchmark calculation time value, and the timing offset is sent to the signal processing unit 3 in real time. During the signal alignment process, the signal processing unit 3 adjusts the phase position of the feature signal according to the timing offset, so that the feature points of the same physical event in the two signal processing results remain synchronized.

[0020] A lightweight convolutional neural network is used as the backbone feature extractor. Multi-scale sliding window scanning is performed on each video frame. Candidate ROI regions are generated through preset object localization anchor boxes. The contextual feature vectors of the candidate ROI regions are extracted through spatial pyramid pooling layers and input into a cascaded bidirectional long short-term memory network for temporal dependency modeling. Redundant candidate regions in adjacent frames are eliminated. Finally, the spatial coordinate data of the ROI region and its corresponding object category confidence are output by a fully connected layer to form the corresponding recognition result signal.

[0021] The spatial coordinate data is converted into a normalized bounding box tensor sequence. At the same time, the YUV color space histogram statistics of pixels within the ROI region are extracted. The bounding box tensor sequence and the YUV color space histogram statistics are concatenated into a dimension-fused feature signal vector. The recognition result signal is filtered through the object category confidence threshold and then a timestamp is added to form a signal pair that is aligned with the feature signal vector.

[0022] A timeline based on the physical event trigger time is established. The key waveform inflection points of the two feature signals are matched by a dynamic time warping algorithm. After time offset compensation, the aligned feature signals are locally weighted and smoothed using a Gaussian kernel function to eliminate high-frequency noise. Finally, the two smoothed signals are input into a feature fusion engine based on an attention mechanism to generate a comprehensive signal matrix containing spatial and temporal correlations.

[0023] The calculation of the time fluctuation value specifically includes:

[0024] The actual computation time of N consecutive frames is continuously recorded to form a sliding time window. The arithmetic mean of the actual computation time within the sliding time window is calculated in real time as the dynamic baseline time. The difference between the actual computation time of the current frame and the dynamic baseline time is divided by the dynamic baseline time to obtain the normalized fluctuation coefficient. The fluctuation coefficient is noise-suppressed by the exponential weighted moving average algorithm to output a stable time fluctuation value.

[0025] Determining the timing offset specifically includes:

[0026] The time fluctuation value is multiplied by the preset system clock cycle conversion coefficient to generate the original time offset. At the same time, the rate of change of the fluctuation value in adjacent frames is detected. When the rate of change of the fluctuation value exceeds the adaptive threshold, the secondary compensation mechanism is activated. The time offset of the previous frame is used as the reference, and the product of the original time offset of the current frame and the rate of change is added to generate the final composite time offset used for phase adjustment.

[0027] A double-buffered timing offset register is established to alternately store the calculation results of the current frame and the previous frame. The transition direction of the rate of change of the fluctuation value is monitored by a finite state machine. When a continuous positive and negative jump is detected, the phase correction factor is activated. The phase correction factor is convolved with the composite timing offset to generate a smooth timing offset output with anti-ringing effect.

[0028] A ring buffer is set up in the signal processing unit 3 to store the feature signal stream. The index offset of the signal sample point is calculated based on the received smooth timing offset. The offset signal waveform is reconstructed using the Lagrange interpolation algorithm. At the same time, the original signal spectrum features are retained through the bypass channel. Finally, the reconstructed signal and the original spectrum are fused by inverse Fourier transform to generate a new feature signal stream with phase calibration.

[0029] A phase error feedback closed loop is constructed to compare the waveform similarity of two new feature signal streams at the event trigger time in real time. When the similarity is lower than the dynamic threshold, the timing deviation compensation unit 4 is triggered to recalibrate the reference calculation time value. At the same time, the order parameter of the interpolation algorithm is optimized through the gradient descent algorithm so that the phase error of the reconstructed signal converges within the preset tolerance range.

[0030] In the comprehensive signal matrix, the spatiotemporal feature vectors of physical events are located, their local maxima are extracted as key feature points, a bidirectional optimal matching channel for the feature points of the two signals is established, the minimum path cost matching pair is solved by the Hungarian algorithm, a backtracking mechanism is initiated for feature points that are not successfully matched, the search is reversed along the time axis to the previous synchronization point, interpolation compensation is performed using the time offset, and the coordinates of the missing feature points are forcibly aligned.

[0031] Further explanation is needed: after the video acquisition unit 1 transmits the video streams from two independent video sources to the AI ​​visual recognition unit 2 in real time, this unit needs to complete the real-time detection and recognition of the ROI for each video frame through an end-to-end AI visual processing flow. The core is to rely on a lightweight model to ensure real-time performance, and to remove redundancy through multi-scale feature extraction and temporal modeling, ultimately outputting accurate ROI features and recognition signals. The specific implementation method is as follows:

[0032] First, a lightweight convolutional neural network is used as the backbone feature extractor. This lightweight convolutional neural network simplifies the structure of a deep learning network through techniques such as depthwise separable convolution and channel pruning. While maintaining feature extraction capabilities, it significantly reduces computational load and memory usage, perfectly adapting to the computational demands of real-time processing of two video streams and avoiding frame delays caused by model complexity. Based on this backbone network, a multi-scale sliding window scan is performed on each video frame. This multi-scale sliding window scan is a detection method that uses rectangular windows of different sizes to slide across the video frame region by region with a fixed stride. The purpose is to cover target objects of different sizes. The specific process is as follows:

[0033] Three sets of sliding windows of different scales (small, medium, and large) are preset, with the window step size set to 1 / 4 of the window's side length (ensuring reasonable overlap between adjacent windows and no target omission). Scanning begins from the top left corner of the video frame, sliding horizontally one row at a time before moving downwards. For the area covered by each window, basic features are extracted using a lightweight convolutional neural network to mark candidate regions that may contain the target, providing feature support for subsequent anchor box matching. Then, candidate ROI regions are generated using preset object localization anchor boxes. These object localization anchor boxes are pre-defined bounding box templates with various aspect ratios and sizes, their dimensions corresponding one-to-one with the multi-scale sliding windows (small windows match small-sized anchor boxes, large windows match large-sized anchor boxes). Their core function is to quickly pinpoint the possible location range of the target. A candidate ROI region is a rectangular area that may contain the target object; the ROI, or Region of Interest, is the core object for subsequent feature extraction and recognition. The specific generation process is as follows:

[0034] For each sliding window, the extracted basic features are matched with all anchor boxes at the corresponding scale. The intersection-union ratio (IU) of the anchor box and the target candidate region in the feature map (measuring the degree of overlap between two rectangular regions) is calculated. When the IU is higher than a preset threshold (experimentally calibrated to 0.5), the anchor box is determined to cover the potential target. The video frame region corresponding to the anchor box is marked as a candidate ROI, and the initial coordinates of the anchor box are recorded, forming a set of regions containing multiple candidate ROIs. After the candidate ROI regions are generated, the context feature vector is extracted through a spatial pyramid pooling layer. The spatial pyramid pooling layer is a network layer that can uniformly convert candidate ROIs of different sizes into fixed-dimensional features, which solves the feature extraction problem caused by inconsistent candidate ROI sizes. The context feature vector is a high-dimensional vector containing local details of the candidate ROI and information related to the global environment, which can comprehensively characterize the feature attributes of the target. The specific extraction process is as follows:

[0035] Each candidate ROI is divided into multiple sub-regions at different spatial scales. Max pooling is performed on each sub-region to extract its key features. The pooling results from different scale sub-regions are then concatenated sequentially to form a fixed-length high-dimensional vector. This vector contains both local details of the candidate ROI and global contextual relationships, thus becoming the context feature vector. This context feature vector is then input into a cascaded bidirectional long short-term memory (LSTM) network for temporal dependency modeling. The cascaded bidirectional LSM network is a temporal network composed of multiple LSM units connected in series. Bidirectional means that the network processes feature sequences both forward (in the order of video frame playback) and backward (in reverse order of video frame playback), enabling it to simultaneously capture the temporal relationships between the current frame and past and future adjacent frames, avoiding the loss of temporal information caused by single-direction modeling. Temporal dependency modeling involves the network learning the relationships between candidate ROIs in adjacent frames, such as the positional changes and morphological consistency of the same target in consecutive frames. The specific process is as follows:

[0036] The contextual feature vectors of candidate Regions of Interest (ROIs) from consecutive frames in the same video stream are arranged in timestamp order and input into a cascaded bidirectional LSTM network. The network, through the synergistic action of forget gates, input gates, and output gates, filters and memorizes effective temporal correlation information while forgetting irrelevant noise. For candidate ROIs with an overlap exceeding a preset threshold in adjacent frames, they are identified as redundant regions. The one with the higher confidence is retained, while other duplicate or meaningless candidate regions are eliminated, achieving temporal optimization of candidate ROIs. Finally, the temporally optimized effective feature vectors are input into a fully connected layer, which outputs the spatial coordinate data of the ROI region and its corresponding object category confidence. The fully connected layer is a network layer in which all neurons are fully connected to the neurons in the previous layer. Its core function is to map high-dimensional contextual feature vectors into specific output results. Spatial coordinate data refers to the precise location information of the ROI region in the video frame. With the top-left corner of the video frame as the origin, the coordinates of the top-left and bottom-right corners of the ROI are output to ensure accurate location of the region of interest. The object category confidence is the model's probability judgment of the category to which the target within the ROI region belongs (values ​​from 0 to 1). The closer the value is to 1, the more reliable the category judgment. The specific output process is as follows: the fully connected layer converts the temporally optimized feature vector into two types of outputs. One type is the spatial coordinate data output by the regression layer, which is normalized and inversely operated (restored to the original pixel coordinates of the video frame). The other type is the probability distribution of each category output by the classification layer. The maximum probability is taken as the object category, and the corresponding probability is the confidence of the category. The spatial coordinate data is bound to the confidence of the object category to form the recognition result signal corresponding to each effective ROI. This signal is aligned with the subsequently extracted feature signal, providing a basis for the alignment operation of signal processing unit 3. The entire process ensures real-time performance through a lightweight model, and the combination of multi-scale scanning and anchor boxes ensures full coverage of candidate ROIs. The spatial pyramid pooling layer unifies the feature dimension, and the bidirectional LSTM removes temporal redundancy. The fully connected layer outputs accurate results. It not only fully discloses the core technical details of AI visual recognition, but also ensures the balance between recognition accuracy and real-time performance through a progressive logic. It perfectly connects the video acquisition and signal processing links and provides standardized input for the subsequent alignment and comprehensive signal generation of the two video streams.

[0037] After the AI ​​visual recognition unit 2 outputs the spatial coordinate data of the ROI region and its corresponding object category confidence score through the fully connected layer, in order to ensure that the feature signal simultaneously contains both location information and visual attribute information, and achieves accurate association with the recognition result signal, it is necessary to complete the construction of the feature signal vector and the generation of signal pairs. The core is to ensure the standardization and homogeneity alignment of the signals through normalization, feature extraction, dimension fusion, and signal filtering. The specific implementation method is as follows:

[0038] First, the spatial coordinate data is converted into a normalized bounding box tensor sequence. The spatial coordinate data is the original position information of the ROI region output by the fully connected layer within the video frame, with the top-left corner of the video frame as the origin, containing the pixel coordinates of the top-left corner (x1, y1) and bottom-right corner (x2, y2) of the ROI. Normalization converts the original pixel coordinates into dimensionless values ​​between 0 and 1, aiming to eliminate scale differences caused by different video frame resolutions and ensure feature consistency. The bounding box tensor sequence is a multidimensional array of normalized coordinate data organized in a fixed format, which can be directly processed by subsequent network layers. The specific conversion process is as follows:

[0039] First, obtain the width W and height H of the current video frame. Divide the original coordinates x1 and x2 by W, and y1 and y2 by H, respectively, to obtain four normalized coordinate values ​​(all between 0 and 1). Then, arrange these four values ​​in the order x1, y1, x2, y2 to form a 1×4 one-dimensional tensor. If the same video frame contains multiple ROIs, sort them from highest to lowest confidence level, forming a 1×4×N (N is the number of ROIs) bounding box tensor sequence to ensure the structured and standardized location features. While standardizing the location features, extract the YUV color space histogram statistics of the pixels within the ROI region. The YUV color space is a color encoding method that separates the brightness (Y channel) and chroma (U channel, V channel) of an image. Compared to the RGB color space, it is more consistent with human visual characteristics and can reduce the redundancy of chroma channels, improving feature extraction efficiency. The histogram statistics are quantitative indicators obtained by statistically analyzing the pixel value distribution of each color channel, reflecting the color distribution characteristics of the ROI region. The specific extraction process is as follows:

[0040] First, each pixel within the ROI region is converted from the RGB color space to the YUV color space, obtaining the Y, U, and V channel values ​​for each pixel. Then, each channel is divided into statistical intervals (25 intervals for the Y channel, and 16 intervals each for the U and V channels, balancing statistical accuracy and computational efficiency). The number of pixels within each interval is counted, resulting in 25 statistical values ​​for the Y channel and 16 statistical values ​​each for the U and V channels. These 57 statistical values ​​are arranged in the order of Y channel—U channel—V channel, forming a one-dimensional color histogram statistic, fully characterizing the color features of the ROI. Next, the bounding box tensor sequence and the YUV color space histogram statistic are concatenated into a dimension-fused feature signal vector. This feature signal vector is a high-dimensional vector that integrates the ROI's location and color features, comprehensively reflecting the core attributes of the ROI and providing rich feature support for subsequent signal processing. The specific concatenation process is as follows:

[0041] First, the bounding box tensor sequence is expanded into a one-dimensional vector. Then, it is concatenated sequentially with a one-dimensional color histogram statistic. For example, the bounding box tensor sequence of four ROIs is expanded into a 1×16 vector, which is then concatenated with a 1×57 histogram statistic to form a 1×73 high-dimensional feature signal vector. Regardless of the number of ROIs in the same video frame, the vector length is uniformly standardized to a fixed value through zero-padding or truncation to ensure the dimensionality consistency of the feature signal vector and meet the requirements of subsequent signal processing. Finally, the recognition result signal is processed to form signal pairs aligned with the feature signal vector. The recognition result signal contains the ROI object category and its corresponding confidence level. The object category confidence threshold is a critical value used to filter out low-reliability recognition results, aiming to eliminate ROI signals with insufficient confidence and unreliable recognition results. The specific process is as follows:

[0042] Each ROI is evaluated to determine whether its object category confidence score is higher than a preset threshold. ROIs with confidence scores below the threshold are discarded along with their corresponding feature signal vectors and recognition results. ROIs with confidence scores above the threshold retain their recognition results (including object category and confidence score) and are appended with a millisecond-level timestamp (generated by a unified system clock source to record the precise moment of ROI recognition). Finally, the retained feature signal vectors and the timestamped recognition results are matched one-to-one to form homologous aligned signal pairs of feature signal vectors and recognition results. This ensures that each set of feature signals can find a corresponding recognition result, providing a precise association basis for the alignment operation of subsequent signal processing unit 3. The entire process eliminates scale differences through normalization, extracts visual features through color histograms, achieves multi-attribute fusion through dimensional stitching, and ensures signal alignment through threshold filtering and timestamp marking. It fully discloses the core technical details of feature construction and signal association, and ensures the integrity and consistency of the signal through a progressive logic, perfectly connecting ROI recognition and signal alignment processing, laying a standardized feature foundation for real-time signal fusion of two video streams.

[0043] After the AI ​​visual recognition unit 2 outputs the feature signal vectors aligned with the same source—the recognition result signal pair, the signal processing unit 3 first needs to solve the temporal alignment problem of the two feature signals, and then generate a comprehensive signal matrix that can reflect the complete information of the physical event through denoising and fusion. The core is to rely on a unified time reference, accurately match inflection points, smooth denoising, and focus on key features to ensure that the spatial and temporal correlation of the two signals is not lost. The specific implementation method is as follows:

[0044] First, a timeline is established based on the physical event trigger moment. The physical event trigger moment refers to the precise moment when the same actual physical event occurs in both video streams. This is determined by identifying abrupt changes in the confidence level of the object category in the identified signal or a waveform jump point in the feature signal. For example, when an object enters both video frames from nothing, the corresponding confidence level jumps from below the threshold to above the threshold; this moment is the physical event trigger moment. The timeline is a time reference system with this trigger moment as the origin (time 0), divided into millisecond increments forward (before the event) and backward (after the event). The system maps all sample points of the two feature signals to this unified timeline according to their timestamps, eliminating the baseline difference in the original timestamps of the two signals and laying the foundation for subsequent alignment. Based on the unified timeline, a dynamic time warping algorithm is used to match the key waveform inflection points of the two feature signals. The dynamic time warping algorithm is a dynamic optimization algorithm specifically designed to solve the problems of inconsistent lengths and time offsets in timing signals. Its core is to find the optimal matching path between the two signals by elastically stretching or compressing the time dimension of one of the signals, avoiding inflection point misalignment caused by rigid alignment. Key waveform inflection points are the core nodes in the characteristic signal that reflect changes in physical events, such as peaks, valleys, and abrupt changes in slope. These inflection points directly correspond to key state changes in physical events. The specific matching process is as follows:

[0045] First, all key waveform inflection points of the two feature signals on a unified time axis are extracted, and the time coordinate and signal amplitude of each inflection point are recorded. Then, the cumulative distance cost of the two inflection point sequences is calculated using a dynamic time warping algorithm to find the inflection point matching path with the minimum cost. For example, the third peak inflection point of the first signal is matched with the fourth peak inflection point of the second signal. Finally, the time coordinate of one of the signals is adjusted according to the matching path to align the corresponding key waveform inflection points on the time axis, completing the initial timing calibration of the two feature signals. After further correcting the alignment deviation through timing offset compensation, a Gaussian kernel function is used to locally weighted smooth the aligned feature signals. The timing offset compensation is the smoothing timing offset output by the timing offset compensation unit 4, which is used to fine-tune the initially aligned signal to ensure that the feature points of the same physical event are completely synchronized. The Gaussian kernel function is a weighting function with a Gaussian distribution. Its core characteristic is that the weight of the signal sample point is normally distributed with the distance from the point, that is, the closer the distance, the greater the weight, and the farther the distance, the smaller the weight. It can smooth the signal while preserving the waveform trend of the core features. Local weighted smoothing is a method of eliminating high-frequency noise by using a Gaussian kernel function to perform a weighted average of each signal sample point and its surrounding local sample points. High-frequency noise refers to rapid, irregular, small-amplitude fluctuations in the signal caused by minute sensor fluctuations, environmental interference, etc. This type of noise can mask the true characteristics of physical events. The specific smoothing process is as follows:

[0046] The local window size of the Gaussian kernel function is preset (experimentally calibrated to be 5 adjacent sample points). The weight of each sample point within the window is calculated (the weight of the central sample point is the largest, gradually decreasing towards both sides). The signal value of each sample point within the window is multiplied by its corresponding weight, summed, and then divided by the total weight to obtain the smoothed signal value of that sample point. This operation is performed point by point along the time axis to generate two smoothed feature signals, which eliminates high-frequency noise while fully preserving the features of key waveform inflection points. Finally, the two smoothed signals are input into a feature fusion engine based on an attention mechanism to generate a comprehensive signal matrix containing spatial-temporal correlations. The feature fusion engine based on an attention mechanism is a fusion module that can automatically focus on important features and weaken secondary information. The attention mechanism calculates the importance weights of each feature dimension of the two signals, allowing the engine to prioritize high-weight features during fusion, thus improving the relevance of the fusion results. Spatial-temporal correlations refer to the comprehensive relationship that simultaneously includes the spatial location correlation and temporal change correlation of the two signals, which can comprehensively depict the spatiotemporal evolution of physical events. The integrated signal matrix is ​​a three-dimensional matrix that organizes the fused features according to the time dimension, spatial dimension, and feature dimension, facilitating the subsequent extraction of complete features of physical events. The specific fusion process is as follows:

[0047] First, the two smoothed feature signals are converted into feature matrices of the same dimensions (the time dimension is the number of scales on a unified time axis, the spatial dimension is the number of coordinate features of the ROI, and the feature dimension is the number of attribute features such as color and texture). Then, the weight of each feature dimension is calculated through an attention mechanism, for example, feature dimensions with higher confidence in the object category are assigned higher weights. Next, the corresponding feature dimensions of the two signals are weighted and summed according to their weights to achieve feature fusion. Finally, the fused features are reorganized according to the three-dimensional structure of time-space-feature to generate a comprehensive signal matrix. Each element in the matrix contains the fused value of a certain time, a certain spatial location, and a certain feature attribute, thus completely preserving the spatial temporal correlation information of the physical event. The entire process ensures the foundation of timing alignment through a unified time base, achieves precise matching of key feature inflection points using a dynamic time warping algorithm, eliminates interference noise with Gaussian kernel smoothing, and focuses on core features to complete fusion by relying on an attention mechanism. It not only fully discloses the core technical details of signal alignment, denoising and fusion, but also ensures that the comprehensive signal matrix can fully and accurately reflect the complete state of physical events through a progressive logic, perfectly connecting the signal generation and feature point matching stages, and providing high-quality structured input for the subsequent accurate signal processing of the system.

[0048] During the real-time monitoring of the computation time of the AI ​​visual recognition unit 2 by the timing deviation compensation unit 4, in order to accurately quantify the fluctuation of computation time and avoid misjudgment caused by abnormal computation time in a single frame, a closed-loop process of multi-frame statistics, benchmark calibration, normalization and smoothing is required to calculate a stable computation time fluctuation value, providing a reliable basis for subsequent determination of timing offset. The specific implementation method is as follows:

[0049] First, a sliding time window is formed by continuously recording the actual computation time of N consecutive frames. The actual computation time is the time difference between the start timestamp when the AI ​​visual recognition unit 2 starts single-frame ROI recognition and the end timestamp when recognition is completed, directly reflecting the computational efficiency of single-frame recognition. The sliding time window is a dynamic data set composed of the actual computation time of N consecutive frames. The value of N was experimentally calibrated to 10 frames (which can cover short-term fluctuations without causing a decrease in real-time performance due to excessive data volume). The specific recording process is as follows: the timing deviation compensation unit 4 allocates an independent time recording buffer for each frame of video stream. When the AI ​​visual recognition unit 2 outputs the actual computation time of a frame, the system immediately writes the value into the buffer. When the data in the buffer reaches N frames, the oldest stored frame is automatically deleted for each new frame of time value added, always keeping the buffer containing the latest N consecutive frames of time data, forming a dynamic sliding time window to ensure the timeliness of statistical data. Based on a sliding time window, the arithmetic mean of the actual computation time within the window is calculated in real time as the dynamic baseline time. The dynamic baseline time is a reference value reflecting the recent average computational efficiency of the AI ​​visual recognition unit 2. Its core feature is that it is adjusted in real time as the sliding time window is updated, avoiding the problem that a fixed baseline cannot adapt to changes in computational efficiency. The specific calculation process is as follows: The system performs a summation operation on the actual computation time values ​​of N frames within the sliding time window, divides the sum by the number of frames N, and the result is the dynamic baseline time corresponding to the current window; for example, if the sum of the time values ​​of 10 frames within the window is 50 milliseconds, the dynamic baseline time is 5 milliseconds. After each update of the sliding time window (adding a frame, deleting a frame), the summation and division operation are re-executed to update the dynamic baseline time in real time, ensuring that the baseline value can accurately track the recent trend of computational efficiency changes. Then, the difference between the actual computation time value of the current frame and the dynamic baseline time is calculated and divided by the dynamic baseline time. The core of this step is to transform absolute fluctuations into relative fluctuations, eliminating the influence of the baseline time magnitude on the fluctuation judgment. The specific process is as follows: First, extract the actual calculation time value of the latest frame in the sliding time window as the current frame time value. Subtract the dynamic baseline time value corresponding to the current window from this value to obtain the absolute difference between the two (if the current frame time value is higher than the baseline, it is a positive value; if it is lower, it is a negative value, which intuitively reflects the direction and absolute amplitude of the fluctuation). Then, divide the absolute difference by the dynamic baseline time value. The resulting ratio is the normalized fluctuation coefficient. The normalized fluctuation coefficient is a relative fluctuation indicator that eliminates the influence of data magnitude. For example, if the current frame time value is 6 milliseconds, the dynamic baseline time value is 5 milliseconds, and the absolute difference is 1 millisecond, the fluctuation coefficient is 0.2 (i.e., 20%). Regardless of whether the baseline time value is 5 milliseconds or 10 milliseconds, the relative severity of the fluctuation can be intuitively judged by the size of the coefficient.Finally, the fluctuation coefficient is noise-suppressed using an exponentially weighted moving average algorithm, outputting a stable time-consuming fluctuation value. The exponentially weighted moving average algorithm is a smoothing algorithm that assigns higher weights to recent data and lower weights to older data, effectively filtering out random noise in the fluctuation coefficient (such as abnormal fluctuations caused by sudden hardware scheduling delays in a single frame) while preserving the core fluctuation trend. Noise suppression is achieved by smoothing out irregular random fluctuations through the algorithm, preventing such fluctuations from interfering with the calculation of subsequent time-series offsets. The time-consuming fluctuation value is the final output, smoothed, stable fluctuation quantification result, and the specific process is as follows:

[0050] A weighting factor is preset, and the smoothing result of the first frame is initialized as the fluctuation coefficient of that frame. Starting from the second frame, the smoothing result of each frame is equal to the weighting factor × the current frame fluctuation coefficient + (1 - weighting factor) × the previous frame smoothing result. Through this iterative calculation, random noise is gradually filtered out, making the output result tend to be stable. For example, if the current frame fluctuation coefficient is 0.2 and the previous frame smoothing result is 0.15, the current frame smoothing result is 0.3 × 0.2 + 0.7 × 0.15 = 0.165, which is the time fluctuation value of the current frame. This preserves the true fluctuation trend and eliminates the interference of random noise. The entire process captures short-term fluctuation patterns through a sliding time window, adapts to changes in computational efficiency with a dynamic benchmark time, transforms it into relative fluctuation using normalization, and smooths noise using an exponentially weighted moving average algorithm. This fully discloses the core technical details of the time fluctuation value calculation and ensures the stability and reliability of the results through a layer-by-layer progressive logic. It perfectly connects the calculation time recording and the determination of timing offset, providing a precise quantitative basis for timing deviation compensation of two-channel signal processing.

[0051] After obtaining a stable time-consuming fluctuation value, the timing deviation compensation unit 4 needs to convert this normalized fluctuation quantization result into a timing offset that can actually be used for signal phase adjustment. The core is to generate an initial offset through basic transformation, and then combine it with dynamic correction of the fluctuation change rate to ensure that the offset can both match normal fluctuations and cope with sudden fluctuations. The specific implementation method is as follows:

[0052] First, the time fluctuation value is multiplied by a preset system clock cycle conversion coefficient to generate the raw time offset. The system clock cycle conversion coefficient is a pre-calibrated proportional parameter used to convert the normalized fluctuation coefficient into actual time units. Its value is equal to the reciprocal of the clock cycle of the system core processor and the signal sampling frequency. After experimental calibration, it is fixedly stored in the system configuration file. The purpose is to convert the dimensionless fluctuation coefficient into a time offset with actual physical meaning. The raw time offset is a basic offset calculated only based on the time fluctuation value of the current frame, reflecting the degree of timing deviation under normal fluctuations. The specific process is as follows:

[0053] The system clock cycle conversion coefficient is read from the configuration file. The output stable time fluctuation value is directly multiplied by this coefficient. For example, if the time fluctuation value is 0.15 and the conversion coefficient is 20, the original time offset is 3 milliseconds. This calculation process is executed quickly by a hardware multiplier to ensure real-time performance. The generated original time offset is temporarily stored in a temporary register for subsequent correction. Simultaneously with the generation of the original time offset, the rate of change of fluctuation values ​​in adjacent frames is monitored in real time. The rate of change of fluctuation values ​​in adjacent frames is an indicator reflecting the speed of change of the time fluctuation value, used to determine whether the fluctuation is abrupt and to avoid the original time offset failing to adapt in time due to sudden fluctuation changes. The specific detection process is as follows:

[0054] The system tracks the current frame number using a frame counter, and retrieves the stable time fluctuation values ​​between the current and previous frames in real time. The fluctuation value of the current frame is subtracted from the fluctuation value of the previous frame to obtain the fluctuation difference. This difference is then divided by the fluctuation value of the previous frame; the resulting ratio is the fluctuation change rate. For example, if the fluctuation value of the current frame is 0.25 and the previous frame's is 0.1, the difference is 0.15, and the change rate is 1.5. This calculation process is performed in parallel with the generation of the original time offset and does not affect the system's real-time performance. When the fluctuation change rate exceeds an adaptive threshold, a secondary compensation mechanism is immediately activated. The adaptive threshold is a dynamically adjusted critical value based on historical fluctuation change rates, rather than a fixed value. Its calibration logic is as follows:

[0055] The system continuously tracks the rate of change of fluctuation values ​​over the past 50 frames, calculates the 95th percentile of these rates (meaning only 5% of historical rates of change exceed this value), and uses this as the adaptive threshold for the current frame. This threshold is recalculated for each updated frame to ensure it adapts to different fluctuation scenarios and avoids false triggers or missed triggers caused by fixed thresholds. The secondary compensation mechanism is a supplementary compensation logic for scenarios with sudden fluctuation changes. Its core is to add an extra compensation term to the original time offset, preventing a single basic compensation from being insufficient to offset the timing deviations caused by sudden fluctuations. The specific activation process is as follows:

[0056] The system compares the rate of change of the fluctuation value in the current frame with an adaptive threshold in real time. If the rate of change exceeds the threshold, it is determined to be a sudden fluctuation, and the secondary compensation module is immediately activated. The module automatically reads the original time offset and the current rate of change from the temporary register to prepare for subsequent corrections. If the rate of change does not exceed the threshold, the mechanism is not activated, and only the original time offset is retained. After the secondary compensation mechanism is activated, the product of the original time offset and the rate of change of the current frame is added based on the time offset of the previous frame. The time offset of the previous frame is the final time offset obtained from the previous complete calculation (which may be the original time offset or the composite time offset), stored in a dedicated register as the benchmark for this compensation, ensuring the continuity of the time offset and avoiding fluctuations caused by starting from zero in each calculation. The product term is the product of the original time offset of the current frame and the rate of change of the fluctuation value. Its function is to dynamically adjust the magnitude of the additional compensation according to the severity of the fluctuation change. The larger the rate of change, the more additional compensation. The specific superposition process is as follows:

[0057] First, the timing offset of the previous frame is read from the register. Then, the product of the original timing offset of the current frame and the rate of change of the fluctuation value is calculated. Subsequently, the timing offset of the previous frame is added to this product term. If the secondary compensation mechanism is not activated, the original timing offset is directly used as the supplement to the current baseline, without the need to add the product term. Finally, through the above basic transformation and dynamic superposition calculation, a composite timing offset for phase adjustment is generated. The composite timing offset is the final timing offset that integrates conventional fluctuation compensation and sudden fluctuation correction. It includes basic compensation based on stable fluctuations and secondary compensation to deal with sudden fluctuation changes, ensuring that the timing offset can accurately match the computation time changes of AI visual recognition unit 2. For example, if the timing offset of the previous frame is 2 milliseconds, the current original offset is 3 milliseconds, and the product term is 4.5 milliseconds, the composite timing offset is 6.5 milliseconds; if the secondary compensation is not activated, the composite timing offset is 3 milliseconds. The offset is written to the output register in real time and sent to the signal processing unit 3, providing a precise time reference for the phase adjustment of the characteristic signal, perfectly connecting the calculation of the time fluctuation value and the signal phase calibration, and ensuring the timing synchronization accuracy of the two signals.

[0058] After generating the composite timing offset, in order to avoid the ringing effect (i.e., frequent positive and negative fluctuations in the offset, affecting the stability of the signal phase adjustment) caused by sudden changes in the fluctuation value, the timing deviation compensation unit 4 needs to further optimize the offset output through a closed-loop process of double buffer storage, status monitoring and phase correction to ensure its smoothness and stability. The specific implementation method is as follows:

[0059] First, a double-buffered timing offset register is established to alternately store the calculation results of the current frame and the previous frame. The double-buffered timing offset register is a storage module composed of two independent registers (Register A and Register B). Its core advantage is read-write separation and alternating operation, avoiding conflicts between storage and read operations and ensuring the real-time and continuous nature of data updates. The calculation result of the current frame is the generated composite timing offset, and the calculation result of the previous frame is the smoothed timing offset (or initial composite timing offset) output after the last optimization. The specific storage process is as follows: The system presets register working status indicators (idle / occupied). Initially, Register A is occupied (storing the first frame's composite timing offset), and Register B is idle. When the second frame's composite timing offset is generated, the system writes it to the idle Register B, while simultaneously reading the previous frame's data from the occupied Register A for subsequent calculations. When the third frame is generated, Register A becomes idle, writes the third frame's data, and reads the second frame's data from Register B, and so on. It always alternates between writing the current frame to an idle register and reading the previous frame's data from another register, ensuring synchronous acquisition of the current and previous frame's data without delay or conflict. While using double buffering, a finite state machine (FSM) monitors the transition direction of the fluctuation rate of change. A FSM is a control module that uses a preset set of states and state transition rules to perform logical judgments and monitoring. Here, three states are preset: positive transition state (current frame fluctuation rate of change is positive), negative transition state (current frame fluctuation rate of change is negative), and stable state (rate of change is close to 0). The transition direction of the fluctuation rate of change refers to the state transition direction between the current frame and the previous frame, and its core function is to capture abrupt changes in fluctuation patterns. The specific monitoring process is as follows: the FSM reads the fluctuation rate of change between the current frame and the previous frame from the double buffer register in real time, judges and records the state (positive, negative, stable) of each frame; then it compares the current frame state with the previous frame state to determine the transition direction. For example, if the previous frame is a positive transition state and the current frame is a negative transition state, the transition direction is positive → negative; if two consecutive frames have the same state, the transition direction remains unchanged. Through this continuous monitoring, sudden changes in the fluctuation rate of change are accurately captured. When a series of positive and negative jumps are detected, the phase correction factor is immediately activated. A series of positive and negative jumps refers to two consecutive reverse transitions in the rate of change of the fluctuation value, either positive→negative→positive or negative→positive→negative. This situation implies sudden and irregular fluctuations in computation time, which can easily lead to frequent fluctuations in the composite time series offset and trigger a ringing effect. The phase correction factor is a preset smoothing coefficient vector used to suppress the ringing effect. Its values ​​follow a Gaussian distribution (large values ​​in the middle and small values ​​at both ends). After experimental calibration, it is stored in the system, and its core function is to offset irregular fluctuations through weighted smoothing.The specific activation process is as follows: The finite state machine continuously counts the transition directions. When two consecutive reverse transitions are detected (e.g., positive → negative, then negative → positive), it is determined to be a continuous positive-negative jump, and the correction factor activation signal is immediately triggered. The system reads the preset phase correction factor from the storage module to prepare for subsequent convolution operations. If no continuous positive-negative jump is detected, the correction factor is not activated, and the composite timing offset directly enters the output stage to avoid over-correction. After the phase correction factor is activated, it is convolved with the composite timing offset to generate a smooth timing offset output with anti-ringing effect. The convolution operation is an operation that multiplies the phase correction factor (weight vector) and the composite timing offset (data vector) at corresponding positions and then sums them. The core is to smooth the abrupt part of the composite timing offset through the weighting effect of the correction factor, preserving the overall trend. Anti-ringing effect refers to eliminating irregular high-frequency fluctuations (i.e., ringing) in the offset through convolution operation, making the output offset change stable and avoiding repeated oscillations during signal phase adjustment. The smoothed temporal offset is the final offset optimized by convolution operations, which combines accuracy and stability. The specific process is as follows:

[0060] The composite temporal offset of the current frame is arranged into a one-dimensional data vector in chronological order, and the phase correction factor is used as the weight vector, ensuring that both have the same length (calibrated to be 5 data points each). Then, according to the convolution rule, the weight vector is multiplied by the corresponding elements of the data vector, and all the product results are summed to obtain the smoothed offset value. For example, if the composite temporal offset vector is [4,6,3,5,7] and the phase correction factor vector is [0.1,0.2,0.4,0.2,0.1], the smoothed offset after the convolution operation is 4×0.1+6×0.2+3×0.4+5×0.2+7×0.1=4.7. This operation is executed quickly by the hardware convolutioner to ensure real-time performance. The final output smoothed temporal offset is sent to the signal processing unit 3 to provide a stable and reliable time reference for the phase calibration of the feature signal. The entire process achieves conflict-free synchronous reading and writing of data through double-buffered registers, accurately captures the fluctuation transition rules with a finite state machine, and suppresses the ringing effect by using phase correction factors and convolution operations. It not only fully discloses the core technical details of offset smoothing optimization, but also ensures the stability and accuracy of the output offset through layer-by-layer progressive logic. It perfectly connects the generation of composite timing offset and signal phase adjustment, laying a solid foundation for the accurate synchronization of the characteristic signals of the two video streams.

[0061] After the timing deviation compensation unit 4 outputs a smooth timing offset to combat ringing, the signal processing unit 3 needs to perform precise phase calibration on the characteristic signal stream based on this offset. The core of this process is to ensure signal continuity through buffer storage, locate the offset position through index conversion, correct timing deviation through waveform reconstruction, and preserve the original features through spectrum fusion. This ensures that the calibrated signal is synchronized with the other signal without losing its own core information. The specific implementation method is as follows:

[0062] First, a circular buffer is set up within signal processing unit 3 to store the feature signal stream. The circular buffer is a contiguous memory space employing a first-in, first-out (FIFO) circular storage mechanism. Its core advantages are that it can cyclically overwrite old data, preventing fragmentation, and allowing read and write operations to be performed in parallel, perfectly meeting the real-time continuous transmission requirements of the feature signal stream. The feature signal stream is a generated sequence of dimensionally fused feature signal vectors containing ROI location and color features, transmitted continuously in the order of video frames. The specific storage process is as follows:

[0063] The system has a preset storage capacity of a circular buffer (experimentally calibrated to 1000 frames of feature signals to ensure sufficient historical data to handle sudden delays). The buffer is virtually connected end-to-end to form a closed loop. When the feature signal stream is continuously input, it is written sequentially to consecutive memory units in the buffer. When the write position reaches the end of the buffer, it automatically jumps to the beginning position to overwrite the earliest stored data. At the same time, the buffer is equipped with independent read and write pointers. The write pointer follows the new data write position in real time, and the read pointer reads the corresponding data according to the phase calibration requirements. The two do not interfere with each other, ensuring the real-time performance of signal storage and processing. Based on the feature signal stream stored in the circular buffer, the index offset of the signal sample point is calculated according to the received smoothed timing offset. The smoothed timing offset is the time offset output by the timing deviation compensation unit 4, which eliminates the ringing effect and reflects the magnitude of the timing deviation between the two signals. The signal sample point is the smallest data unit after the feature signal stream is discretized at a fixed sampling frequency. Each sample point corresponds to a fixed time node and feature value. The index offset is the offset value of the smoothed timing offset in the time dimension converted into the sample point position dimension, used to locate the corresponding position of the calibrated sample point in the original signal stream. The specific calculation process is as follows: First, obtain the preset sampling frequency of the characteristic signal stream (calibrated to 1000 Hz, i.e., 1000 sample points per second). The reciprocal of the sampling frequency is the time interval (1 millisecond) for each sample point. Then, multiply the smoothed timing offset (time value) by the sampling frequency to obtain the index offset (number of sample points). For example, if the smoothed timing offset is 3 milliseconds, the index offset is 3, meaning that the current signal sample point needs to be shifted backward by 3 index positions to achieve timing alignment with another signal. After the index offset is determined, the Lagrange interpolation algorithm is used to reconstruct the offset signal waveform. The Lagrange interpolation algorithm is a polynomial interpolation method that calculates the value of the interpolation point at any position using known values ​​of adjacent sample points. Its core advantage is that it can accurately fit the signal value at the offset position using a small number of adjacent sample points, avoiding waveform distortion. The reconstructed offset signal waveform refers to obtaining the calibrated value for each sample point that needs to be offset through interpolation calculation, and then reassembling it in time sequence to form a complete signal waveform. The specific process is as follows: For each target sample point to be reconstructed (i.e., the offset index position), the system extracts multiple adjacent original sample points before and after the position from the circular buffer (experimentally calibrated to 4 adjacent points, balancing computational accuracy and efficiency); then, according to the Lagrange interpolation formula, the weight of each adjacent sample point to the target sample point is calculated (the weight is inversely proportional to the distance from the sample point to the target position, the closer the distance, the greater the weight); the feature value of each adjacent sample point is multiplied by the corresponding weight and summed to obtain the reconstructed feature value of the target sample point; the interpolation operation is performed on all sample points one by one in chronological order, and finally a complete reconstructed signal waveform after offset is formed, ensuring that the waveform is continuous and smooth, without discontinuities or distortions.During waveform reconstruction, the original signal spectral characteristics are preserved through a bypass channel. This bypass channel is an independent data channel running parallel to the main waveform reconstruction channel. Its core function is to replicate the original characteristic signal stream in real time and extract key information without interfering with the main channel's processing flow, thus avoiding the loss of the original signal's frequency domain characteristics during reconstruction. The original signal spectral characteristics are the frequency domain distribution information obtained after the original characteristic signal stream undergoes a Fourier transform. They include the signal's frequency components and the amplitude corresponding to each frequency, reflecting the signal's essential characteristics. The specific process is as follows:

[0064] When the original feature signal stream is input into the circular buffer, the bypass channel synchronously copies the signal stream and performs a Fast Fourier Transform (FFT) on the copied signal in real time, converting the time-domain signal into a frequency-domain signal. Core spectral features such as peak frequency and frequency amplitude distribution are extracted from the frequency-domain signal and stored in an independent spectral buffer. The timestamps of the spectral buffer and the circular buffer correspond one-to-one, ensuring accurate matching of the reconstructed signal and the original spectrum at the same moment during subsequent fusion. Finally, the reconstructed signal and the original spectrum are fused using an Inverse Fourier Transform (IFT) to generate a new feature signal stream with phase calibration. IFT fusion involves first converting the time-domain reconstructed signal to the frequency domain, performing feature fusion with the retained original spectrum, and then converting it back to the time domain. The core is to correct the spectral deviation of the reconstructed signal at the frequency domain level, ensuring that the fused signal corrects the timing deviation while retaining the essential features of the original signal. The specific fusion process is as follows:

[0065] A Fast Fourier Transform (FFT) is performed on the reconstructed signal to convert it from the time domain to the frequency domain. This frequency domain signal is then weighted and fused with the original spectrum stored in the bypass channel at the corresponding time point (the original spectrum weight is 0.3, and the reconstructed signal's frequency domain weight is 0.7, preserving the original characteristics while highlighting the calibration effect). After fusion, an Inverse Fourier Transform (IFT) is performed on the resulting frequency domain fused signal to convert it back to the time domain. These time domain signals are then organized into a continuous characteristic signal stream in chronological order, which is the new characteristic signal stream for phase calibration. This signal stream corrects the timing deviation with the other signal and preserves the core characteristics of the original signal through spectral fusion, laying a high-quality foundation for further alignment and fusion of the two signals. The entire process ensures signal transmission continuity through a circular buffer, achieves precise mapping of time offset to sample position through index conversion, realizes smooth waveform reconstruction through Lagrange interpolation algorithm, fully preserves the original spectrum through bypass channel, and finally achieves the dual goals of timing synchronization and feature fidelity through frequency domain fusion. It not only fully discloses the core technical details of phase calibration, but also ensures the calibration effect through layer-by-layer logic. It perfectly connects the smooth timing offset input with the similarity comparison of the two signals, so that the synchronization accuracy of the feature signals of the two video streams reaches the practical standard.

[0066] After the signal processing unit 3 generates a new characteristic signal stream for phase calibration, in order to avoid the cumulative residual error from a single phase calibration affecting the synchronization accuracy of subsequent signals, a dynamic optimization mechanism of calibration-monitoring-correction needs to be constructed to form a phase error feedback closed loop. The core is to dynamically adjust the calibration parameters by monitoring the synchronization quality of the two signals in real time, so as to ensure that the phase error is always within an acceptable range. The specific implementation method is as follows:

[0067] A phase error feedback closed loop is constructed. This closed-loop control system consists of a signal monitoring module, a judgment module, a correction module, and an execution module. Its core characteristic is that it uses the error result after phase calibration as input to adjust the key parameters in the calibration process in reverse, forming a dynamic optimization loop to avoid error accumulation caused by open-loop calibration. The specific construction process is as follows:

[0068] A bidirectional data channel is established between the signal processing unit 3 and the timing deviation compensation unit 4. The signal processing unit 3 outputs the synchronization quality index (waveform similarity) of the two new characteristic signal streams in real time. After receiving the index, the timing deviation compensation unit 4 and the signal processing unit 3 adjust the benchmark calculation time value and interpolation algorithm parameters according to the preset rules. The adjusted parameters are then used for the next round of phase calibration, forming a closed-loop link of calibration output - quality monitoring - parameter adjustment - recalibration, ensuring continuous optimization of the calibration process. Based on this closed loop, the waveform similarity of two new feature signal streams at the event trigger time is compared in real time. The new feature signal stream is a feature signal sequence that has been phase-calibrated and generated after inverse Fourier transform fusion. It contains complete features such as the position and color of the ROI, and the timing deviation has been initially corrected. The event trigger time is the precise time when the same physical event occurs in the two video streams. It is determined by the confidence change point in the recognition result signal to ensure that the signal segment corresponding to the same physical event is being compared. The waveform similarity is a quantitative index (value from 0 to 1) that measures the consistency of the waveform shape of the two signals. The closer the value is to 1, the higher the synchronization accuracy. The closer it is to 0, the more significant the phase error. The specific comparison process is as follows:

[0069] First, timestamp matching is used to extract signal segments from two new feature signal streams corresponding to the same event trigger time (the segment length is calibrated to 10 consecutive sample points, balancing comparison accuracy and efficiency). Then, a cosine similarity algorithm is used to calculate the waveform similarity between the two segments. By calculating the cosine of the angle between the feature values ​​of the two signal sample points, the degree of waveform overlap is quantified. For example, if the feature value changes of the two signal segments are completely consistent, the similarity is 1; if the trends are completely opposite, it is 0. The calculation result is output to the judgment module in real time. When the waveform similarity is lower than the dynamic threshold, the timing deviation compensation unit 4 is immediately triggered to recalibrate the baseline calculation time. The dynamic threshold is a critical value dynamically adjusted based on historical waveform similarity data, not a fixed value. Its calibration logic is as follows:

[0070] The waveform similarity after the past 100 phase calibrations is continuously statistically analyzed, and the 80th percentile of these similarities is calculated as the current dynamic threshold (i.e., only 20% of historical similarities are below this value). This threshold is recalculated after each closed-loop monitoring cycle to ensure it adapts to signal synchronization requirements in different scenarios and avoids false triggering caused by a fixed threshold. Recalibrating the benchmark calculation time means that timing deviation compensation unit 4 abandons the currently used dynamic benchmark time and regenerates a benchmark value more suitable for the current calculation efficiency based on the latest calculation time data. The specific process is as follows:

[0071] When the judgment module detects that the waveform similarity is lower than the dynamic threshold, it immediately sends a calibration trigger signal to the timing deviation compensation unit 4. After receiving the signal, the timing deviation compensation unit 4 clears the original sliding time window data, restarts the recording and statistics of the actual calculation time value for N consecutive frames (N is still 10 frames), generates a new dynamic reference time according to the arithmetic mean calculation method, and replaces the original reference value for subsequent timing offset calculation, thus correcting the phase error caused by the reference deviation from the root. While recalibrating the baseline calculation time, the order parameter of the interpolation algorithm is optimized using the gradient descent algorithm to ensure that the phase error of the reconstructed signal converges within a preset tolerance range. The gradient descent algorithm is an optimization algorithm that minimizes the error objective function by iteratively adjusting parameters. Its core is to gradually adjust the parameters along the gradient direction of the error function to achieve rapid error convergence. The order parameter of the interpolation algorithm refers to the order of the Lagrange interpolation algorithm used to reconstruct the signal waveform (initially calibrated to order 4). A higher order results in higher reconstruction accuracy but also a greater computational burden. A low order may lead to distortion of the reconstructed waveform and the generation of phase error. Phase error refers to the time offset error between the reconstructed signal and the ideal synchronization signal, quantifying the degree of timing misalignment between the two signals. The preset tolerance range is an experimentally calibrated acceptable phase error range (0 to 0.5 milliseconds) to ensure that the error does not affect subsequent signal fusion and physical event recognition. The specific optimization process is as follows:

[0072] Using phase error as the objective function and the order parameter of the interpolation algorithm as the optimization variable, the optimization step size is initialized (calibrated to 0.5; a larger step size results in faster parameter adjustment, while a smaller step size leads to more stable convergence). In each iteration, the phase error gradient corresponding to the current order is calculated, and the order parameter is adjusted along the negative gradient direction (if the error is greater than the upper limit of the tolerance interval, the order is appropriately increased; if the error is too small and the computational cost is too high, the order is appropriately decreased). During the iteration process, the phase error corresponding to the adjusted parameters is calculated in real time. When the error falls within the preset tolerance interval, the iteration stops, and the current order parameter is fixed to the optimal value. If the number of iterations reaches the preset upper limit (1... If convergence is not achieved after 0 iterations, the current order is maintained and an alarm is triggered, prompting manual investigation of potential problems. The entire process achieves dynamic optimization of calibration parameters through a phase error feedback closed loop, accurately captures synchronization quality through waveform similarity monitoring, ensures the rationality of judgment through dynamic thresholds, and corrects errors through a two-pronged approach of benchmark recalibration and order parameter optimization, ultimately making the phase error stably converge within the preset range. This fully discloses the core technical details of the feedback closed loop and ensures continuous optimization of synchronization accuracy through a layer-by-layer progressive logic, perfectly connecting phase calibration and comprehensive signal matrix generation, providing long-term guarantee for high-precision synchronization of feature signals of two video streams.

[0073] After ensuring high-precision synchronization of two new feature signal streams through a phase error feedback closed loop, and generating a comprehensive signal matrix containing spatial and temporal correlations via a feature fusion engine, it is necessary to further extract the core features of physical events from the matrix and achieve accurate matching. The core is to ensure that the feature points of the same physical event in the two signals are completely aligned through feature localization, key extraction, bidirectional matching, and backtracking compensation, thus providing support for the accurate identification and analysis of the final physical event. The specific implementation method is as follows:

[0074] First, the spatiotemporal feature vectors of physical events are located within the comprehensive signal matrix. This comprehensive signal matrix is ​​a three-dimensional matrix generated by a feature fusion engine based on an attention mechanism. Its dimensions correspond to the time axis, spatial coordinate dimension, and feature attribute dimension, respectively. It comprehensively records the spatiotemporal evolution and feature association information of physical events in both video streams. The spatiotemporal feature vector of a physical event is a high-dimensional vector extracted from the comprehensive signal matrix that uniquely represents a specific physical event. It includes the event's time coordinates (corresponding to the time axis dimension), spatial location information (corresponding to the spatial coordinate dimension), and core feature attributes (corresponding to the feature attribute dimension). The specific localization process is as follows:

[0075] Preset feature selection criteria for physical events (such as feature amplitude exceeding a set threshold, continuous spatial coordinate changes conforming to physical laws), traverse the comprehensive signal matrix step by step according to the time axis, and simultaneously traverse the feature values ​​corresponding to all spatial coordinates at each time scale; when a feature value of a certain spatial coordinate is detected to meet the selection criteria and forms a coherent feature change sequence within a continuous time scale, it is determined that the sequence corresponds to a physical event. The time, space, and feature data of the sequence in the matrix are extracted in sequence and combined to form the spatiotemporal feature vector of the physical event, ensuring that each physical event can be accurately located and quantified. After location, its local maxima are extracted as key feature points. Local maxima refer to nodes whose feature amplitude is higher than all adjacent points within a local range of the spatiotemporal feature vector. These nodes correspond to the key states of the physical event and are key anchor points representing the core features of the event. Key feature points are core nodes obtained after filtering local maxima and can reflect the essential attributes of the physical event to the greatest extent. The specific extraction process is as follows:

[0076] For each spatiotemporal feature vector, a local sliding window is set (the window size is experimentally calibrated to five consecutive time scales, balancing local feature capture and efficiency). The sliding window traverses the entire spatiotemporal feature vector in chronological order. For each feature point within the window, the amplitude of the central feature point is compared with that of all other adjacent feature points within the window. If the amplitude of the central feature point is the largest, it is marked as a local maximum. After traversal, local maximum points with amplitudes lower than the global feature mean are removed (to avoid false feature points caused by noise). The remaining points are key feature points, and each key feature point records its corresponding time coordinates, spatial coordinates, and feature amplitude. Subsequently, a bidirectional optimal matching channel is established for the two signal feature points. The two signal feature points refer to the sets of key feature points extracted from the comprehensive signal matrices corresponding to the two video streams. Each set contains its corresponding physical event key anchor points. The bidirectional optimal matching channel is a logical link established for the two feature points to mutually find the optimal matching object. The core is to ensure that each feature point in the first channel can find the most similar feature point in the second channel, while each feature point in the second channel can also verify the matching object in the first channel, ensuring the bidirectional validity of the matching. The specific establishment process is as follows: First, calculate the matching similarity between all pairwise feature points in the two feature point sets (using cosine similarity as the metric, combined with the time coordinate difference, spatial coordinate difference, and feature amplitude difference; higher similarity indicates better matching). For each feature point in the first path, select the top 3 candidate feature points with the highest similarity in the second path to form a one-way candidate matching set. Similarly, for each feature point in the second path, select the top 3 candidate feature points with the highest similarity in the first path to form a reverse candidate matching set. The intersection of the one-way and reverse candidate matching sets is taken as the valid candidate pair, constructing a bidirectional association link between the first-path feature points and the second-path feature points, i.e., a bidirectional optimal matching channel, ensuring the reliability of the matching relationship. Based on this channel, the Hungarian algorithm is used to find the matching pair with the minimum path cost. The Hungarian algorithm is an optimization algorithm specifically designed to solve assignment problems, capable of finding the optimal matching combination with the minimum cost between elements of two sets, perfectly suited to the one-to-one matching requirements of two-path feature points. Minimum path cost is a quantitative metric for evaluating the quality of feature point matching. A lower cost indicates a more accurate match. Its value is obtained by inversely transforming the similarity between the two feature points. The specific solution process is as follows:

[0077] First, a cost matrix is ​​constructed. The number of rows in the matrix corresponds to the number of feature points in the first path, and the number of columns corresponds to the number of feature points in the second path. Each element in the matrix represents the path cost corresponding to the two feature points. The cost matrix is ​​then input into the Hungarian algorithm. The algorithm iteratively searches for the optimal assignment scheme, ensuring that each feature point finds a unique matching object and that the total path cost of all matching pairs is minimized. After iteration, the final set of optimal matching pairs is output. Each matching pair corresponds to a key feature point of the same physical event in both signals, achieving precise alignment of core features. For feature points that fail to match (i.e., feature points without a valid matching object in the cost matrix), a backtracking mechanism is immediately initiated. Unmatched feature points are key feature points that cannot find an optimal matching object due to sudden changes in feature amplitude, local noise interference, or incomplete features at the start / end of the event. Failure to address this would result in missing physical event features. The backtracking mechanism is a fallback strategy that completes the matching relationship for unmatched feature points by tracing back historical synchronization information. The core is to deduce the matching object of the unmatched point using verified synchronization points. The specific initiation process is as follows:

[0078] All unmatched feature points are automatically marked, and their corresponding time and spatial coordinates are recorded. Then, a reverse search is performed along the time axis to find the nearest successfully matched synchronization point (i.e., the time node corresponding to the effective matching pair obtained through the Hungarian algorithm). This synchronization point is a key node confirming the alignment of the two signals, possessing a reliable temporal and spatial correlation. After finding the previous synchronization point, interpolation compensation is performed using the obtained temporal offset to forcibly align the coordinates of the missing feature point. The previous synchronization point is the nearest effective matching point found in the reverse search; its coordinates and temporal relationship are completely aligned, serving as the benchmark for compensation calculation. Interpolation compensation is based on the synchronization point. The method of calculating the coordinates of the unmatched point in another signal by combining the timing interval with the timing offset ensures the accuracy of coordinate alignment. The specific process is as follows: First, calculate the time difference between the unmatched feature point and the previous synchronization point, and determine the timing deviation compensation amount within this time difference by combining the timing offset. Then, use the Lagrange interpolation algorithm, with the coordinates of the synchronization point as the reference, to calculate the theoretical coordinates of the unmatched point in the other signal based on the compensation amount and the time difference. Finally, use the theoretical coordinates as the matching coordinates of the unmatched feature point to forcibly establish a matching relationship, ensuring that all key feature points of physical events in the two signals can be fully aligned without omissions or misalignments. The entire process locks the core of physical events by accurately locating spatiotemporal feature vectors, extracts key anchor points by local maxima, establishes reliable associations through bidirectional matching channels, achieves optimal matching using the Hungarian algorithm, and relies on backtracking mechanisms and interpolation compensation to cover missing features. It not only fully discloses the core technical details of feature point matching and alignment, but also ensures that the feature points of the two signals are completely synchronized through a progressive logic, perfectly connecting the generation of the comprehensive signal matrix with the complete identification of subsequent physical events. This allows the system to form a closed loop in the signal processing of the two video streams, and finally outputs accurate and complete physical event feature association results.

[0079] This invention acquires two video streams through a video acquisition unit, and the AI ​​visual recognition unit uses a lightweight convolutional neural network and a bidirectional long short-term memory network to detect and recognize ROIs in real time and output features and recognition signals. The timing deviation compensation unit monitors the fluctuation of computation time and generates a smooth timing offset to compensate for the deviation and resist ringing effect. The signal processing unit generates a comprehensive signal matrix by dynamic time warping, Gaussian kernel smoothing and attention fusion, and achieves accurate alignment of feature points by combining phase error feedback closed loop and Hungarian algorithm.

[0080] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A real-time ROI acquisition and signal processing system for two video streams integrating AI vision, characterized in that, include: The video acquisition unit (1) is used to acquire two video streams from two independent video sources in real time and transmit the two video streams to the AI ​​visual recognition unit (2); the AI ​​visual recognition unit (2) is used to perform real-time ROI detection and recognition on each video frame in the two video streams, and output the feature signal of the ROI region and the corresponding recognition result signal; the signal processing unit (3) is used to perform alignment processing on the feature signal of the ROI region and the corresponding recognition result signal to generate a comprehensive signal of the physical event; the timing deviation compensation unit (4) is used to monitor the calculation time fluctuation value of the AI ​​visual recognition unit (2) in real time, and compensate the two signal processing units based on the calculation time fluctuation value. The timing deviations include recording the start timestamp when the AI ​​visual recognition unit (2) starts ROI recognition for each video frame, recording the end timestamp when recognition is completed, calculating the time difference between the start timestamp and the end timestamp as the actual calculation time value of the video frame, and setting a benchmark calculation time value as a reference. The timing offset is determined by comparing the deviation between the actual calculation time value and the benchmark calculation time value, and the timing offset is sent to the signal processing unit (3) in real time. The signal processing unit (3) adjusts the phase position of the feature signal according to the timing offset during the signal alignment process, so that the feature points of the same physical event in the two signal processing results remain synchronized.

2. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 1, characterized in that: A lightweight convolutional neural network is used as the backbone feature extractor. Multi-scale sliding window scanning is performed on each video frame. Candidate ROI regions are generated through preset object localization anchor boxes. The contextual feature vectors of the candidate ROI regions are extracted through spatial pyramid pooling layers and input into a cascaded bidirectional long short-term memory network for temporal dependency modeling. Redundant candidate regions in adjacent frames are eliminated. Finally, the spatial coordinate data of the ROI region and its corresponding object category confidence are output by a fully connected layer to form the corresponding recognition result signal.

3. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 2, characterized in that: The spatial coordinate data is converted into a normalized bounding box tensor sequence, and the YUV color space histogram statistics of pixels within the ROI region are extracted. The bounding box tensor sequence and the YUV color space histogram statistics are concatenated into a dimension-fused feature signal vector. The recognition result signal is then filtered through an object category confidence threshold and a timestamp is added to form a signal pair that is aligned with the feature signal vector.

4. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 1, characterized in that: A timeline based on the physical event trigger time is established. The key waveform inflection points of the two feature signals are matched by a dynamic time warping algorithm. After the time offset is compensated, the aligned feature signals are locally weighted and smoothed using a Gaussian kernel function to eliminate high-frequency noise. Finally, the two smoothed signals are input into a feature fusion engine based on an attention mechanism to generate a comprehensive signal matrix containing spatial and temporal correlations.

5. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 1, characterized in that: The calculation of the time fluctuation value specifically includes: continuously recording the actual calculation time values ​​of N consecutive frames to form a sliding time window; calculating the arithmetic mean of the actual calculation time values ​​within the sliding time window in real time as the dynamic reference time; dividing the difference between the actual calculation time value of the current frame and the dynamic reference time by the dynamic reference time to obtain the normalized fluctuation coefficient; and using an exponentially weighted moving average algorithm to suppress noise in the fluctuation coefficient and output a stable time fluctuation value.

6. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 5, characterized in that: The determination of the timing offset specifically includes: multiplying the time fluctuation value by a preset system clock cycle conversion coefficient to generate the original timing offset; simultaneously detecting the rate of change of fluctuation values ​​in adjacent frames; and when the rate of change of fluctuation values ​​exceeds an adaptive threshold, initiating a secondary compensation mechanism; using the timing offset of the previous frame as a reference, superimposing the product of the original timing offset of the current frame and the rate of change to generate the final composite timing offset used for phase adjustment.

7. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 6, characterized in that: A double-buffered timing offset register is established to alternately store the calculation results of the current frame and the previous frame. The transition direction of the rate of change of the fluctuation value is monitored by a finite state machine. When a continuous positive and negative jump is detected, the phase correction factor is activated. The phase correction factor is convolved with the composite timing offset to generate a smooth timing offset output with anti-ringing effect.

8. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 7, characterized in that: A ring buffer is set up in the signal processing unit (3) to store the feature signal stream. The index offset of the signal sample point is calculated according to the received smooth timing offset. The Lagrange interpolation algorithm is used to reconstruct the offset signal waveform. At the same time, the original signal spectrum features are retained through the bypass channel. Finally, the reconstructed signal and the original spectrum are fused by inverse Fourier transform to generate a new feature signal stream with phase calibration.

9. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 8, characterized in that: A phase error feedback closed loop is constructed to compare the waveform similarity of two new feature signal streams at the event trigger time in real time. When the similarity is lower than the dynamic threshold, the timing deviation compensation unit (4) is triggered to recalibrate the reference calculation time value. At the same time, the order parameter of the interpolation algorithm is optimized by the gradient descent algorithm so that the phase error of the reconstructed signal converges within the preset tolerance range.

10. The integrated AI vision dual-channel video stream real-time ROI acquisition and signal processing system according to claim 1, characterized in that: In the comprehensive signal matrix, the spatiotemporal feature vector of the physical event is located, its local maximum point is extracted as the key feature point, a bidirectional optimal matching channel for the feature points of the two signals is established, the minimum path cost matching pair is solved by the Hungarian algorithm, a backtracking mechanism is initiated for the feature points that are not successfully matched, the search is reversed along the time axis to the previous synchronization point, the time offset is used for interpolation compensation, and the coordinates of the missing feature points are forcibly aligned.

Images