AI-based moving subject tracking and streaming optimization live broadcast system

Abstract

The application discloses an AI-based sports subject tracking and streaming optimization live broadcast system, and particularly relates to the technical field of sports subject tracking and streaming optimization, wherein AI-driven sports subject detection and tracking and streaming encoding adaptive control are closely integrated, real-time sensing of encoding end code rate fluctuation and network end bandwidth change is realized, a continuous and interpretable live broadcast jitter risk analysis index is constructed, potential picture jitter and delay risk are identified in advance through the fusion of causal rule scoring and time sequence prediction, and the live broadcast jitter risk analysis index is directly mapped into an adaptive encoding and transmission adjustment instruction, so that key area priority guarantee, code rate distribution smoothing and delay control optimization are realized, the live broadcast picture definition and playing fluency are significantly improved, and sudden jitter and lagging phenomena are reduced.

Description

Technical Field

[0001] This invention relates to the field of motion subject tracking and streaming media optimization technology, and more specifically, to an AI-based motion subject tracking and streaming media optimization live streaming system. Background Technology

[0002] In AI-based motion tracking and live streaming optimization methods, key subjects in the frame are typically identified through detection and tracking, and different bitrates are allocated based on the importance of these subjects to ensure clarity in critical areas within limited bandwidth. However, in actual operation, the Region of Interest (ROI) frequently switches with the movement of the subject, causing drastic fluctuations in the bitrate demand for critical frames at the encoding end. If the bandwidth estimation at the network transport layer is lagging or inaccurate, it cannot respond to these sudden bitrate changes in a timely manner. The fluctuations at the encoding end and the lag at the network end interact, forming a strong coupling effect: the dynamic switching of ROI amplifies the pressure on bandwidth adaptation, while inaccurate bandwidth estimation further exacerbates the bitrate imbalance at the encoding end. The combined effect not only causes drastic fluctuations in image clarity within a short period but also leads to a sudden increase in playback latency or even stuttering. Summary of the Invention

[0003] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide an AI-based motion tracking and streaming media optimization live broadcast system to address the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] The AI-based motion tracking and streaming optimization live broadcast system includes a multi-source spatiotemporal data acquisition module, a spatiotemporal data standardization and ROI candidate generation module, an ROI-aware contrastive representation learning module, a task fine-tuning and confidence calibration module, an online time-series image feature inference module, a windowed perturbation feature extraction module, and a risk inference and adaptive coding control module.

[0006] The multi-source spatiotemporal data acquisition module is used to acquire multi-source remote sensing images and their sensor trajectories, attitudes and timestamps to form raw spatiotemporal data streams;

[0007] The spatiotemporal data standardization and ROI candidate generation module is used to perform high-precision geometric correction, cross-modal radiometric uniformity and adaptive cloud and fog removal on the original spatiotemporal data stream, and generate a standardized time-series image set and preliminary region of interest candidates with metadata.

[0008] The ROI-aware contrastive representation learning module is used to construct semantically weighted contrastive learning sample pairs with the standardized temporal image set and the initial region of interest candidates according to the ROI-aware strategy, and drive the contrastive representation learning with training batches to produce pre-trained representation parameters and contrastive queue statistics.

[0009] The task fine-tuning and confidence calibration module is used to perform task fine-tuning and confidence calibration on labeled small samples using pre-trained representation parameters and contrast queue statistics, generating task-sensitive feature extractors and classifiers and outputting confidence calibration parameters.

[0010] The online temporal image feature inference module is used to infer the online temporal image stream frame by frame with a task-sensitive feature extractor and classifier, and generate detection / classification results, region of interest feature trajectories and classification confidence time series, while simultaneously acquiring encoder instantaneous bit rate distribution, key frame records and bandwidth observation samples.

[0011] The windowed perturbation feature extraction module is used to calculate and extract windowed perturbation features with the detection / classification results, region of interest feature trajectory, classification confidence time series and encoder / transmission observation alignment as input, perform temporal smoothing, decomposition and normalization on the perturbation features, and output a normalized windowed perturbation feature vector.

[0012] The risk inference and adaptive coding control module is used to perform risk inference by inputting normalized windowed perturbation feature vectors in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules. The module then merges the outputs of the two modules to generate a continuous live jitter risk analysis index and maps the live jitter risk analysis index to adaptive coding and transmission adjustment instructions.

[0013] In a preferred embodiment, a standardized temporal image set and preliminary region of interest (ROI) candidates are used to construct semantically weighted contrastive learning sample pairs according to an ROI-aware strategy. Contrastive representation learning is then driven by training batches to produce pre-trained representation parameters and contrastive queue statistics, as follows:

[0014] The standardized temporal image set and preliminary region of interest candidates are cropped to generate a multi-scale region of interest segment sequence, and a segment set with location, scale, timestamp and uncertainty annotations is obtained;

[0015] Perform view construction and enhancement using the aforementioned fragment set;

[0016] Construct sample pairs from a weighted collection of views;

[0017] Perform importance-balanced sampling and queue strategy configuration on batches of samples with type and weight annotations;

[0018] Comparative representation optimization is driven by training mini-batch and momentum queue update plans.

[0019] The updated characterization parameters are serialized with the comparison queue statistics and the sampling strategy metadata is saved.

[0020] In a preferred embodiment, task fine-tuning and confidence calibration are performed on labeled small samples using pre-trained representation parameters and contrast queue statistics to generate a task-sensitive feature extractor and classifier, and output confidence calibration parameters, as follows:

[0021] Load pre-trained representation parameters and contrast queue statistics, and aggregate labeled small sample sets and their metadata to form a traceable fine-tuning data package;

[0022] Perform sample balancing and semantic preservation enhancement on traceable fine-tuning data packages according to category / scale / uncertainty and region of interest importance, and generate training / validation splits and enhanced training batches;

[0023] To enhance training batch implementation through phased fine-tuning;

[0024] The model parameters are fine-tuned to predict the validation set sample by sample and the prediction confidence distribution and uncertainty estimate are collected. A lightweight confidence calibrator is trained / fitted based on the distribution to generate confidence calibration parameters and calibration mapping table.

[0025] The confidence calibration parameters are used to map the fine-tuned model output and the calibration effect is evaluated on the validation set. The output of the calibrated model performance metric and the final calibration parameters are then used.

[0026] The calibrated model performance metrics and final calibration parameters are serialized with fine-tuned representation parameters, classifier weights, confidence calibration parameters, and fine-tuning strategy metadata to form a deployable task-sensitive feature extractor and classifier, and output confidence calibration parameters and reproducible fine-tuning package.

[0027] In a preferred embodiment, a task-sensitive feature extractor and classifier perform frame-by-frame inference on the online temporal image stream to generate detection / classification results, region of interest feature trajectories, and classification confidence time series. Simultaneously, the encoder's instantaneous bitrate distribution, keyframe records, and bandwidth observation samples are acquired, as detailed below:

[0028] Receive online time-series video streams and corresponding timestamps, frame IDs and sensor tags to form a frame-by-frame raw input frame stream;

[0029] Perform fast preprocessing on the raw input frame stream frame by frame;

[0030] The system takes a preprocessed image frame stream with frame-level quality labels as input and calls a task-sensitive feature extractor and classifier to infer frame by frame. It outputs the candidate region of interest, the local feature vector of each candidate, the classification probability distribution and confidence estimate for each frame, and adds a trigger mark for inference for low-confidence candidates.

[0031] Temporal correlation and online feature fusion are performed using the candidate set and its local feature vectors for each frame as input.

[0032] Using trajectory entries in the region of interest as input, calculate trajectory-level statistics and anomaly indicators, and output trajectory-level summaries and event signals;

[0033] The instantaneous bit rate distribution of the encoder, keyframe insertion records and bandwidth observation samples are synchronously acquired using trajectory-level summaries and event signals as trigger conditions. The acquired encoder / transmission observations and the trajectory entries of the region of interest within the corresponding time window are precisely aligned and packaged according to timestamps to generate an aligned observation package.

[0034] In a preferred embodiment, the windowed perturbation features are calculated and extracted using the detection / classification results, region of interest feature trajectories, classification confidence time series, and encoder / transmission observation alignment as inputs, as follows:

[0035] Taking the aligned detection / classification results, region of interest feature trajectories, classification confidence time series and encoder / transmission observations as input, the system segments and aligns the timestamps according to a fixed window length and overlap ratio to generate a synchronous windowed observation batch. It also performs time interpolation or forward padding on missing samples within the window and outputs the synchronous windowed observation batch.

[0036] Using the synchronous windowed observation batch as input, the original perturbation index sequence is calculated: the switching frequency of the region of interest, the rate of change of the scale of the region of interest, the trajectory velocity and acceleration sequence, the trajectory break count, the classification confidence oscillation frequency, the instantaneous bit rate allocation skew, the mean of the bandwidth prediction residual, the end-to-end delay jitter and packet loss burst count, and the windowed original perturbation index matrix is ​​output.

[0037] In a preferred embodiment, a normalized windowed perturbation feature vector is input in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules for risk inference. The outputs of the two are fused to generate a continuous live stream jitter risk analysis index, and the live stream jitter risk analysis index is mapped to adaptive coding and transmission adjustment instructions, as follows:

[0038] Normalized windowed perturbation feature vectors are input in parallel into an interpretable scorer and a lightweight time series predictor based on causal rules: the interpretable scorer driven by causal rules decomposes the input into several factor scores according to a predefined causal graph, and performs fast local counterfactual detection for each factor.

[0039] The lightweight time series predictor performs short-term trajectory prediction and event probability estimation on the input, and outputs the perturbation trajectory prediction, event occurrence probability curve and prediction uncertainty for several time steps in the future;

[0040] Based on the outputs of the causal scorer and the time series predictor, an interpretable factor vector and a prediction distribution vector are formed respectively. The factor vector contains the standardized score and counterfactual sensitivity of each perturbation factor; the prediction distribution vector contains the probability of event occurrence and the prediction uncertainty.

[0041] The live streaming jitter risk analysis index is generated by weighting and summing the factor vector and the predicted distribution vector.

[0042] In a preferred embodiment, the live streaming jitter risk analysis index is used as an input and mapped to a multidimensional adaptive control vector.

[0043] The mapping employs both hierarchical thresholding and continuous mapping:

[0044] When the live stream jitter risk analysis index is less than the first threshold of the live stream jitter risk analysis index, conservative optimization is performed;

[0045] When the live stream jitter risk analysis index is greater than or equal to the first threshold and less than the second threshold, a smoothing-first strategy is executed; when the live stream jitter risk analysis index is greater than or equal to the second threshold, a preservation-first strategy is executed; the control vector specifically includes several quantifiable instruction fields.

[0046] The technical effects and advantages of this invention are as follows:

[0047] 1. This invention tightly integrates AI-driven motion detection and tracking with adaptive control of streaming media encoding to achieve accurate identification of key subjects in the picture and dynamic management of the area of ​​interest. At the same time, it senses real-time bitrate fluctuations at the encoding end and changes in network bandwidth, thereby constructing a continuous and interpretable live broadcast jitter risk analysis index. Under complex operating conditions with frequent ROI switching and bandwidth prediction lag, it can identify potential image quality jitter and latency risks in advance through the fusion of causal rule scoring and temporal prediction. The live broadcast jitter risk analysis index is directly mapped to adaptive encoding and transmission adjustment instructions, realizing priority protection of key areas, smoothing of bitrate allocation and optimization of latency control. This significantly improves the clarity and smoothness of live broadcast images, reduces sudden jitter and stuttering, and improves the viewer experience. At the same time, it provides interpretable risk quantification indicators and closed-loop adaptive control, making the system more stable, robust and intelligent in complex dynamic environments. Attached Figure Description

[0048] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0049] Figure 1 This is a flowchart of the system according to an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] Example: The present invention provides, as follows Figure 1 The AI-based motion tracking and streaming optimization live broadcast system shown includes a multi-source spatiotemporal data acquisition module, a spatiotemporal data standardization and ROI candidate generation module, an ROI-aware contrastive representation learning module, a task fine-tuning and confidence calibration module, an online time-series image feature inference module, a windowed perturbation feature extraction module, and a risk inference and adaptive coding control module.

[0052] The multi-source spatiotemporal data acquisition module is used to acquire multi-source remote sensing images and their sensor trajectories, attitudes and timestamps to form raw spatiotemporal data streams;

[0053] The spatiotemporal data standardization and ROI candidate generation module is used to perform high-precision geometric correction, cross-modal radiometric uniformity and adaptive cloud and fog removal on the original spatiotemporal data stream, and generate a standardized time-series image set and preliminary region of interest candidates with metadata.

[0054] The ROI-aware contrastive representation learning module is used to construct semantically weighted contrastive learning sample pairs with the standardized temporal image set and the initial region of interest candidates according to the ROI-aware strategy, and drive the contrastive representation learning with training batches to produce pre-trained representation parameters and contrastive queue statistics.

[0055] The task fine-tuning and confidence calibration module is used to perform task fine-tuning and confidence calibration on labeled small samples using pre-trained representation parameters and contrast queue statistics, generating task-sensitive feature extractors and classifiers and outputting confidence calibration parameters.

[0056] The online temporal image feature inference module is used to infer the online temporal image stream frame by frame with a task-sensitive feature extractor and classifier, and generate detection / classification results, region of interest feature trajectories and classification confidence time series, while simultaneously acquiring encoder instantaneous bit rate distribution, key frame records and bandwidth observation samples.

[0057] The windowed perturbation feature extraction module is used to calculate and extract windowed perturbation features with the detection / classification results, region of interest feature trajectory, classification confidence time series and encoder / transmission observation alignment as input, perform temporal smoothing, decomposition and normalization on the perturbation features, and output a normalized windowed perturbation feature vector.

[0058] The risk inference and adaptive coding control module is used to perform risk inference by inputting normalized windowed perturbation feature vectors in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules, fuse the outputs of the two to generate a continuous live jitter risk analysis index, and map the live jitter risk analysis index into adaptive coding and transmission adjustment instructions.

[0059] Multi-source remote sensing images, along with their sensor trajectories, attitudes, and timestamps, are acquired to form a raw spatiotemporal data stream, as detailed below:

[0060] The system collects multi-source imaging observations, GNSS / IMU observations, star sensor attitude observations, platform dynamics records, trigger pulses, and environmental context to form a raw acquisition package.

[0061] Perform clock unification and time reference calibration on the original acquisition packets (align GNSS PPS / PTP / IRIG with trigger pulses) to generate a unified timestamp observation set;

[0062] The unified timestamp observation set is fused with trajectory and attitude data (multi-constellation GNSS, RTK / PPP, IMU accumulation, star-aware attitude, odometry constraints) and the lever arm and trigger delay are compensated to generate a unified timestamp observation set carrying a high-precision pose trajectory.

[0063] Time-attitude alignment and rolling shutter timing correction were performed using a unified timestamped observation set carrying high-precision pose trajectories to obtain an aligned image sequence and pose annotations for each frame.

[0064] By combining the aligned image sequence and the attitude annotations of each frame with the initial values ​​of the imager's intrinsic / extrinsic parameters and the platform arm parameters, geometric pre-correction is performed to obtain a preliminary georeferenced image sequence.

[0065] By integrating preliminary georeferenced image sequences with atmospheric and environmental context (atmospheric profiles, aerosols, solar zenith angle, topographic relief, and surface albedo a priori), a set of radiation-corrected environmental parameters is generated.

[0066] The radiometrically corrected environmental parameter set is applied to the preliminary georeferenced image sequence to perform radiometric pre-uniformation and exposure compensation, resulting in a radiometrically normalized image sequence.

[0067] Image quality assessment and anomaly detection (blur, strip noise, ghosting, glare, dynamic range compression, motion artifacts) are performed on radiometrically normalized image sequences to generate image sequences with quality masks and frame-level health labels.

[0068] The image sequences with attached quality masks and frame-level health labels are subjected to cloud / haze / fog separation and visibility annotation to obtain effective image sequences;

[0069] Perform cross-source spatial alignment and multi-resolution assimilation (orthorectification, topographic correction, resampling, and color gamut mapping) on ​​the effective image sequences to form a cross-registered image stack;

[0070] By combining the registered image stack with pose trajectory and dynamic records, frame-by-frame localization and attitude uncertainty are estimated, and a spatiotemporal image stack with uncertainty label is output.

[0071] Perform data integrity signing and source identification (hash chain, timestamp evidence, source fingerprint) on the spatiotemporal image stack with uncertainty markers to obtain a traceable original spatiotemporal data stream;

[0072] High-precision geometric correction, cross-modal radiometric homogenization, and adaptive cloud and fog removal are performed on the original spatiotemporal data stream to generate a standardized temporal image set and preliminary region of interest (ROI) candidates along with metadata, as detailed below:

[0073] Receive the raw spatiotemporal data stream and divide it into time window data packets according to the acquisition time window and the sensor source;

[0074] Perform pose and trigger pulse verification on time window data packets based on a unified timestamp, correct time offset, and generate a time-consistent frame sequence;

[0075] Perform coarse registration based on feature matching using a time-consistent frame sequence and high-precision pose, and output a coarsely registered image sequence;

[0076] The coarsely registered image sequence is subjected to high-order geometric fine correction using ground control points (GCP) and digital elevation model (DEM) to eliminate topographic parallax, resulting in a geometrically fine-corrected image sequence and georeferenced markers for each frame;

[0077] Subpixel-level registration is performed between geometrically corrected image sequences and georeferenced markers. Pixel-level alignment is achieved by combining classic features (such as SIFT / ORB) with a deep similarity network, and a registration confidence map and a registered image set are output.

[0078] Based on the registered image set and the sensor spectral response, perform sensor intrinsic / extrinsic parameter calibration and spectral response mapping to generate a spectral response mapping table and a pre-calibrated image set;

[0079] Specialized radiometric preprocessing is applied to the pre-corrected image set: atmospheric correction is performed on optical images (based on physical radiative transfer or learning-based inversion), band response uniformity is performed on multispectral images, speckle suppression and amplitude calibration are performed on SAR images, and a radiometric pre-corrected image set is output.

[0080] Cross-modal radiometric uniformity is performed using a radiometrically pre-corrected image set. A combination of spectral domain mapping, histogram constraints, and learned style transfer (e.g., generative adversarial network style transfer and contrastive normalization) is used to output a normalized surface reflectance image sequence.

[0081] BRDF correction and solar geometry normalization were performed on the normalized surface reflectance image sequence to eliminate differences in observation angle and solar illumination, generating a reflectance sequence with consistent geometry and illumination.

[0082] Quality diagnosis is performed using a reflectivity sequence consistent with geometry and illumination. Image sharpness, strip noise, exposure anomalies and motion artifact indices are calculated and frame-level quality masks and health labels are generated.

[0083] The quality mask, health label, and reflectance sequence are jointly input into the cloud and fog detection process. Combined with physical rules (multispectral threshold, color temperature / exponential method) and deep semantic segmentation network, and incorporating temporal consistency constraints, the output is a cloud and fog probability map, thin cloud transparency estimation, and cloud boundary confidence map.

[0084] Adaptive cloud and fog removal is performed based on cloud and fog probability maps and thin cloud transparency estimates: multi-temporal synthesis or multi-view reconstruction is used to restore high-probability thick cloud areas, and learning-driven deconvolution is used to defog thin cloud / fog areas, and cloud and fog restoration images and restoration confidence are output.

[0085] The cloud and fog restoration images and restoration confidence are compared with the original frames for detail fidelity. For low confidence restoration areas, a review annotation is triggered or super-resolution reconstruction is used to restore details, and a restoration image sequence with confidence annotations is output.

[0086] Multi-scale spatial-frequency domain analysis was performed on the restored image sequence with confidence annotations and combined with temporal saliency measures. Potential salient region candidates were extracted using spatial texture operators, frequency domain spectral energy and short-term motion activation detection to form a preliminary set of region of interest candidate sets with metadata of location, scale, saliency score and uncertainty.

[0087] The standardized temporal image set and the initial region of interest (ROI) candidates are used to construct contrastive learning sample pairs with semantic weights according to the ROI-aware strategy. Contrastive representation learning is driven by training batches, and pre-trained representation parameters and contrastive queue statistics are produced, as follows:

[0088] The standardized temporal image set and preliminary region of interest candidates are cropped to generate a multi-scale region of interest segment sequence, and a segment set with location, scale, timestamp and uncertainty annotations is obtained;

[0089] View construction and enhancement are performed on the fragment set: semantically preserved locally enhanced views, cross-temporal sampling views, cross-modal mapping views and panoramic context views are generated, and semantic weights are assigned to each view based on fragment size, saliency and uncertainty to obtain a weighted set of views;

[0090] Sample pairs are constructed using a weighted set of views: positive sample pairs are constructed based on different views within the same segment, positive sample pairs are constructed based on the same object across time series, and positive sample pairs are constructed based on the same position across modalities. Difficult negative sample pairs are generated through adjacent spatial / spectral similarity and feature nearest neighbor mining. At the same time, each pair is attached with a pair type label and weight annotation to obtain a batch of sample pairs with type and weight annotations.

[0091] The importance-balanced sampling and queue strategy configuration are performed on batches of samples with type and weight annotations: oversampling by small target, adjusting the positive and negative ratio within the batch by significance, and prioritizing high-weight segments into the momentum queue. The output is the training batch and momentum queue update plan after weighted sorting.

[0092] The contrastive representation optimization is driven by training mini-batch and momentum queue update plan: a weighted contrastive loss is implemented on the main encoder and temporal consistency and cross-modal alignment objectives are constrained in parallel. The parameters of the main / momentum encoder are updated and the contrastive queue statistics (feature mean, variance, queue length occupancy rate, and the ratio of enqueued to dequeued sample diversity) are summarized. The updated representation parameters and contrastive queue statistics (the contrastive queue statistics include feature mean, variance, queue length occupancy rate, and the ratio of enqueued to dequeued sample diversity) are output.

[0093] The updated representation parameters and comparison queue statistics are serialized and the sampling strategy metadata (including view weight rules, hard-to-bear mining threshold and queue retention strategy) is saved as a reproducible pre-training package for subsequent fine-tuning and online sampling.

[0094] The pre-trained representation parameters and comparative cohort statistics are used to fine-tune the task and calibrate the confidence level on a small number of labeled samples. This generates a task-sensitive feature extractor and classifier, and outputs the confidence calibration parameters, as detailed below:

[0095] Load pre-trained representation parameters and comparative queue statistics, and aggregate labeled small sample sets and their metadata (category labeling, region of interest labeling, sample collection conditions and uncertainty) to form a traceable fine-tuning data package;

[0096] Perform sample balancing and semantic preservation enhancement (including prototype enhancement of small samples, feature domain synthesis and multi-view temporal synthesis) on traceable fine-tuning data packages according to category / scale / uncertainty and importance of region of interest, and generate training / validation splits and enhanced training batches;

[0097] To enhance training batches, phased fine-tuning is implemented: first, the low-level representations are frozen and a lightweight classification head and inter-layer adapter are trained; then, the middle layers are unfrozen for semantic transfer fine-tuning and contrastive regularization is introduced; finally, the low-level layers are jointly fine-tuned to minimize task loss and temporal consistency deviation, and the fine-tuned model parameters and training logs are output.

[0098] The model parameters are fine-tuned to predict the validation set sample by sample and the predicted confidence distribution and uncertainty estimate are collected (Monte Carlo sampling or small-scale model ensemble can be used in parallel to estimate the model uncertainty). A lightweight confidence calibrator (such as temperature scaling or small calibration network) is trained / fitted based on the distribution to generate confidence calibration parameters and calibration mapping table.

[0099] The confidence calibration parameters are mapped to the fine-tuned model output and the calibration effect is evaluated on the validation set (measuring confidence-accuracy consistency and uncertainty detection rate). If necessary, the evaluation error is fed back to the fine-tuning hyperparameters or enhancement strategies for limited iterations. The calibrated model performance metrics and final calibration parameters are output.

[0100] The calibrated model performance metrics and the final calibration parameters are serialized with fine-tuned representation parameters, classifier weights, confidence calibration parameters, and fine-tuning strategy metadata to form a deployable task-sensitive feature extractor and classifier, and output confidence calibration parameters and reproducible fine-tuning package.

[0101] A task-sensitive feature extractor and classifier perform frame-by-frame inference on an online temporal image stream, generating detection / classification results, region-of-interest feature trajectories, and classification confidence time series. Simultaneously, instantaneous encoder bitrate distribution, keyframe records, and bandwidth observation samples are acquired, as detailed below:

[0102] Receive online time-series video streams and corresponding timestamps, frame IDs and sensor tags to form a frame-by-frame raw input frame stream;

[0103] Perform fast preprocessing on the raw input frame stream frame by frame: perform fast color / radiometry normalization, visibility screening and low-latency quality scoring, and output a preprocessed image frame stream with frame-level quality labels;

[0104] The system takes a preprocessed image frame stream with frame-level quality labels as input and calls a task-sensitive feature extractor and classifier to infer frame by frame. It outputs the candidate region of interest, the local feature vector of each candidate, the classification probability distribution and confidence estimate for each frame, and adds a trigger mark for inference for low-confidence candidates.

[0105] Using the candidate set and its local feature vectors of each frame as input, temporal association and online feature fusion are performed: the region of interest feature trajectory is updated based on position prediction (such as Kalman / motion prior), feature similarity (cosine) and prototype matching, and the output is a region of interest trajectory entry with trajectory ID, time series embedding, position sequence and confidence sequence;

[0106] Using trajectory entries in the region of interest as input, calculate trajectory-level statistics and anomalous event indicators: extract trajectory velocity and acceleration, rate of change of scale in the region of interest, confidence oscillation intensity, and uncertainty accumulation, and trigger important events (such as sudden motion, sudden scale change, or confidence collapse) based on rules or learning thresholds, and output trajectory-level summaries and event signals;

[0107] The trajectory velocity and acceleration are used to measure the instantaneous speed and dynamic changes of the target on the ground or pixel plane, reflecting the target's motion and sudden motion events (such as acceleration and turning).

[0108] Trajectory velocity: Select a time interval Δt (e.g., the interval between two adjacent frames or the average interval within a window), and divide the spatial distance between the current position and the previous position of the target center point by Δt within this interval to obtain the instantaneous trajectory velocity;

[0109] Acceleration: Between two adjacent velocity samples, the instantaneous acceleration is obtained by subtracting the previous velocity from the latter velocity and then dividing by the corresponding time interval.

[0110] The scale change rate is used to measure the rate of change of the target size at the imaging scale, reflecting the target's approach / movement from the sensor, abrupt scale changes, or mismatches (such as occlusion causing a sudden change in the detection box).

[0111] Region of Interest (ROI) Scale Change Rate: At time t, the difference between the area of ​​the detection box of the target at time t and the area at time t−1 is divided by the time interval Δt to obtain the instantaneous scale change rate (area unit / second).

[0112] The confidence oscillation intensity is used to measure the fluctuation amplitude of the confidence assigned to the target by the detection / classifier within a short time or window scale, reflecting the tracking stability, false detection / false detection tendency and appearance uncertainty.

[0113] Confidence oscillation intensity: Within a specified time window W (e.g., the past N frames), take the confidence sequence of each frame of the target, and calculate the standard deviation of the sequence as the oscillation intensity index;

[0114] The uncertainty accumulation is used to measure the accumulation trend of the uncertainty of target identification / location over time, reflecting the cumulative effect of declining long-term tracking confidence (such as cumulative drift after occlusion) or unreliable model predictions.

[0115] Uncertainty accumulation: The uncertainty measures obtained in each frame (such as entropy based on model output, variance of Monte Carlo sampling estimate, or confidence interval width) are accumulated and summed within a time window W;

[0116] The instantaneous bit rate distribution of the encoder, keyframe insertion records and bandwidth observation samples are synchronously acquired using trajectory-level summaries and event signals as trigger conditions. The acquired encoder / transmission observations and the trajectory entries of the region of interest within the corresponding time window are precisely aligned and packaged according to timestamps to generate an aligned observation package for jitter risk assessment.

[0117] The aligned observation package is subjected to lightweight feature compression (e.g., local embedding quantization and hashing), secure signature and timestamp link encapsulation and then streamed to the jitter risk judgment entry and long-term log. At the same time, event signals drive real-time sampling actions (e.g. triggering high-resolution frame capture or extending the keyframe window), forming a closed-loop data flow of inference-monitoring-sampling.

[0118] Using detection / classification results, region of interest feature trajectories, classification confidence time series, and encoder / transmission observation alignment as input, windowed perturbation features are calculated and extracted, as follows:

[0119] Taking the aligned detection / classification results, region of interest feature trajectories, classification confidence time series and encoder / transmission observations as input, the system segments and aligns the timestamps according to a fixed window length and overlap ratio to generate a synchronous windowed observation batch. It also performs time interpolation or forward padding on missing samples within the window and outputs the synchronous windowed observation batch.

[0120] Using the synchronous windowed observation batch as input, the original perturbation index sequence is calculated: the switching frequency of the region of interest, the rate of change of the scale of the region of interest, the trajectory velocity and acceleration sequence, the trajectory break count, the classification confidence oscillation frequency, the instantaneous bit rate allocation skew, the mean of the bandwidth prediction residual, the end-to-end delay jitter and packet loss burst count, and the windowed original perturbation index matrix is ​​output.

[0121] The region of interest switching frequency is used to measure the rate at which the identification / assignment of objects considered "interesting" (i.e., those with trajectory IDs or candidate identifiers) changes within a time window, reflecting the frequency of jumps in points of interest. The switching frequency is obtained by counting the number of events that change all trajectory IDs or region of interest identifiers within a specified time window (i.e., a pixel or region switching from A to B, or a new appearance / disappearance event within the same field of view is also counted as a switching event), and then dividing the total number of switching events by the time window length (seconds).

[0122] The trajectory breakage count is used to measure tracking stability and count the number of times the trajectory is interrupted due to occlusion / missed detection / ID switching within a time window. For each trajectory ID, its appearance-disappearance-reappearance pattern is scanned within the time window: when the trajectory reappears after being missing for several consecutive frames and is determined to be the same object, it is counted as a breakage. The number of breaks of all trajectories is counted and summed to obtain the trajectory breakage count.

[0123] The classification confidence oscillation frequency is used to measure the volatility of the classifier's output confidence over time, reflecting the frequent fluctuations in confidence caused by unstable predictions or easy confusion of categories. By obtaining the confidence time series for each region of interest (or trajectory), local extreme points (local maximum or minimum) in the confidence series are identified. The oscillation frequency is obtained by dividing the number of local extreme points by the time window length.

[0124] The instantaneous bitrate allocation skew is used to measure whether the encoder's bitrate allocation for multiple regions of interest or the entire frame of pixels is too concentrated on a few targets within a certain time window, reflecting the fairness of resource allocation; by obtaining the average allocated bitrate list of each region of interest within the time window, the ratio of the maximum bitrate to the median bitrate in the list is calculated as the instantaneous bitrate allocation skew.

[0125] The mean of the bandwidth prediction residuals is used to measure the accuracy and lag of the bandwidth estimator, reflecting the network's ability to match instantaneous bit rate demand. By calculating the difference (either absolute or relative difference) between the predicted bandwidth value and the subsequently measured transmission rate sample at each sampling time, the mean of these residuals is calculated within a time window. The mean reflects the bias (systematic overestimation or underestimation).

[0126] The end-to-end delay jitter is used to measure the volatility of end-to-end transmission delay, which directly affects the smoothness and delay stability of the real-time experience. It is calculated by averaging the absolute values ​​of the differences between adjacent samples using a continuously sampled sequence of end-to-end delay samples (e.g., sampled RTT / arrival delay for each frame or segment).

[0127] The packet loss burst count is used to measure the number and severity of consecutive packet loss events at the network layer, affecting retransmission and playback recovery capabilities. It identifies a sequence of consecutively lost packets (consecutive packet loss length ≥ preset threshold, e.g., ≥ 3 packets) within a time window as a burst packet loss event, and counts the number of such events and the number of packets in each event as the packet loss burst count.

[0128] The perturbation features are subjected to temporal smoothing, decomposition, and normalization to output a normalized windowed perturbation feature vector;

[0129] Normalized windowed perturbation feature vectors are input in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules for risk inference. The outputs of the two are fused to generate a continuous live streaming jitter risk analysis index, which is then mapped to adaptive encoding and transmission adjustment instructions, as follows:

[0130] Normalized windowed perturbation feature vectors are input in parallel into a causal rule-driven interpretable scorer and a lightweight temporal predictor: The causal rule-driven interpretable scorer decomposes the input into several factor scores according to a predefined causal graph (e.g., region of interest switching frequency score, region of interest scale change rate score, trajectory velocity score, acceleration score, trajectory breakage count score, classification confidence oscillation frequency score, instantaneous bit rate allocation skew score, bandwidth prediction residual mean score, end-to-end delay jitter score, packet loss burst count score), and performs fast local counterfactual probing on each factor (perturb the factor by a small amplitude and measure the score change);

[0131] The lightweight time series predictor performs short-term trajectory prediction and event probability estimation on the input, and outputs the perturbation trajectory prediction, event occurrence probability curve and prediction uncertainty (such as quantile interval or variance estimation) for several future time steps.

[0132] Based on the outputs of the causal scorer and the time series predictor, an interpretable factor vector and a prediction distribution vector are formed respectively. The factor vector contains the standardized score and counterfactual sensitivity of each perturbation factor (calculated by the ratio of the score change of each perturbation factor to the score change of all factors); the prediction distribution vector contains the probability of event occurrence and the prediction uncertainty.

[0133] The factor vector and the predicted distribution vector are weighted and summed to generate a continuous live streaming jitter risk analysis index.

[0134] Based on the aforementioned live stream jitter risk analysis index, a multi-dimensional adaptive control vector is mapped as the input.

[0135] The mapping employs both hierarchical thresholding and continuous mapping:

[0136] When the live stream jitter risk analysis index is less than the first threshold (low risk) of the live stream jitter risk analysis index, conservative optimization (reducing ROI and strengthening) is performed.

[0137] When the live stream jitter risk analysis index is greater than or equal to the first threshold and less than the second threshold (medium risk), a smoothing-first strategy is implemented (increasing the ROI switching smoothing window, adjusting bitrate allocation, and retaining the lowest background bitrate). When the live stream jitter risk analysis index is greater than or equal to the second threshold (high risk), a preservation-first strategy is implemented (retaining the minimum bitrate of key ROIs, increasing the redundancy level, shortening the key frame interval, or switching to reliable transmission mode). The control vector specifically includes several quantifiable instruction fields, such as: ROI switching smoothing window (ms), minimum retained bitrate for each ROI (kbps), bitrate allocation value, FEC level, minimum key frame interval (number of frames), transmission mode flag, etc.

[0138] The control vector is serialized into an instruction packet with timestamp, window ID and priority and sent to the encoder and the transmission end to perform encoding / transmission adjustment and record the execution record;

[0139] This invention tightly integrates AI-driven motion detection and tracking with adaptive control of streaming media encoding to achieve accurate identification of key subjects in the frame and dynamic management of the area of ​​interest. Simultaneously, it senses real-time bitrate fluctuations at the encoding end and bandwidth changes at the network end, thereby constructing a continuous and interpretable live streaming jitter risk analysis index. Under complex operating conditions with frequent ROI switching and bandwidth prediction lag, it can identify potential image quality jitter and latency risks in advance through the fusion of causal rule scoring and temporal prediction. The live streaming jitter risk analysis index is directly mapped to adaptive encoding and transmission adjustment instructions, achieving priority protection for key areas, smooth bitrate allocation, and optimized latency control. This significantly improves the clarity and smoothness of the live stream, reduces sudden jitter and stuttering, and enhances the viewer experience. Furthermore, it provides interpretable risk quantification indicators and closed-loop adaptive control, giving the system stronger stability, robustness, and intelligence in complex dynamic environments.

[0140] The above formulas are all dimensionless calculations. The formulas are derived from software simulations based on a large amount of collected data to obtain the most recent real-world results. The preset parameters in the formulas are set by those skilled in the art according to the actual situation.

[0141] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more sets of available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium. A semiconductor medium can be a solid-state drive.

[0142] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0143] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An AI-based motion tracking and streaming media optimization live broadcast system, characterized by: It includes a multi-source spatiotemporal data acquisition module, a spatiotemporal data standardization and ROI candidate generation module, an ROI-aware contrastive representation learning module, a task fine-tuning and confidence calibration module, an online time-series image feature inference module, a windowed perturbation feature extraction module, and a risk inference and adaptive coding control module; The multi-source spatiotemporal data acquisition module is used to acquire multi-source remote sensing images and their sensor trajectories, attitudes and timestamps to form raw spatiotemporal data streams; The spatiotemporal data standardization and ROI candidate generation module is used to perform high-precision geometric correction, cross-modal radiometric uniformity and adaptive cloud and fog removal on the original spatiotemporal data stream, and generate a standardized time-series image set and preliminary region of interest candidates with metadata. The ROI-aware contrastive representation learning module is used to construct semantically weighted contrastive learning sample pairs with the standardized temporal image set and the initial region of interest candidates according to the ROI-aware strategy, and drive the contrastive representation learning with training batches to produce pre-trained representation parameters and contrastive queue statistics. The task fine-tuning and confidence calibration module is used to perform task fine-tuning and confidence calibration on labeled small samples using pre-trained representation parameters and contrast queue statistics, generating task-sensitive feature extractors and classifiers and outputting confidence calibration parameters. The online temporal image feature inference module is used to infer the online temporal image stream frame by frame with a task-sensitive feature extractor and classifier, and generate detection / classification results, region of interest feature trajectories and classification confidence time series, while simultaneously acquiring encoder instantaneous bit rate distribution, key frame records and bandwidth observation samples. The windowed perturbation feature extraction module is used to calculate and extract windowed perturbation features with the detection / classification results, region of interest feature trajectory, classification confidence time series and encoder / transmission observation alignment as input, perform temporal smoothing, decomposition and normalization on the perturbation features, and output a normalized windowed perturbation feature vector. The risk inference and adaptive coding control module is used to perform risk inference by inputting normalized windowed perturbation feature vectors in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules, fuse the outputs of the two to generate a continuous live jitter risk analysis index, and map the live jitter risk analysis index into adaptive coding and transmission adjustment instructions. A task-sensitive feature extractor and classifier perform frame-by-frame inference on an online temporal image stream, generating detection / classification results, region-of-interest feature trajectories, and classification confidence time series. Simultaneously, instantaneous encoder bitrate distribution, keyframe records, and bandwidth observation samples are acquired, as detailed below: Receive online time-series video streams and corresponding timestamps, frame IDs and sensor tags to form a frame-by-frame raw input frame stream; Perform fast preprocessing on the raw input frame stream frame by frame; The system takes a preprocessed image frame stream with frame-level quality labels as input and calls a task-sensitive feature extractor and classifier to infer frame by frame. It outputs the candidate region of interest, the local feature vector of each candidate, the classification probability distribution and confidence estimate for each frame, and adds a trigger mark for inference for low-confidence candidates. Temporal correlation and online feature fusion are performed using the candidate set and its local feature vectors for each frame as input. Using trajectory entries in the region of interest as input, calculate trajectory-level statistics and anomaly indicators, and output trajectory-level summaries and event signals; The instantaneous bit rate distribution of the encoder, keyframe insertion records and bandwidth observation samples are synchronously acquired using trajectory-level summaries and event signals as trigger conditions. The acquired encoder / transmission observations and the trajectory entries of the region of interest within the corresponding time window are precisely aligned and packaged according to timestamps to generate an aligned observation package. The lightweight time series predictor performs risk inference by: the lightweight time series predictor performs short-term trajectory prediction and event probability estimation on the input, and outputs the disturbance trajectory prediction, event occurrence probability curve and prediction uncertainty for several future time steps.

2. The AI-based motion tracking and streaming media optimization live broadcast system according to claim 1, characterized in that: The standardized temporal image set and the initial region of interest (ROI) candidates are used to construct contrastive learning sample pairs with semantic weights according to the ROI-aware strategy. Contrastive representation learning is driven by training batches, and pre-trained representation parameters and contrastive queue statistics are produced, as follows: The standardized temporal image set and preliminary region of interest candidates are cropped to generate a multi-scale region of interest segment sequence, and a segment set with location, scale, timestamp and uncertainty annotations is obtained; Perform view construction and enhancement using the aforementioned fragment set; Construct sample pairs from a weighted collection of views; Perform importance-balanced sampling and queue strategy configuration on batches of samples with type and weight annotations; Comparative representation optimization is driven by training mini-batch and momentum queue update plans. The updated characterization parameters are serialized with the comparison queue statistics and the sampling strategy metadata is saved.

3. The AI-based motion tracking and streaming media optimization live broadcast system according to claim 2, characterized in that: The pre-trained representation parameters and comparative cohort statistics are used to fine-tune the task and calibrate the confidence level on a small number of labeled samples. This generates a task-sensitive feature extractor and classifier and outputs the confidence calibration parameters, as detailed below: Load pre-trained representation parameters and contrast queue statistics, and aggregate labeled small sample sets and their metadata to form a traceable fine-tuning data package; Perform sample balancing and semantic preservation enhancement on traceable fine-tuning data packages according to category / scale / uncertainty and region of interest importance, and generate training / validation splits and enhanced training batches; To enhance training batch implementation through phased fine-tuning; The model parameters are fine-tuned to predict the validation set sample by sample and the prediction confidence distribution and uncertainty estimate are collected. A lightweight confidence calibrator is trained / fitted based on the distribution to generate confidence calibration parameters and calibration mapping table. The confidence calibration parameters are used to map the fine-tuned model output and the calibration effect is evaluated on the validation set. The output of the calibrated model performance metric and the final calibration parameters are then used. The calibrated model performance metrics and final calibration parameters are serialized with fine-tuned representation parameters, classifier weights, confidence calibration parameters, and fine-tuning strategy metadata to form a deployable task-sensitive feature extractor and classifier, and output confidence calibration parameters and reproducible fine-tuning package.

4. The AI-based motion tracking and streaming media optimization live broadcast system according to claim 1, characterized in that: Using detection / classification results, region of interest feature trajectories, classification confidence time series, and encoder / transmission observation alignment as input, windowed perturbation features are calculated and extracted, as follows: Taking the aligned detection / classification results, region of interest feature trajectories, classification confidence time series and encoder / transmission observations as input, the system segments and aligns the timestamps according to a fixed window length and overlap ratio to generate a synchronous windowed observation batch. It also performs time interpolation or forward padding on missing samples within the window and outputs the synchronous windowed observation batch. Using the synchronous windowed observation batch as input, the original perturbation index sequence is calculated: the switching frequency of the region of interest, the rate of change of the scale of the region of interest, the trajectory velocity and acceleration sequence, the trajectory break count, the classification confidence oscillation frequency, the instantaneous bit rate allocation skew, the mean of the bandwidth prediction residual, the end-to-end delay jitter and packet loss burst count, and the windowed original perturbation index matrix is ​​output.

5. The AI-based motion tracking and streaming media optimization live broadcast system according to claim 4, characterized in that: Normalized windowed perturbation feature vectors are input in parallel to an interpretable scorer and a lightweight time series predictor based on causal rules for risk inference. The outputs of the two are fused to generate a continuous live streaming jitter risk analysis index, which is then mapped to adaptive encoding and transmission adjustment instructions, as follows: Normalized windowed perturbation feature vectors are input in parallel into an interpretable scorer and a lightweight time series predictor based on causal rules: the interpretable scorer driven by causal rules decomposes the input into several factor scores according to a predefined causal graph, and performs fast local counterfactual detection for each factor. The lightweight time series predictor performs short-term trajectory prediction and event probability estimation on the input, and outputs the perturbation trajectory prediction, event occurrence probability curve and prediction uncertainty for several time steps in the future; Based on the outputs of the causal scorer and the time series predictor, an interpretable factor vector and a prediction distribution vector are formed respectively. The factor vector contains the standardized score and counterfactual sensitivity of each perturbation factor; the prediction distribution vector contains the probability of event occurrence and the prediction uncertainty. The live streaming jitter risk analysis index is generated by weighting and summing the factor vector and the predicted distribution vector.

6. The AI-based motion tracking and streaming media optimization live broadcast system according to claim 5, characterized in that: Based on the aforementioned live stream jitter risk analysis index, a multi-dimensional adaptive control vector is mapped as the input. The mapping employs both hierarchical thresholding and continuous mapping: When the live stream jitter risk analysis index is less than the first threshold of the live stream jitter risk analysis index, conservative optimization is performed; When the live stream jitter risk analysis index is greater than or equal to the first threshold and less than the second threshold, a smoothing-first strategy is executed; when the live stream jitter risk analysis index is greater than or equal to the second threshold, a preservation-first strategy is executed; the control vector specifically includes several quantifiable instruction fields.

Images