End-edge-cloud collaborative real-time video analysis system for dynamic data drift

Abstract

The invention relates to the field of computer vision and edge computing, and discloses a dynamic data drift-oriented end-edge-cloud collaborative real-time video analysis system, which comprises an end side device, an edge node and a cloud server which are connected in sequence. And the end side equipment executes video acquisition and feature extraction, and caches and uploads data. The edge node carries out parallel calculation on a feature space and prediction distribution drift based on the uploaded data, and carries out adaptive sampling based on a comprehensive drift score; and meanwhile, a teacher model is utilized to acquire a pseudo label and monitor accuracy, and a sending period is dynamically scheduled according to the change rate of the accuracy. And the cloud server adaptively adjusts the training frequency based on reinforcement learning, updates the global model by using a linear layer fine tuning strategy and issues the global model. According to the method, real-time response to dynamic data drift and model performance optimization are realized in a resource-constrained environment through an end-side cloud collaborative closed-loop mechanism.

Description

Edge-Cloud Collaborative Real-Time Video Analytics System for Dynamic Data Drift Technical Field

[0001] This invention relates to the fields of computer vision and edge computing, specifically to an end-edge-cloud collaborative real-time video analysis system for dynamic data drift. Background Technology

[0002] Deep learning models often face the challenge of data drift (including covariate drift and concept drift) in dynamic scenarios such as video surveillance, where data distribution changes over time. When the distribution of inference data differs from that of training data, the model's generalization ability significantly decreases. Existing solutions mostly rely on centralized cloud processing or pre-set retraining mechanisms, but this static strategy is difficult to adapt to the intensity of dynamically changing drift: it wastes computation and energy during periods of stable data, while during periods of drastic change, the performance cannot recover in time due to response lag. At the same time, relying on full data backhaul puts enormous pressure on limited uplink bandwidth, making it difficult to meet the timeliness requirements of real-time analysis.

[0003] In terms of drift detection and data processing, existing statistical test-based algorithms have high computational complexity, making them difficult to deploy directly on edge devices with limited computing power. Furthermore, they often lack joint monitoring of the feature space and predicted distribution, making them prone to missed detections in mixed drift scenarios. Meanwhile, the random or fixed-frequency sampling commonly used on edge devices lacks effective evaluation of sample value, resulting in a large amount of redundant data consuming bandwidth, while high-value samples containing key drift features may be discarded, reducing the efficiency of subsequent model fine-tuning.

[0004] Furthermore, in existing architectures, edge acquisition, edge transmission, and cloud training are typically optimized independently, lacking a unified coordination mechanism and closed-loop feedback. Cloud training requirements cannot guide edge sampling, and edge bandwidth status fails to dynamically constrain transmission behavior. This fragmented architecture makes it difficult to achieve a global balance between model accuracy, communication bandwidth, and computational energy consumption under resource-constrained conditions.

[0005] Therefore, this invention proposes an edge-cloud collaborative real-time video analytics system for dynamic data drift to address the shortcomings of existing technologies. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides an edge-cloud collaborative real-time video analytics system for dynamic data drift, which solves the problems of model performance degradation caused by dynamic changes in data distribution in resource-constrained edge computing environments, as well as the difficulty in balancing real-time performance, bandwidth consumption, and computing power consumption in static strategies.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a real-time video analysis system for edge-cloud collaboration oriented towards dynamic data drift, characterized in that it comprises: an end-side device, an edge node, and a cloud server that establish communication connections sequentially; the end-side device includes: a feature extraction and lightweight inference module, used to extract feature vectors from video frames and output preliminary prediction results; a local caching module, used to cache video frames, feature vectors, and prediction results, and upload the cached data to the edge node when the network is available; the edge node includes: a drift detection and sampling decision module, used to calculate feature space drift index and prediction distribution drift in parallel based on the data uploaded from the end-side device. The system includes: a drift index module to generate a comprehensive drift score and perform adaptive sampling decisions based on it; an edge inference module configured with a high-precision teacher model to generate highly reliable pseudo-labels and monitor the real-time accuracy of the model; and a dynamic transmission control module to calculate the transmission cycle based on the accuracy change rate and upload the selected labeled samples and performance indicators to the cloud server. The cloud server includes: an adaptive training module to adjust actions based on the system state vector and the output model training frequency of the policy network; and a model update module to fine-tune the global model using the uploaded sample data according to the determined training frequency and send the updated model parameters only to the edge device to complete closed-loop optimization.

[0008] Preferably, the drift detection and sampling decision module has a built-in domain classifier for distinguishing between the input current feature vector and the reference feature vector to calculate the feature space drift index; the drift detection and sampling decision module determines the predicted distribution drift index by calculating the divergence between the current predicted distribution and the reference predicted distribution.

[0009] Preferably, the drift detection and sampling decision module uses a weighted fusion method to combine the feature space drift index and the predicted distribution drift index to generate the comprehensive drift score.

[0010] Preferably, when calculating the sampling probability, the drift detection and sampling decision module comprehensively considers the sample uncertainty based on prediction entropy, the model's confidence in the sample, and the current available network bandwidth status; the drift detection and sampling decision module is configured to use bandwidth constraints to correct and reduce the final transmission probability when the available network bandwidth decreases.

[0011] Preferably, the edge inference module calculates the real-time accuracy by comparing the preliminary prediction results uploaded by the end-side device with the pseudo-labels generated by the high-precision teacher model; the accuracy change rate is used to characterize the degree of deviation of the current real-time accuracy from the historical benchmark.

[0012] Preferably, the dynamic transmission control module uses a nonlinear mapping method to calculate the transmission period, so that the transmission period decreases as the performance degradation of the accuracy change rate decreases, and approaches the preset maximum period value when the performance is stable.

[0013] Preferably, the system state vector received by the adaptive training module includes the current training frequency, model accuracy, system cumulative energy consumption, and the comprehensive drift score uploaded by the edge node; the model training frequency adjustment action includes increasing the frequency, maintaining the frequency, or decreasing the frequency.

[0014] Preferably, the policy network is evaluated based on a multi-objective reward function, which includes a resource penalty term for suppressing high-frequency training and a drift response incentive term for generating positive incentives to offset the resource penalty term when high drift is detected and accuracy decreases.

[0015] Preferably, the model update module adopts a linear layer fine-tuning strategy, which is configured to freeze the backbone network parameters of the global model during training and only update the weights and bias parameters of the final linear classification layer of the global model.

[0016] Preferably, the model update module extracts only the updated linear classification layer parameters to construct a lightweight update package, and broadcasts the lightweight update package only to the edge device to achieve online hot update of the model.

[0017] This invention provides an edge-cloud collaborative real-time video analytics system for dynamic data drift. It offers the following advantages: 1. By implementing a dual-modal drift detection mechanism at edge nodes, this invention simultaneously calculates feature space drift indices and predicted distribution drift indices. The system utilizes a domain classifier to monitor changes in the statistical characteristics of the input data, while simultaneously using weighted divergence to monitor changes in the distribution of the output categories, and integrates both into a comprehensive drift score. This mechanism can distinguish and cover two different data distribution changes: covariate drift and concept drift. Compared to single-dimensional detection methods, it can more accurately capture the risk of model performance degradation in complex environments, providing precise trigger signals for subsequent resource scheduling.

[0018] 2. This invention employs a strategy combining adaptive sampling and dynamic transmission control during the data uplink transmission phase. The edge device locally caches video data and its inference results, and uploads the cached data to the edge node when the network is available, without participating in real-time data filtering decisions. The edge node assesses the uncertainty of the samples based on the samples uploaded from the edge device, combining prediction entropy and model confidence, and adaptively calculates the sampling probability according to the overall drift degree, thereby filtering out key samples with high training value. Simultaneously, the edge node dynamically adjusts the data transmission cycle to the cloud based on the real-time accuracy change rate of the model through a nonlinear mapping function. This design significantly reduces the transmission of redundant data and saves network bandwidth when model performance is stable; while rapidly increasing the transmission frequency of high-value samples when performance degradation is detected, ensuring that the cloud can obtain key data in a timely manner for model repair, achieving a dynamic balance between communication overhead and response speed.

[0019] 3. This invention introduces an adaptive training frequency adjustment mechanism based on reinforcement learning in the cloud. By constructing a Markov decision process that includes multiple dimensions such as drift score, accuracy, and energy consumption, and training the policy network using a multi-objective weighted reward function, the system can intelligently decide whether to start the training task based on the current drift intensity and resource status. This mechanism avoids the waste of computing resources or response lag caused by traditional fixed-period training. In particular, under the influence of the drift response incentive term, it can automatically break resource constraints in high-risk scenarios to prioritize model updates, thereby maintaining the high availability of the model in a dynamically changing environment.

[0020] 4. This invention adopts a fine-tuning strategy based on linear layers and a lightweight parameter closed-loop update mechanism. During the model update process, the system freezes the backbone network parameters, performs gradient updates only on the terminal classification layer, and packages and sends the updated small number of parameters to the edge device. This method not only preserves the general feature extraction capability obtained by the model in the large-scale pre-training stage and prevents overfitting to noisy data in the case of drastic drift, but also significantly reduces the training energy consumption in the cloud and the downlink bandwidth occupation, realizing low-cost, low-latency online hot update of the model. Attached Figure Description

[0021] Figure 1 is a schematic diagram of the system architecture of the present invention; Figure 2 is a schematic diagram of the method flow of the present invention; Figure 3 is a schematic diagram of the hardware structure of the electronic device of the present invention.

[0022] Among them, 100 is the end-side device; 101 is the video acquisition module; 102 is the feature extraction and lightweight inference module; 103 is the local cache module; 200 is the edge node; 201 is the data cache module; 202 is the edge inference module; 203 is the dynamic transmission control module; 204 is the drift detection and sampling decision module; 300 is the cloud server; 301 is the status monitoring module; 302 is the adaptive training module; 303 is the model update module; 401 is the processor; 402 is the memory; 403 is the communication interface; and 404 is the communication bus. Detailed Implementation

[0023] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Referring to Figure 1, Figure 1 is a schematic diagram of an end-edge-cloud collaborative real-time video analysis system architecture for dynamic data drift according to an embodiment of the present invention. The present invention provides an end-edge-cloud collaborative real-time video analysis system for dynamic data drift, comprising: an end-side device 100, an edge node 200, and a cloud server 300; the end-side device 100, the edge node 200, and the cloud server 300 sequentially establish communication connections to form a three-layer collaborative processing architecture.

[0025] The edge device 100 is deployed at the video acquisition site to perform video data acquisition, feature extraction, local inference, and data caching and uploading. The edge device 100 includes: a video acquisition module 101 for acquiring real-time video stream data and inputting the video frame sequence to the feature extraction and lightweight inference module 102; the feature extraction and lightweight inference module 102 is equipped with a lightweight deep neural network for performing operations on the input video frames and outputting the corresponding feature vectors and preliminary prediction results.

[0026] The local caching module 103 is used to receive and cache video frames, feature vectors and preliminary prediction results, and send the cached data packets to the edge node 200 when the network connection is available.

[0027] Edge node 200 is deployed at the network edge to receive sample data from end-side device 100, perform data drift detection and adaptive sampling, and schedule data transmission cycles. Edge node 200 includes: a data caching module 201, which receives video frames, feature vectors, and preliminary prediction results from end-side device 100 and stores them in a queue in chronological order; an edge inference module 202, configured with a high-precision teacher model, which performs inference on the received video frames to generate pseudo-labels and calculates the real-time accuracy of the end-side model; a dynamic transmission control module 203, which dynamically calculates the data transmission cycle based on changes in real-time accuracy and controls the frequency of data transmission from the edge node to the cloud; and a drift detection and sampling decision module 204, which calculates feature space drift index and predicted distribution drift index in parallel based on the data uploaded from the end-side device, generates a comprehensive drift score, and performs adaptive sampling decisions to screen high-value samples.

[0028] The cloud server 300 is deployed in the central data center to perform global model training strategy optimization and parameter updates. The cloud server 300 includes a state monitoring module 301 for receiving data uploaded by the edge node 200 and parsing the current state vector of the system. The state vector includes training frequency, model accuracy, system energy consumption, processing latency, drift score and buffer occupancy data.

[0029] The adaptive training module 302 is configured with a policy network based on reinforcement learning; the adaptive training module 302 is used to input the state vector into the policy network and output the model training frequency adjustment action for the current time; the adjustment action includes increasing the training frequency, maintaining the training frequency, or decreasing the training frequency.

[0030] The model update module 303 is connected to the adaptive training module 302 and is used to fine-tune the global model using uploaded sample data according to a determined training frequency. The model update module 303 adopts a linear layer fine-tuning strategy, updating only the weight parameters of the model's classification layer while keeping the backbone network parameters unchanged.

[0031] The model update module 303 is also used to send the updated model parameters to the edge device 100 via the network; after receiving the updated parameters, the edge device 100 updates the local feature extraction and lightweight inference modules 102 respectively to complete the closed-loop optimization.

[0032] Referring to Figure 2, Figure 2 is a flowchart of an end-edge-cloud collaborative real-time video analysis method for dynamic data drift according to an embodiment of the present invention. The present invention provides an end-edge-cloud collaborative real-time video analysis method for dynamic data drift, comprising the following steps: S100, constructing an end-edge-cloud collaborative communication architecture, establishing communication connections between the end-side device, edge nodes, and cloud server, and initializing functional modules at each level; S200, the end-side device acquires real-time video frames, extracts video frame features and performs inference, and caches the video frame features, prediction results, and corresponding data in local storage; when the network is available, the end-side device uploads the cached data to the edge node offline; S300, the edge node receives and caches data from the end-side device, calculates the feature space drift index and the prediction distribution drift index in parallel based on sample features and prediction results, and generates a comprehensive drift score; the edge node uses a high-precision teacher model to generate highly reliable pseudo-labels for the samples. The system combines comprehensive drift scores, sample uncertainty, and bandwidth status to calculate sampling probabilities and screen high-value samples. Simultaneously, edge nodes use a high-precision teacher model to label samples and monitor the model's real-time accuracy. The data transmission cycle is dynamically calculated based on the rate of change in accuracy, and labeled samples and performance metrics are uploaded to the cloud server according to the transmission cycle. At S400, the cloud server receives samples and performance metrics, constructs a system state vector containing training frequency, accuracy, energy consumption, and drift scores, and uses a reinforcement learning strategy network to output the current training frequency adjustment action. Based on the adjusted frequency, a linear layer fine-tuning strategy is used to update the global model. At S500, the cloud server only sends the updated model parameters to the edge devices, which then load the model parameters to complete closed-loop adaptive optimization.

[0033] The above modules and steps will be described in detail below with reference to specific embodiments.

[0034] Referring to the edge device 100 shown in Figure 1, this device serves as the system's data sensing and preliminary analysis node, performing efficient video stream processing through its internally integrated hardware processor. The data processing flow running in the edge device 100 is mainly completed collaboratively by the feature extraction and lightweight inference module 102 and the local caching module 103. To achieve real-time response in a resource-constrained edge environment, this processing flow is specifically implemented through the following steps S201 to S202: S201, performing video frame feature extraction and preliminary inference; the feature extraction and lightweight inference module 102 receives real-time video frame data transmitted by the video acquisition module 101; the feature extraction and lightweight inference module 102 is pre-loaded with a lightweight deep neural network model (e.g., ViT-Small or a pruned CNN model); for the video frame input at the current moment, the feature extraction and lightweight inference module 102 maps it to a high-dimensional feature space and extracts the current feature vector. ; Eigenvector It is a deep semantic representation output from the backbone layer of the neural network, which can reflect the texture, shape, and color distribution information of the input image; at the same time, the feature extraction and lightweight inference module 102 utilizes the model's classification layer to process the feature vector. The process is performed to output the predicted probability distribution of the current frame belonging to each preset category. ,in, Indicates the total number of categories. Indicates that the current frame belongs to the first... The probability value of the class.

[0035] In step S202, local data caching and uploading are performed. The local caching module 103 receives the raw video frames, feature vectors, and preliminary prediction results output in step S201 and writes them into a local circular buffer. When a network connection is detected to be available or the buffer reaches a preset threshold, the end device sends data packets to the edge node 200 through the communication interface. This step eliminates the need for complex drift detection and data filtering on the end side, thus significantly reducing the computational load and power consumption of the end device.

[0036] Referring to the edge node 200 shown in Figure 1, this node serves as a relay and control hub connecting the end-side device 100 and the cloud server 300, primarily undertaking data aggregation, quality assessment, and transmission scheduling functions. The edge node 200 runs data processing logic through its internally integrated edge computing processor. This logic is specifically executed collaboratively by the data caching module 201, the edge inference module 202, the dynamic transmission control module 203, and the drift detection and sampling decision module 204. To achieve efficient data backhaul under limited edge bandwidth resources, the specific processing flow of the edge node 200 is implemented through the following steps S301 to S303: S301, performing data caching, pseudo-label generation, and accuracy monitoring; the data caching module 201 is equipped with a high-speed memory for receiving raw sample data packets from the end-side device 100 via a network interface; the sample data packets not only contain cached video frame image data but also the end-side device's preliminary prediction results and feature vectors for that frame image; the data caching module 201 temporarily stores the received data in a receiving queue and transmits it to the edge inference module 202 in timestamp order.

[0037] The edge inference module 202 is pre-deployed with a high-precision deep neural network model (i.e., the teacher model). The teacher model has a greater network depth and more parameters than the lightweight model on the edge device, enabling it to provide more accurate classification results. For each received video image frame, the edge inference module 202 uses the teacher model to perform inference operations and outputs a high-confidence predicted category for that frame. This high-confidence predicted category is regarded by the system as a "pseudo-label" for that frame and is used as a benchmark ground truth for evaluating the performance of the edge model. The edge inference module 202 compares the preliminary prediction results uploaded by the edge device with the pseudo-label generated by the teacher model: if they match, the edge model is judged to have made a correct prediction; if they do not match, the prediction is judged to be incorrect. Based on the comparison results within a certain time window, the edge inference module 202 statistically calculates and performs calculations on the accuracy index of the edge model under the current data distribution in real time.

[0038] Meanwhile, the edge inference module 202 calculates the accuracy change index. This indicator It is defined as the difference between the current real-time accuracy and the minimum accuracy threshold that the system can tolerate, or it can be defined as a statistic that reflects the positive trend of accuracy. When the edge model adapts to the current environment, the prediction accuracy is high and remains at a large positive value. When data drift causes the model performance to degrade, the accuracy decreases and then decreases or becomes negative.

[0039] S302, Perform dual-modal drift detection and adaptive sampling decision; this step is performed by the drift detection and sampling decision module 204 based on the feature vectors and preliminary prediction results within a time window provided by the data cache module 201.

[0040] First, drift detection of the feature space and the predicted distribution is performed in parallel. Regarding feature space drift detection, the drift detection and sampling decision module 204 maintains a set of reference feature vectors and uses a built-in lightweight domain classifier to distinguish whether the feature vector within the current time window comes from the reference distribution or the current real-time distribution. Based on the classification results, the feature drift score is calculated using the following formula: In the formula, The area under the receiver operating characteristic curve of the domain classifier; The value is used to measure the separability between the reference data and the current data; feature drift score. It quantifies the degree of shift in the statistical characteristics of the input data in the feature space, and is used to characterize the strength of covariate drift; As a domain classifier, it is configured as a binary classifier to distinguish whether the input feature vector comes from the reference distribution or the current real-time distribution; The reference feature vector is derived from the baseline data during the system initialization phase or the sampled data from the previous stable period, representing the data distribution characteristics during model training or in a stable state. This is the current feature vector.

[0041] In terms of predicted distribution drift detection, the drift detection and sampling decision module 204 calculates the difference between the predicted distribution and the reference predicted distribution within the current time window, using weighted Kullback-Leibler (KL) divergence as the metric. The formula for calculating the predicted distribution drift score is as follows: In the formula, Indicates category index, Total number of categories; The current frame belongs to the first Predicted probability of class For the reference distribution, the th The probability of a class; In order to target the The weighting coefficients set for the class; weighting coefficients This is used to enhance the monitoring of specific categories, assigning larger weight values ​​to easily confused or high-value categories, so that the calculated drift score can more sensitively reflect the distribution changes of overlapping or concerned categories; The concept of drift strength is used to characterize the distribution of output labels.

[0042] Secondly, a comprehensive drift assessment is performed. The drift detection and sampling decision module 204 linearly weights and fuses the calculated feature drift score with the predicted distribution drift score to generate a comprehensive drift score: In the formula, The balance coefficient has a value range of [0,1]. Adjust the settings based on the specific needs of the application scenario: when environmental factors such as lighting and weather change drastically, increase the adjustment. The value is adjusted to focus on monitoring the feature space; when the target type changes frequently in the scene, the value is reduced. The value focuses on monitoring the predicted distribution.

[0043] Next, the global sampling rate is dynamically updated. The drift detection and sampling decision module 204 periodically adjusts the system's global sampling rate based on the overall drift score and bandwidth status. The calculation formula is as follows: In the formula, Indicates the current moment. The global sampling rate at the current moment; It is a smoothing factor with a value range between (0,1). It is used to control the degree of influence of historical states on the current update and to prevent the sampling rate from fluctuating violently due to instantaneous fluctuations. The target sampling rate is calculated based on the current system state; target sampling rate The drift detection and sampling decision module 204 calculates the drift score based on multiple state indicators, including the overall drift score. The gradient change value, the change in current model accuracy, the difference between the current frame prediction entropy and historical entropy, and the occupancy rate of the internal transmission buffer of the edge node; when the overall drift score is considered... When the accuracy of the model increases significantly or decreases, the calculated Increase, thereby improve This allows the system to capture more samples for subsequent model adaptation.

[0044] Next, the sampling selection probability of a single frame image is calculated. The drift detection and sampling decision module 204 calculates the instantaneous sampling probability for each video frame, taking into account both the uncertainty of the sample and the model confidence. The calculation formula is as follows: In the formula, The global sampling rate for the time window determined in the aforementioned global sampling rate update step; For the current frame image The prediction entropy is used to quantify the uncertainty of the model's prediction result for the sample; the higher the entropy value, the more difficult it is for the model to identify the sample, and the higher the value of the sample for model updates. For the current frame image The maximum predicted probability value (i.e., confidence level); the lower the confidence level, the more likely the sample is to be located on the classification boundary or belong to an unknown distribution, and thus has higher sampling value. and These are positive adjustment coefficients for prediction entropy and confidence, used to increase the probability weight of a sample when it has high uncertainty. The function ensures that the calculated probability value does not exceed 1.0; through this formula, the system can automatically increase the selection probability of those samples that are on the drift boundary or difficult samples, thereby achieving accurate screening of high-value data.

[0045] Finally, the final transmission probability is adjusted based on the available bandwidth, and filtering is performed. To ensure that the data transmission strategy matches the current edge-to-cloud network environment, the drift detection and sampling decision module 204 adjusts the sampling probability using the currently available bandwidth: In the formula, The instantaneous sampling probability is obtained from the aforementioned calculation; This represents the available uplink bandwidth from the current edge node to the cloud. To achieve the current global sampling rate Downlink Image The estimated uplink bandwidth consumption required for its related metadata; this correction mechanism ensures that when network conditions are poor ( When the transmission probability is reduced, the system automatically reduces the transmission probability to prioritize the transmission of extremely high-value samples and prevent data loss or increased delays caused by network congestion.

[0046] S303, execute the dynamic transmission strategy; the dynamic transmission control module 203 is connected to the edge inference module 202 and the drift detection and sampling decision module 204, and is used to adjust the accuracy change index according to the dynamic transmission strategy. The frequency of data transmission to the cloud is dynamically adjusted. To quickly upload data and trigger cloud training when model performance degrades, the dynamic transmission control module 203 uses a non-linear mapping function to calculate the current data transmission cycle. Sending period The calculation formula is as follows: In the formula, This indicates the time interval at which edge nodes send data packets to the cloud; The preset minimum transmission period (e.g., 50 milliseconds) represents the highest transmission frequency the system can transmit when it detects severe drift or a sharp drop in performance, ensuring that the cloud can obtain the latest samples in a timely manner for model updates; The preset maximum transmission period (e.g., 550 milliseconds) represents the minimum transmission frequency of the system when the model performance is stable, in order to minimize long-term bandwidth occupation.

[0047] This is a sensitivity adjustment parameter, and This parameter is used to control the system's response rate to changes in accuracy: a larger value indicates a lower response rate. A value that increases the slope of the Sigmoid function means that when the accuracy metric... When a tiny drop occurs, the transmission period It will quickly move from near The state slipped to near The state reflects the system's high sensitivity to drift events and low-latency response characteristics; smaller The value makes the adjustment process smoother.

[0048] Through the above formula, the dynamic transmission control module 203 implements an adaptive transmission mechanism: when When the value is large (i.e., the model accuracy is high and the system is stable), the exponential term in the denominator... Approaching 0, making Approaching The system enters low-frequency transmission mode; when When the value decreases (i.e., model accuracy decreases, drift occurs), the exponential term increases rapidly, the denominator becomes larger, and thus... Rapidly decrease and approach The system automatically switches to high-frequency transmission mode; the dynamic transmission control module 203, based on the calculated... Set a timer to periodically package and send the sample data and performance statistics filtered in step S302 to the cloud server 300.

[0049] Referring to the cloud server 300 shown in Figure 1, this server serves as the global control center of the system, executing a closed-loop optimization strategy based on reinforcement learning through the integrated adaptive training module 302. To achieve the optimal balance between model performance and computing resources in a dynamically changing data environment, the cloud server 300 dynamically adjusts the training frequency of the global model by constructing a Markov Decision Process (MDP) model. This process is specifically implemented through the following steps S401 to S402: S401, constructing the reinforcement learning environment and state-action space; the state monitoring module 301 receives system operation data uploaded from edge nodes in real time and structures it into a system state vector. System state vector It includes not only current performance metrics, but also characteristics of environmental changes, as specifically defined below: In the formula, Indicates the current time The model training frequency, which directly determines the rate at which cloud computing resources are consumed; This indicates the accuracy of the current global model on the validation set or edge feedback. This indicates the system's current cumulative energy consumption. This represents the average latency of data processing and transmission; The overall drift score uploaded by the edge nodes is used to characterize the degree of data distribution offset in the current environment; The rate of change of accuracy; This represents the buffer occupancy rate of the edge nodes. Given the current data transmission rate; the adaptive training module 302 will... As the observation input for reinforcement learning agents.

[0050] The adaptive training module 302 defines a discrete action space. Action set contains Decrease, Maintain, Increase Three types of operation commands; the system updates the training frequency for the next time step based on the action commands output by the policy network. If the action is Increase, then use the preset step size. Increase the frequency, but do not exceed the maximum frequency. If the action is Decrease, then reduce the frequency but not below the minimum frequency. If the action is Maintain, the current frequency remains unchanged. This action mechanism allows the system to dynamically adjust the computational load with fine granularity.

[0051] S402, calculate the comprehensive reward function and optimize the strategy; the adaptive training module 302, based on the reward function... The decision-making of the policy network is evaluated and feedback is provided; to address the resource waste or response lag caused by the single optimization objective in existing technologies, a reward function is proposed. The design employs a multi-objective weighted coupling approach, and the specific calculation formula is as follows: In the formula, to These are positive weighting coefficients; Part 1 As a basic performance reward, it guides the model to always pursue higher classification accuracy; Part Two This is a resource penalty term; it suppresses excessively high training frequency and energy consumption through a negative feedback mechanism. When the model performance is stable, this penalty term prompts the agent to choose actions that reduce the training frequency, thereby saving computational resources during non-drift periods.

[0052] Part Three This is the drift response stimulus term; this is the core design for solving the key technical problem of "when to start training under resource constraints"; when high drift is detected ( The model accuracy decreased (approaching 1). When the value increases, this item generates a larger positive reward value. This positive reward value is used to offset the resource penalty caused by increasing the training frequency (i.e., offset the effect of the second part), so that the system determines that it is a positive benefit to consume more resources for model updates in high-risk scenarios. Conversely, if no drift occurs or the model has adapted (high accuracy), this item value is small, and the system will be subject to the resource penalty item again and tend to reduce the frequency.

[0053] Part IV and subsequent terms introduce auxiliary constraints; among which To predict stability constraint terms, This represents the Kullback-Leibler divergence between the current time step and the previous time step of the global model output distribution, used to penalize drastic oscillations in the prediction distribution; The project establishes a reward correlation between sampling rate and drift level, encouraging the system to match higher data throughput when drift occurs; This is a threshold reward item; it is awarded when the accuracy rate exceeds a preset threshold. Additional rewards are given to guide the model to converge quickly to the high-performance range.

[0054] Based on the calculated reward value The adaptive training module 302 uses the Proximal Policy Optimization (PPO) algorithm to update the policy network parameters. By maximizing the expected cumulative reward, the policy network gradually learns the optimal frequency scheduling policy under different drift states and resource constraints, realizing the adaptive evolution of the end-edge-cloud collaborative system in a complex dynamic environment.

[0055] Referring to the cloud server 300 shown in Figure 1, in order to ensure that the policy network can converge stably to the optimal solution and achieve efficient iteration of the global model, the adaptive training module 302 and the model update module 303 work together to execute the following steps S403 to S405 to complete the PPO policy optimization and closed-loop update based on linear fine-tuning.

[0056] S403, perform parameter optimization of the PPO policy network; the adaptive training module 302 uses the collected states, actions, and calculated comprehensive reward values ​​to optimize the parameters of the policy network. Iterative updates are performed; to address the training instability issue caused by the difficulty in determining the step size in traditional policy gradient methods, this embodiment employs a proximal policy optimization algorithm; this algorithm introduces a shearing mechanism to limit the difference between the old and new policies, ensuring the monotonicity and stability of policy updates; the adaptive training module 302 constructs and maximizes the following objective function. : In the formula, Indicates the expectation of a time step; For new strategies Compared to the old strategy In state Select action The probability ratio; This is the estimate of the advantage function, used to measure action. The degree of superiority or inferiority relative to the average level; This is a trimming parameter, usually set to a small positive number (such as 0.1 or 0.2), used to define the trust region for policy updates.

[0057] In the formula The function will use probability ratios Forced truncation within the interval Internally; the function of this mechanism is: when the dominant function When the objective function is positive (i.e., the current action is better than the average), the upper limit of the objective function's growth is limited to prevent excessively large policy update steps from causing parameter oscillations; when the advantage function... When the value is negative, the lower bound of the objective function is restricted; by maximizing the objective function, the adaptive training module 302 enables the policy network to gradually learn the optimal rule for adjusting the training frequency under different drift intensities and resource states.

[0058] S404, Perform model update based on linear fine-tuning; After determining the training frequency at the current moment, the model update module 303 starts the training task of the global deep neural network model; In view of the technical characteristics of drastic changes in data distribution and limited edge backpropagation bandwidth in dynamic data drift scenarios, the model update module 303 adopts a linear fine-tuning strategy instead of a full parameter fine-tuning strategy; Specifically, the global model consists of a feature extraction backbone network (such as a Transformer layer or convolutional layer) and a terminal linear classification layer; During training, the model update module 303 freezes all weight parameters of the backbone network so that they remain unchanged during backpropagation and do not participate in gradient calculation and update; The model update module 303 only unfreezes the weight parameters and bias parameters of the terminal linear classification layer, calculates the loss function using the new sample data uploaded by the edge nodes, and performs gradient descent update only for this linear classification layer.

[0059] This linear fine-tuning method achieves two significant technical effects: First, by maintaining the stability of the backbone network parameters, it preserves the general feature extraction capabilities acquired by the model during the large-scale pre-training phase, effectively preventing overfitting of the model to noisy data when faced with severe covariate drift such as Gaussian noise or blurring, thus maintaining a higher generalization accuracy than full parameter fine-tuning during complex drift phases; Second, since it involves only a very small number of parameter calculations, it significantly reduces the energy consumption and time overhead of training in the cloud.

[0060] S405, execute parameter compression and closed-loop distribution; after completing linear fine-tuning, the model update module 303 extracts the updated linear classification layer parameter matrix; since the backbone network parameters have not changed, the model update module 303 only needs to build a lightweight update package containing the linear layer parameters, without packaging the entire model file; the data size of this update package is usually only at the megabyte (MB) level (e.g., 2.4MB), which is an order of magnitude compression compared to the full model parameters (usually hundreds of MB or even GB); the cloud server 300 broadcasts this lightweight update package through the downlink communication link and distributes it only to the end device 100.

[0061] After receiving the update packet, the edge device 100 parses the linear classification layer parameters and replaces the corresponding layer structure in the local model, completing the online hot update of the model. Thus, the system completes a complete closed loop from edge data collection, edge drift detection, data filtering and transmission, cloud adaptive training to parameter feedback, realizing continuous optimization of model performance with low latency and low bandwidth consumption in a dynamic data drift environment.

[0062] Referring to Figure 3, Figure 3 is a schematic diagram of the hardware structure of an electronic device for implementing the above-described end-edge-cloud collaborative real-time video analysis method for dynamic data drift according to an embodiment of the present invention. The electronic device provided in this embodiment of the present invention is intended to serve as a hardware entity carrier for the end-side device 100, the edge node 200, or the cloud server 300; as shown in Figure 3, the electronic device includes: a processor 401, a memory 402, a communication interface 403, and a communication bus 404 connecting the various components.

[0063] The communication bus 404 is used to realize the connection and communication between these components; the communication bus 404 is divided into address bus, data bus, control bus, etc., and its structural forms include Industry Standard Architecture (ISA) bus, Peripheral Component Interconnect (PCI) bus or Extended Industry Standard Architecture (EISA) bus, etc.

[0064] The processor 401 is the control center of the electronic device. It connects various parts of the electronic device through various interfaces and lines. It executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 402 and calling data stored in the memory 402. The processor 401 integrates an arithmetic logic unit and a control unit. In view of the different computing load requirements of the end-side device, edge node and cloud server in this invention, the specific implementation of the processor 401 includes, but is not limited to: general-purpose central processing unit (CPU), graphics processing unit (GPU), neural network processor (NPU) and digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component.

[0065] Specifically, when the electronic device is used as an edge device 100, the processor 401 preferably integrates a low-power NPU or DSP for hardware-accelerated feature extraction and lightweight network inference operations in the lightweight inference module 102; when the electronic device is used as a cloud server 300, the processor 401 is preferably configured as a high-performance GPU cluster or TPU to support parallel computation of the reinforcement learning policy network in the adaptive training module 302 and backpropagation gradient computation of the model update module 303.

[0066] The memory 402 is used to store software programs and modules, such as the program instructions / modules corresponding to the dual-modal drift detection algorithm, adaptive sampling strategy, and PPO reinforcement learning algorithm in the embodiments of the present invention. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402, that is, to implement the method steps described above. The memory 402 mainly includes a program storage area and a data storage area. The program storage area stores the operating system and at least one application program required for a function (such as a video acquisition program, a network transmission program, etc.). The data storage area stores data created according to the use of the electronic device (such as video frame feature vectors, reference distribution data, pseudo-labels, policy network parameters, etc.). In addition, the memory 402 includes high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0067] The communication interface 403 is used for network communication between the aforementioned electronic devices and other devices. In the end-edge-cloud collaborative architecture of the present invention, the communication interface 403 is a key component for realizing data interaction in the three-layer architecture. For the end device 100, the communication interface 403 supports wireless communication protocols (such as Wi-Fi, 5G / 6G, NB-IoT) or wired Ethernet protocols to send filtered sample data in bandwidth-limited and fluctuating network environments. For the edge node 200 and the cloud server 300, the communication interface 403 supports high-throughput fiber optic networks or high-speed local area network interfaces to meet the requirements of receiving multiple concurrent data and distributing model parameters.

[0068] This invention also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the edge-cloud collaborative real-time video analysis method for dynamic data drift as described in any of the above embodiments. The computer-readable storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk. When the instruction sequence stored in the storage medium is executed, it specifically drives the processor to perform the following operations: controlling the edge device to perform video acquisition, feature extraction, local inference, and data caching and uploading; controlling the edge node to perform bimodal drift detection, adaptive sampling and filtering, pseudo-label generation, and dynamic transmission; and controlling the cloud server to build a reinforcement learning environment, calculate a multi-objective reward function, and perform model closed-loop update based on linear fine-tuning.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A real-time video analytics system with edge-cloud collaboration for dynamic data drift, characterized in that, include: The communication connection is established sequentially between the end-side device, the edge node, and the cloud server. The end-side device includes: a feature extraction and lightweight inference module for extracting feature vectors from video frames and outputting preliminary prediction results; and a local caching module for caching video frames, feature vectors, and prediction results, and uploading the cached data to the edge node when the network is available. The edge node includes: a drift detection and sampling decision module for parallel calculation of feature space drift and prediction distribution drift indicators based on the data uploaded from the end-side device, generating a comprehensive drift score, and performing adaptive sampling decisions accordingly; an edge inference module configured with a high-precision teacher model for generating highly reliable pseudo-labels and monitoring the model's real-time accuracy; and a dynamic transmission control module for calculating the transmission cycle based on the accuracy change rate and uploading the selected labeled samples and performance indicators to the cloud server. The cloud server includes: an adaptive training module for adjusting the model training frequency based on the system state vector using a policy network; and a model update module for fine-tuning the global model using uploaded sample data at a determined training frequency, and sending the updated model parameters only to the end-side device to complete closed-loop optimization.

2. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The drift detection and sampling decision module has a built-in domain classifier to distinguish between the input current feature vector and the reference feature vector to calculate the feature space drift index; the drift detection and sampling decision module determines the prediction distribution drift index by calculating the divergence between the current prediction distribution and the reference prediction distribution.

3. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 2, characterized in that, The drift detection and sampling decision module uses a weighted fusion method to combine the feature space drift index and the predicted distribution drift index to generate the comprehensive drift score.

4. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, When calculating the sampling probability, the drift detection and sampling decision module comprehensively considers the sample uncertainty based on prediction entropy, the model's confidence in the sample, and the current available network bandwidth. The drift detection and sampling decision module is configured to use bandwidth constraints to correct and reduce the final transmission probability when the available network bandwidth decreases.

5. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The edge inference module calculates real-time accuracy by comparing the preliminary prediction results uploaded by the end-side device with the pseudo-labels generated by the high-precision teacher model; the accuracy change index is used to characterize the degree of deviation of the current real-time accuracy from the historical benchmark.

6. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The dynamic transmission control module uses a nonlinear mapping method to calculate the transmission period, so that the transmission period decreases as the performance degradation indicated by the accuracy change index decreases, and approaches the preset maximum period value when the performance is stable.

7. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The system state vector received by the adaptive training module includes the current training frequency, model accuracy, system cumulative energy consumption, and the comprehensive drift score uploaded by the edge node; the model training frequency adjustment action includes increasing the frequency, maintaining the frequency, or decreasing the frequency.

8. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The policy network is evaluated based on a multi-objective reward function, which includes a resource penalty term for suppressing high-frequency training and a drift response incentive term for generating positive incentives to offset the resource penalty term when high drift is detected and accuracy decreases.

9. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 1, characterized in that, The model update module adopts a linear layer fine-tuning strategy, which is configured to freeze the backbone network parameters of the global model during training and only update the weights and bias parameters of the final linear classification layer of the global model.

10. The end-edge-cloud collaborative real-time video analysis system for dynamic data drift as described in claim 9, characterized in that, The model update module extracts only the updated linear classification layer parameters to construct a lightweight update package, and sends the lightweight update package only to the edge device through network communication to realize online hot update of the model.

Images