A video adaptive transmission and reconstruction method and device based on end-cloud cooperation

Abstract

The application provides a video adaptive transmission and reconstruction method and device based on end-cloud cooperation. The application provides a low-bandwidth key frame priority transmission and intermediate frame reconstruction method and device of camera end and server cooperation, realizes a cooperative mechanism of end-side semantic key frame intelligent labeling and cloud-side high-quality intermediate frame reconstruction, adaptively switches a transmission mode according to a real-time network state, and effectively guarantees video event integrity and visual coherence in a low-bandwidth environment.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a method and apparatus for adaptive video transmission and reconstruction based on end-to-end cloud collaboration. Background Technology

[0002] In real-time video transmission applications such as video surveillance, vehicle-mounted cameras, and drone video transmission, network bandwidth fluctuations and limitations are common challenges. To ensure transmission, traditional technologies often employ methods such as reducing bitrate or resolution, adjusting keyframe (I-frame) intervals, or utilizing inter-frame differential compression. While these methods alleviate bandwidth pressure to some extent, they often come at the cost of sacrificing video quality, detail, or smoothness.

[0003] In recent years, some technical solutions have emerged aimed at optimizing bandwidth utilization and video quality.

[0004] Methods based on redundant frame filtering and region recoding: As proposed in prior art 1, this method is a video transmission method for machine vision. At the camera end, it calculates the difference between subsequent frames and the most recent keyframe, filters out redundant frames with differences less than a threshold, and recodes the retained keyframes by dividing them into regions of different importance levels (assigning different quantization parameters), aiming to reduce bandwidth consumption while maintaining the accuracy of the machine vision algorithm. Essentially, this method is a frame-level selective discarding and compression, and does not involve the regeneration and reconstruction of missing frames.

[0005] Video frame segmentation transmission method: Another approach, such as existing technology 2, divides the video frame to be encoded (except for the first frame) into multiple sub-video frames and encodes and transmits them separately. This ensures that the number of keyframes in the sub-video frames is less than the number of segments, aiming to avoid the delay caused by transmitting large keyframes all at once. This is an approach of breaking down large frames into smaller, separate transmissions, focusing on optimizing the transmission process rather than intelligent content selection and recovery.

[0006] Cloud-based intermediate frame generation method: Existing technology 3 proposes a method for generating intermediate frames. This method performs frequency domain decomposition on the preceding and following image frames, extracts multi-scale features using a preset image generation model, generates a fused feature map, and finally synthesizes the intermediate frame. This method focuses on the server-side interpolation algorithm itself (frequency domain analysis and feature fusion), and does not involve the end-side collaborative keyframe selection strategy, bandwidth adaptive transmission mechanism, or a complete closed-loop system linked to network status.

[0007] Methods based on coordinating keyframe transmission from multiple cameras: Existing technology 4 addresses the keyframe transmission conflict problem in environments with multiple network cameras. It coordinates through a network platform to prevent multiple cameras from transmitting large amounts of keyframes simultaneously, thus preventing network congestion. This solution addresses the timing issue of keyframe generation, but it does not explore the intelligence of keyframe selection (such as semantic importance) or the recovery of lost frames.

[0008] Based on the above background, the existing technology mainly has the following disadvantages:

[0009] The criteria for judging the importance of frames are too simplistic: existing solutions mostly select key frames based on pixel differences or fixed intervals, which cannot effectively identify the semantic importance of video content (such as whether a specific event or target appears). This may result in the transmitted frame sequence not fully retaining key information, affecting the readability of subsequent event analysis.

[0010] Insufficient edge-cloud collaboration capabilities: Most solutions focus on single-side optimization on either the edge or the cloud. Edge-side solutions lack intelligent adjustment capabilities guided by cloud feedback; cloud-side solutions lack targeted, semantic-based keyframe preprocessing and provision from the edge, making it difficult to achieve globally optimal resource allocation and performance assurance.

[0011] Lack of Adaptive Transmission and Reconstruction Closed Loop: Existing technologies fail to construct a complete adaptive closed-loop system that dynamically adjusts transmission strategies based on real-time network bandwidth (such as switching between "full-frame transmission" and "key frame priority transmission" modes) and triggers corresponding targeted reconstruction mechanisms on the server side. This results in the system being unable to make a smooth and optimal trade-off between bandwidth saving and visual continuity when bandwidth fluctuates drastically.

[0012] Insufficient quality and reliability of reconstructed frames: Some intermediate frame generation technologies focus primarily on the generation algorithm itself, without fully considering the spatiotemporal consistency and semantic rationality of the generated frames with the preceding and following frames. They also lack effective verification and remediation mechanisms for the quality of the generated frames, which may lead to the risk of image distortion or semantic errors. Summary of the Invention

[0013] The present invention aims to at least partially solve one of the technical problems in the related art.

[0014] Therefore, the first objective of this invention is to propose a video adaptive transmission and reconstruction method based on end-to-cloud collaboration.

[0015] The second objective of this invention is to propose a video adaptive transmission and reconstruction device based on end-to-cloud collaboration.

[0016] To achieve the above objectives, a first aspect of the present invention proposes a video adaptive transmission and reconstruction method based on end-to-cloud collaboration, comprising:

[0017] S1, using the semantic keyframe annotation module at the camera end, semantic events in video frames are detected and keyframes are labeled based on a lightweight visual model. The keyframes include coded keyframes and semantic keyframes.

[0018] S2, based on the real-time network bandwidth monitoring results, dynamically switch between full-frame transmission mode and key frame priority transmission mode through the bandwidth adaptive transmission strategy module. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded.

[0019] S3, through the missing frame identification module in the cloud, the missing interval is identified based on the received key frame timestamp and index, and the two-stage intermediate frame reconstruction module is triggered to generate the missing frame. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module.

[0020] S4. The spatiotemporal consistency and semantic rationality of the reconstructed frame are evaluated by the quality verification module. If the evaluation result does not meet the preset threshold, the feedback mechanism is triggered to request the camera to retransmit the original frame in the missing interval.

[0021] S5 reassembles the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis.

[0022] S6 uses a multimodal semantic analysis module and an audio event detection algorithm to annotate video frames. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

[0023] In one embodiment of the present invention, S1 further includes:

[0024] S11, a target detection model is used to analyze video frames in real time. When the confidence of a preset target category exceeds the threshold, the corresponding frame is marked as a semantic key frame.

[0025] S12 employs a scene change detection algorithm, which calculates the structural similarity index (SSIM) between the current frame and the previous frame to determine scene switching, and marks the scene switching frame as a semantic keyframe.

[0026] In one embodiment of the present invention, S2 further includes:

[0027] S21, calculate the bandwidth fluctuation index based on RTT and packet loss rate. When the fluctuation index exceeds the first preset threshold, switch to key frame priority transmission mode. When the fluctuation index is lower than the second preset threshold, switch to full frame transmission mode.

[0028] S22, based on the bandwidth fluctuation index, subdivides the transmission mode into three levels, including full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and degrades them in order of priority.

[0029] In one embodiment of the present invention, S3 further includes:

[0030] S31 uses optical flow to calculate the dense motion field of the preceding and following keyframes, and generates a preliminary version of the missing frames through forward / backward warping.

[0031] S32, the initial version and the input keyframes of the preceding and following keyframes are used to refine the details of the generative model and generate the final missing frames.

[0032] In one embodiment of the present invention, S4 further includes:

[0033] S41, calculate the SSIM index of the reconstructed frame and the preceding and following keyframes, and detect the presence of spatiotemporal flicker or semantic anomalies through a lightweight neural network;

[0034] S42, calculate the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determine whether to trigger the feedback mechanism based on the preset similarity threshold.

[0035] To achieve the above objectives, a second aspect of the present invention provides a video adaptive transmission and reconstruction device based on end-to-cloud collaboration, comprising:

[0036] The semantic keyframe annotation module is used to detect semantic events in video frames and label keyframes using a lightweight visual model at the camera end. The keyframes include coded keyframes and semantic keyframes.

[0037] The bandwidth adaptive transmission strategy module is used to dynamically switch between full frame transmission mode and key frame priority transmission mode based on real-time network bandwidth monitoring results. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded.

[0038] A two-stage intermediate frame reconstruction module is used to identify missing intervals based on the received key frame timestamps and indices, and to generate missing frames. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module.

[0039] The quality verification module is used to evaluate the spatiotemporal consistency and semantic rationality of the reconstructed frames. If the evaluation result does not meet the preset threshold, a feedback mechanism is triggered to request the camera to retransmit the original frames in the missing interval.

[0040] The video sequence reconstruction module is used to reconstruct the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis.

[0041] The multimodal semantic analysis module is used to annotate video frames in conjunction with audio event detection algorithms. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

[0042] This invention discloses a video adaptive transmission and reconstruction method and apparatus based on end-to-cloud collaboration, which realizes a collaborative mechanism between intelligent annotation of semantic keyframes on the end side and high-quality intermediate frame reconstruction in the cloud. It adaptively switches the transmission mode according to the real-time network status, effectively ensuring the integrity and visual continuity of video events in low-bandwidth environments.

[0043] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0045] Figure 1 This is a flowchart of a video adaptive transmission and reconstruction method based on end-to-cloud collaboration according to an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a video adaptive transmission and reconstruction system based on end-to-cloud collaboration according to an embodiment of the present invention;

[0047] Figure 3 This is a structural diagram of a video adaptive transmission and reconstruction system based on end-to-cloud collaboration according to an embodiment of the present invention;

[0048] Figure 4 This is a structural diagram of a video adaptive transmission and reconstruction device based on end-to-cloud collaboration according to an embodiment of the present invention. Detailed Implementation

[0049] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0050] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0051] The following description, with reference to the accompanying drawings, describes a video adaptive transmission and reconstruction method and apparatus based on end-to-cloud collaboration according to an embodiment of the present invention.

[0052] Example 1

[0053] Figure 1 This is a flowchart of a video adaptive transmission and reconstruction method based on end-to-cloud collaboration according to an embodiment of the present invention, including:

[0054] S1, using the semantic keyframe annotation module at the camera end, detects semantic events in video frames and labels keyframes based on a lightweight visual model. The keyframes include coded keyframes and semantic keyframes.

[0055] Specifically, its technical implementation principle is based on a joint decision-making mechanism of lightweight semantic perception and key frame encoding on the edge side. It aims to intelligently identify and retain video frames with semantic value under limited computing resources and network bandwidth, thereby improving the semantic integrity and visual coherence of video reconstruction.

[0056] Specifically, the semantic keyframe annotation module is deployed at the camera end, typically running on an embedded AI chip with edge computing capabilities. This module employs a lightweight object detection model (such as quantized YOLOv5 or YOLOv7) to perform real-time semantic analysis on video frames. The model input is the original video frame or a pre-encoded YUV image, and the output is the presence of a semantic target (such as "person" or "vehicle") within the frame and its confidence level. When a target is detected and the confidence level is higher than a preset threshold (such as 0.65), the frame is labeled as a semantic keyframe. Simultaneously, this module also identifies I-frames output by the encoder (i.e., encoded keyframes) and incorporates them, along with the semantic keyframes, into a unified keyframe set, forming a dual keyframe annotation mechanism.

[0057] The labeling frequency of semantic keyframes can be set to 1-3 frames per second (FPS) to adapt to the rate of semantic change in different scenarios. The input resolution of the object detection model is typically 320×240 or 640×480 to achieve a balance between accuracy and computational cost. The model inference latency should be controlled within 30ms to ensure real-time performance. In addition, the keyframe labeling results will be accompanied by metadata, including timestamps (accurate to milliseconds), object category, confidence score, previous and subsequent keyframe indices, motion vector cues, etc., for subsequent server-side frame reconstruction and quality assessment.

[0058] Furthermore, S1 includes:

[0059] S11, the semantic event detection uses a target detection model to analyze video frames in real time. When the confidence of a preset target category exceeds a threshold, the corresponding frame is marked as a semantic key frame.

[0060] Specifically, the step in this invention, "semantic event detection uses a target detection model to analyze video frames in real time. When the confidence of a preset target category (such as a person or a vehicle) exceeds a threshold, the corresponding frame is marked as a semantic key frame," is based on the combination of target detection algorithms in computer vision and video coding structures. It aims to prioritize the transmission of video frames with semantic value under limited bandwidth conditions, thereby ensuring the integrity and analyzability of video content.

[0061] Specifically, this step first deploys a lightweight object detection model (such as a quantized YOLOv5, YOLOv7, or MobileNet-SSD) on the camera. This model needs to meet real-time requirements and typically runs on an embedded AI chip. The model input is real-time captured video frames, and the output is the detected object category (such as "person" or "vehicle") and its corresponding confidence score. In some implementations, the model can support multi-class detection and remove redundant detection boxes using the non-maximum suppression (NMS) algorithm to ensure the accuracy of object recognition. When the confidence score of a preset object category (such as "person" or "vehicle") detected in a frame exceeds a set threshold (such as 0.75), the frame is determined to contain a semantic event and is marked as a semantic keyframe. Furthermore, metadata such as the frame's timestamp, object type, confidence score, and object location are recorded and uploaded to the server to provide contextual information for subsequent intermediate frame reconstruction.

[0062] The confidence threshold of the object detection model can be dynamically adjusted according to the actual application scenario. For example, in high-density traffic monitoring, a lower confidence threshold (e.g., 0.6) can be set to ensure that all potential vehicles are identified; while in low-traffic scenarios, the threshold can be increased (e.g., 0.85) to reduce false detections and redundant frame transmission. Furthermore, the inference latency of the object detection model should be controlled within 30ms to meet the needs of real-time video analysis. The model's input resolution is typically 720p or 1080p, and the output is JSON format or binary structured data of the detection results.

[0063] S12, the semantic event detection adopts a scene change detection algorithm, which determines scene switching by calculating the structural similarity index (SSIM) between the current frame and the previous frame, and marks the scene switching frame as a semantic key frame.

[0064] Specifically, the semantic event detection step employs a scene change detection algorithm, which calculates the Structural Similarity Index (SSIM) between the current frame and the previous frame to determine whether a scene change has occurred, and marks the scene change frame as a semantic keyframe.

[0065] In its implementation, this step first preprocesses consecutive frames in the video stream, including grayscale conversion, normalization, and size alignment, to eliminate interference from non-semantic factors such as lighting and resolution differences on SSIM calculation. Then, the SSIM algorithm is used to evaluate the structural similarity between the current frame and the previous frame. When the SSIM value is below a preset threshold (e.g., 0.95), it is determined that a significant structural change has occurred between the current and previous frames, i.e., a scene transition. At this point, the system marks the current frame as a semantic keyframe and records its timestamp, frame index, semantic tags (e.g., "person," "vehicle"), and confidence level, among other metadata. This metadata will serve as crucial clues for subsequent cloud-based reconstruction of missing frames.

[0066] S2, based on the real-time network bandwidth monitoring results, dynamically switch between full-frame transmission mode and key frame priority transmission mode through the bandwidth adaptive transmission strategy module. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded.

[0067] Specifically, this step involves dynamically switching between full-frame transmission mode and key-frame priority transmission mode through the bandwidth adaptive transmission strategy module based on real-time network bandwidth monitoring results. In the key-frame priority transmission mode, only the marked key frames and their metadata are uploaded.

[0068] In some implementations, this step relies on the coordinated operation of the bandwidth monitoring module and the transmission control module at the camera end. The bandwidth monitoring module periodically collects network status parameters, such as round-trip time (RTT), packet loss rate, and available bandwidth, typically sampling 1-5 times per second. The transmission control module then makes mode switching decisions based on a preset bandwidth threshold (e.g., set to 3 Mbps). When the current bandwidth is detected to be higher than the threshold, the system enters full-frame transmission mode, that is, it uploads all frames (including I-frames, P-frames, and B-frames) completely according to the standard H.264 or H.265 encoding protocol; when the bandwidth is lower than the threshold, the system automatically switches to keyframe priority transmission mode, uploading only I-frames or semantic keyframes marked by the semantic keyframe annotation module (e.g., P-frames indicating "person" or "vehicle"), along with their metadata, such as timestamps, target type, confidence level, and motion vector cues.

[0069] Specifically, the bandwidth monitoring module typically employs the mobile IP network status assessment method defined in RFC 3775, combining factors such as TCP window size and RTT fluctuation range for comprehensive judgment. The switching logic of the transmission control module can be based on the sliding window average bandwidth (SWAB) algorithm, setting a dynamic switching lag time (e.g., 3 seconds) to avoid frequent switching. Metadata transmission format can follow JSON or a custom binary protocol to ensure efficient parsing of structured information.

[0070] Furthermore, S2 includes:

[0071] S21, calculate the bandwidth fluctuation index based on RTT and packet loss rate. When the fluctuation index exceeds the first preset threshold, switch to key frame priority transmission mode. When the fluctuation index is lower than the second preset threshold, switch to full frame transmission mode.

[0072] This step involves deploying a bandwidth monitoring module at the camera end to collect two key network performance indicators in real time: round-trip time (RTT) and packet loss rate. RTT is measured by periodically sending probe packets and recording the round-trip time, usually in milliseconds (ms). Packet loss rate is measured by statistically analyzing the proportion of data packets that are not successfully received within a set time window, usually expressed as a percentage (%).

[0073] The first preset threshold (Threshold1) and the second preset threshold (Threshold2) are used to define the critical points of the network state. Threshold1 typically indicates that the network bandwidth is unstable or limited, requiring a switch to key frame priority transmission mode; Threshold2 indicates that the network bandwidth has stabilized, allowing a switch back to full frame transmission mode. This threshold setting needs to be optimized based on the bandwidth fluctuation characteristics of the specific application scenario. For example, in 5G networks, Threshold1 can be appropriately lowered to improve response sensitivity, while in Wi-Fi networks, Threshold2 can be increased to avoid the control overhead caused by frequent switching.

[0074] This step is applicable to real-time video transmission scenarios, such as campus security monitoring, vehicle-mounted video backhaul, and drone image transmission. In these scenarios, network bandwidth may fluctuate drastically due to environmental interference, changes in user density, or transmission path switching. By dynamically monitoring RTT and packet loss rate and calculating BFI, the system can promptly switch to a keyframe-priority transmission mode when the network deteriorates, reducing data volume and avoiding video stuttering or interruption; when the network recovers, it resumes full-frame transmission to ensure video integrity and visual quality.

[0075] Furthermore, S2 also includes:

[0076] S22, based on the bandwidth fluctuation index, subdivides the transmission mode into three levels: full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and degrades them in order of priority.

[0077] Specifically, the transmission strategy decision module subdivides the transmission mode into three levels based on the bandwidth fluctuation index: full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and then degrades them in order of priority. This step is technically implemented based on real-time monitoring and quantitative evaluation of network bandwidth status, combined with the semantic features of the video content, to dynamically adjust the transmission strategy in order to maximize the preservation of video semantic information and maintain visual coherence under bandwidth-constrained conditions.

[0078] In its implementation, the transmission strategy decision module first obtains real-time network status parameters, including round-trip time (RTT), packet loss rate, and available bandwidth, through the bandwidth monitoring module. Based on these parameters, the system calculates the Bandwidth Fluctuation Index (BFI), which can be defined as the weighted ratio of the bandwidth change rate to the currently available bandwidth, for example: BFI = α × (ΔB / B avg ) + β × P loss Where α and β are adjustable weighting coefficients, ΔB is the bandwidth fluctuation amplitude per unit time, and B avg For average bandwidth, P loss This represents the packet loss rate. The BFI value is usually set to [0,1] to characterize the stability and availability of the current network.

[0079] Based on the BFI threshold, the system classifies transmission modes into three levels: When BFI < 0.2, a full-frame transmission mode is adopted, transmitting the complete sequence of I-frames, P-frames, and B-frames; when 0.2 ≤ BFI < 0.6, it switches to an I-frame-only transmission mode, uploading only I-frames and some metadata to reduce data volume; when BFI ≥ 0.6, it further degrades to a high-confidence semantic keyframe transmission mode, uploading only I-frames or P-frames marked as related to high-confidence events by the semantic keyframe annotation module, along with their corresponding semantic tags, timestamps, motion vector cues, and other metadata. This degradation strategy ensures that when bandwidth is limited, the system prioritizes retaining frames with high semantic value, thereby guaranteeing the analyzability and integrity of the video content.

[0080] S3, through the missing frame identification module in the cloud, the missing interval is identified based on the received key frame timestamp and index, and the two-stage intermediate frame reconstruction module is triggered to generate the missing frame. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module.

[0081] Specifically, this step involves using a cloud-based missing frame identification module to identify missing intervals based on the received key frame timestamps and indices, and triggering a two-stage intermediate frame reconstruction module to generate missing frames. The two-stage reconstruction module includes a motion estimation module and a generative repair module.

[0082] The cloud-based missing frame identification module first parses the received keyframe sequence using timestamps and frame indices. Keyframes are typically I-frames or semantic keyframes marked by the edge-side semantic annotation module, with timestamp precision usually in the millisecond range (e.g., 16ms or 33ms), and frame indices being a continuously increasing integer sequence. The missing frame identification module identifies the missing frame interval by comparing the difference between the timestamp interval of adjacent keyframes and the expected frame rate (e.g., 25fps or 30fps). For example, if the expected frame sequence is frames 1 to 10, but only frames 1 and 10 are actually received, then frames 2 to 9 are determined to be the missing interval. This identification process can be based on the product of the timestamp difference and the frame rate; if the difference is greater than the expected number of frames, a reconstruction process is triggered.

[0083] Furthermore, the cloud system triggers a two-stage intermediate frame reconstruction module. The first stage is the motion estimation module, which uses optical flow estimation algorithms (such as RAFT, PWC-Net, etc.) to calculate the motion vector field between the preceding and following keyframes, generating the preliminary structure of the intermediate frame through forward / backward warping. This stage mainly recovers large-scale motion information, such as object displacement and background changes, and its output is a low-quality but structurally reasonable intermediate frame. The second stage is the generative inpainting module, which inputs the initially generated intermediate frame and the preceding and following keyframes into a pre-trained generative model (such as a diffusion model based on the U-Net structure) to perform detail inpainting and texture enhancement to eliminate artifacts, occlusion areas, and other defects generated by warping, ultimately outputting a visually coherent and semantically reasonable high-quality intermediate frame.

[0084] Specifically, the motion estimation module typically uses a two-stream convolutional network structure, taking two frames of images and their timestamps as input and outputting a motion vector map with a spatial resolution of 1 / 4 or 1 / 8 of the original frame to reduce computational load. The generative inpainting module operates at a higher resolution (e.g., 1 / 2 or full resolution of the original frame) and employs a multi-scale feature fusion mechanism to improve reconstruction quality. The quality verification module can use metrics such as SSIM (Structural Similarity Index) or VMAF (Multi-Method Evaluation Fusion) to set thresholds (e.g., SSIM > 0.85) to determine whether the reconstructed frame meets application requirements.

[0085] Furthermore, S3 includes:

[0086] S31 uses optical flow to calculate the dense motion field of the preceding and following keyframes, and generates a preliminary version of the missing frames through forward / backward warping.

[0087] S32, the initial version and the input keyframes of the preceding and following keyframes are used to refine the details of the generative model and generate the final missing frames.

[0088] Specifically, the two-stage intermediate frame reconstruction step is a crucial technical step in this invention for maintaining visual coherence and semantic integrity of video under low bandwidth conditions. This step is executed on the server side, and its core lies in efficiently and with high quality reconstructing missing frames between keyframes uploaded from the client side through a combination of staged algorithms. This reduces the amount of data transmitted while ensuring the continuity and analyzability of the video sequence.

[0089] This process is divided into two stages: the first stage is motion estimation and warping, and the second stage is generative model inpainting. In the first stage, the server uses optical flow estimation algorithms (such as RAFT or GMFlow) to perform dense motion field calculations on the preceding and following keyframes to obtain pixel-level motion vectors. Based on these motion vectors, the pixel information of the reference frame is mapped to the intermediate frame position through forward or backward warping techniques to generate a preliminary intermediate frame image. This stage mainly restores large-scale motion structures, such as object movement trajectories and background changes, and features high computational efficiency and fast response speed.

[0090] The second stage introduces a generative model (such as a diffusion model based on the U-Net structure or a GAN) to take the coarse intermediate frames generated in the first stage and the preceding and following keyframes as input for context-aware detail restoration. By learning spatiotemporal features from a large amount of video data, the generative model can effectively fill in artifacts and occluded areas generated during the warping process and enhance texture details, making the reconstructed frames visually more natural and coherent. The output frames in this stage must have an SSIM (structural similarity index) of at least 0.85 to ensure a high degree of consistency with the preceding and following frames in terms of structure and content.

[0091] Specifically, the output resolution of the optical flow algorithm is typically 1 / 4 or 1 / 2 of the original frame to reduce computational load; the input channels of the generative model include the original reference frame, warping results, and motion vector graphics, with an input size of 256×256 or 512×512, and the output size is consistent with the original video. The total latency of the reconstruction process is controlled within 50ms to meet the latency requirements of real-time monitoring scenarios.

[0092] S4 evaluates the spatiotemporal consistency and semantic rationality of the reconstructed frame through the quality verification module. If the evaluation result does not meet the preset threshold, a feedback mechanism is triggered to request the camera to retransmit the original frame in the missing interval.

[0093] Specifically, this step, "evaluating the spatiotemporal consistency and semantic rationality of the reconstructed frames through the quality verification module, and triggering a feedback mechanism to request the retransmission of the original frames in the missing interval from the camera end if the evaluation result does not meet the preset threshold," is based on a video quality assessment and feedback control mechanism, aiming to ensure that the reconstructed video still has high availability in terms of temporal continuity and semantic integrity under low bandwidth conditions.

[0094] The quality verification module first performs spatiotemporal consistency analysis on the received reconstructed frames. Specifically, it uses standard video quality assessment metrics such as the Structural Similarity Index (SSIM) and the Visual Distortion Assessment Model (VMAF) to quantitatively evaluate the motion continuity in the temporal dimension and the texture consistency in the spatial dimension between the reconstructed frames and the preceding and following keyframes. Simultaneously, a lightweight semantic verification network (such as a MobileNetV3 + Transformer architecture) deployed on the server side detects the semantic content in the reconstructed frames to determine whether semantic breaks occur (such as sudden disappearance of the target or unreasonable motion trajectories). This network can perform semantic consistency comparison based on the output features of object detection models such as YOLO or Faster R-CNN.

[0095] Specifically, SSIM evaluation typically sets a threshold of 0.85; a score below this value indicates significant distortion between the reconstructed frame and the preceding and following frames. VMAF evaluation sets a threshold of 85 points (out of 100); a score below this value indicates substandard visual quality. If the output confidence score of the semantic verification network is below 0.75, a semantic anomaly is identified. These thresholds can be dynamically adjusted according to specific application scenarios (such as security monitoring and vehicle video) to balance quality and bandwidth consumption.

[0096] Furthermore, S4 includes:

[0097] S41 calculates the SSIM index of the reconstructed frame and the preceding and following keyframes, and uses a lightweight neural network to detect whether there is spatiotemporal flicker or semantic anomaly.

[0098] Specifically, in the quality verification step of this invention, the server calculates the Structural Similarity Index (SSIM) between the reconstructed frame and the preceding and following keyframes, and combines this with a lightweight neural network to perform spatiotemporal consistency and semantic reasonableness checks on the reconstructed frame, in order to evaluate its quality and determine whether further processing is required. This step ensures that the video generated under low bandwidth conditions reaches an acceptable level in terms of visual coherence and semantic integrity.

[0099] In its implementation, the SSIM index is calculated based on the similarity of the reconstructed frame to its adjacent keyframes in terms of luminance, contrast, and structural information. Specifically, the system employs a sliding window mechanism (typically an 11×11 pixel window with Gaussian weighting σ=1.5) to perform local contrast and structural similarity analysis in the luminance channel (Y channel) of the YUV color space. The SSIM value ranges from [0,1], with values ​​closer to 1 indicating greater similarity between the reconstructed frame and the reference frame. In this invention, the SSIM threshold is set to 0.85. When the SSIM value of the reconstructed frame falls below this threshold, the system triggers further quality checks.

[0100] Furthermore, the system introduces a lightweight neural network model (such as a variant of MobileNetV3 or SqueezeNet) to detect spatiotemporal flicker or semantic anomalies in the reconstructed frames. During training, this model uses datasets containing both normal and anomalous video sequences (such as UCF101 and Kinetics) to identify phenomena like discontinuous motion between frames, sudden disappearance of targets, or unreasonable semantic changes through supervised learning. The model input is a stitched image of the reconstructed frame and its preceding and following keyframes (e.g., a 3-frame stacked input), and the output is an anomaly probability value. When the anomaly probability exceeds a set threshold (e.g., 0.3), the system marks the frame as a potentially defective frame and can request the camera to retransmit denser keyframes within that time period.

[0101] S42, calculate the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determine whether to trigger the feedback mechanism based on the preset similarity threshold.

[0102] Specifically, in this invention, the quality verification module calculates the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determines whether to trigger the feedback mechanism based on a preset similarity threshold. This step is technically implemented based on deep learning feature extraction and similarity matching mechanisms, aiming to evaluate the semantic and visual coherence between the intermediate frames generated by the server and the original keyframes, thereby determining whether it is necessary to request supplementary data from the camera to improve the reconstruction quality.

[0103] In some implementations, the quality verification module first extracts feature vectors from the preceding and following keyframes. Typically, a pre-trained convolutional neural network (CNN) model (such as ResNet-50, EfficientNet, etc.) is used to encode the features of the keyframes, outputting fixed-dimensional feature vectors (e.g., 2048-dimensional). The reconstructed frame also generates feature vectors using the same feature extraction model. Subsequently, the module calculates the cosine similarity between the feature vectors of the reconstructed frame and the preceding and following keyframes as a metric for semantic consistency.

[0104] Furthermore, the system sets a preset similarity threshold (e.g., 0.85). When the cosine similarity between the reconstructed frame and the preceding and following keyframes is lower than this threshold, the quality verification module will trigger a feedback mechanism, requesting the camera to upload more dense keyframes or original frames within that time period to assist the server in performing more accurate reconstruction. This feedback mechanism can be implemented based on protocols such as HTTP / HTTPS, MQTT, or WebRTC, and the specific request content includes parameters such as the timestamp interval and the required frame type (e.g., I-frame or semantic keyframe).

[0105] S5 reassembles the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis.

[0106] Specifically, this step involves reassembling the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order. The technical implementation principle is based on the video frame timestamp synchronization mechanism and frame sequence splicing algorithm, aiming to restore the original video's temporal logical structure and ensure the video's readability and usability during storage or intelligent analysis.

[0107] In some implementations, this step begins with the server-side video sequence reconstruction module receiving keyframes (I-frames or semantic keyframes) and their metadata uploaded from the camera, including timestamps, previous and subsequent keyframe indices, motion vector cues, and semantic labels. Intermediate frames are then generated by the server-side intermediate frame reconstruction module after identifying the missing frame intervals, using a two-stage hybrid frame interpolation algorithm. The first stage employs optical flow methods (such as RAFT or GMFlow) for motion estimation and warping, initially recovering the large-scale motion structure of the missing frames. The second stage utilizes generative models (such as U-Net-based diffusion models or GANs) to perform detail restoration and texture enhancement on the warping results, generating intermediate frames with higher visual quality.

[0108] After receiving keyframes and intermediate frames, the reassembly module sorts and stitches all frames according to the original video's chronological order, based on timestamps and frame index information. Specifically, the frame reassembly process must meet the following technical indicators: timestamp error should be controlled within ±50ms to ensure the consistency of video playback timing; the frame rate recovery target is 25fps or 30fps, consistent with the original video; the stitched video sequence must be verified through quality evaluation indicators such as SSIM (Structural Similarity Index) and VMAF (Multi-Method Evaluation Fusion) to ensure that the reconstructed video reaches an acceptable level in terms of visual quality and semantic integrity.

[0109] This step is widely applicable in practical applications to scenarios with high requirements for the integrity of video events, such as intelligent security, vehicle-mounted video transmission, and drone inspection. For example, in a park monitoring system, when network bandwidth is limited, only keyframes and metadata are uploaded. The server reconstructs intermediate frames and reassembles them into a complete video, which can effectively reduce the amount of data transmitted while ensuring the analyzability and smooth playback of the video content.

[0110] Furthermore, the technical value of this step lies in achieving high-quality video reconstruction and storage under low bandwidth conditions through an edge-cloud collaboration mechanism, solving the problem of event information loss caused by frame dropping in traditional methods, and improving the accuracy and robustness of intelligent video analysis.

[0111] The video adaptive transmission and reconstruction method based on end-to-cloud collaboration in this invention realizes intelligent selection of semantic keyframes and dynamic adaptive transmission of video transmission in low-bandwidth environments. Combined with cloud-based multi-stage reconstruction and quality verification mechanisms, it effectively ensures the integrity and visual coherence of video events.

[0112] S6 uses a multimodal semantic analysis module and an audio event detection algorithm to annotate video frames. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

[0113] Specifically, this step involves intelligently labeling video frames through the synergy of a multimodal semantic analysis module and an audio event detection algorithm. When a preset audio event (such as the sound of breaking glass or an alarm) is detected, the corresponding frame is marked as a semantic keyframe and uploaded. This step aims to improve the semantic fidelity and event integrity of video transmission in low-bandwidth environments.

[0114] In some implementations, the multimodal semantic analysis module consists of a lightweight object detection model (such as a quantized YOLOv5 or YOLOv7) and an audio event detection (AED) model (such as an acoustic classifier based on ResNet-18), deployed on the edge computing unit at the camera end. Video frames and synchronously acquired audio signals are input into their respective models for processing. The audio event detection algorithm uses a sliding window mechanism to perform spectral analysis and feature extraction on the audio signals to identify preset audio event categories. When a frame's corresponding audio signal detects a high-semantic-value event such as the sound of breaking glass or an alarm, that frame is marked as a semantic keyframe. The marking process includes appending metadata (such as event type, confidence level, timestamp, etc.) to the frame header and sending it to the frame buffer to await uploading.

[0115] The audio event detection model uses an input audio sampling rate of 16kHz, a window size of 256 points, and a sliding step size of 128 points. STFT (Short Time Fourier Transform) is employed to extract spectral features. A detection threshold of ≥0.85 confidence level is set to ensure accurate event recognition. Video frames are labeled at a frequency of 30 frames per second (30fps), sampled synchronously with the audio signal to ensure temporal consistency between events and video frames. Semantic keyframes have a higher upload priority than ordinary P-frames or B-frames, with their transmission priority weight set to 1.5~2.0 times that of regular frames.

[0116] This step is applicable to real-time video surveillance systems, especially in scenarios with limited bandwidth or frequent network fluctuations, such as urban security, traffic monitoring, and drone patrols. Through audio event-driven semantic keyframe annotation, the system can retain crucial visual information when an emergency occurs, ensuring the integrity of subsequent video reconstruction and event retrospection. For example, in nighttime monitoring, if the sound of breaking glass is detected, the system will automatically upload the video frame at that moment for further analysis and event confirmation in the cloud.

[0117] This step significantly improves the intelligence and event relevance of keyframe selection by introducing audio events as triggering conditions for semantic annotation. Compared to traditional keyframe selection methods based on inter-frame difference or fixed intervals, this scheme can more accurately retain frames with semantic value, thus ensuring the understandability of video content and the traceability of events even under low bandwidth conditions. Furthermore, this step provides high-quality reference frames and contextual information for subsequent intermediate frame reconstruction in the cloud, enhancing the spatiotemporal coherence and semantic consistency of the reconstructed video, and is a key technical support for realizing edge-cloud collaborative intelligent video transmission.

[0118] This invention discloses a video adaptive transmission and reconstruction method based on end-to-cloud collaboration. In low-bandwidth environments, by introducing multimodal semantic analysis and audio event detection, it achieves more accurate identification and key frame marking of key events in video content, thereby improving the priority transmission efficiency of key information and further enhancing the integrity of video events and anomaly detection response capabilities.

[0119] Example 2

[0120] The following describes in detail an embodiment of the present invention, a video adaptive transmission and reconstruction method based on end-to-cloud collaboration, with reference to the accompanying drawings.

[0121] Imagine a security monitoring scenario in a park, where cameras continuously capture images of the entrance area.

[0122] Camera operation:

[0123] The operation of the camera end is controlled by Figure 2 The modules shown on the left work together to complete the task, and their processes correspond to... Figure 3 The steps in the process.

[0124] Video Acquisition and Encoding: The image sensor acquires raw video, and the encoder encodes it into a video stream containing I-frames, P-frames, and B-frames.

[0125] Semantic keyframe annotation: The semantic keyframe annotation module runs a lightweight object detection model (such as the quantized YOLO series). When a "person" or "vehicle" is detected, it not only marks the encoded I-frame as a keyframe, but also marks frames where the event was detected but the encoded frame might be a P-frame as semantic keyframes, and records metadata such as timestamp, object type, and confidence level, and sends them to the frame buffer. This process is supplementary. Figure 2 The specific implementation in [the document / framework].

[0126] Bandwidth Monitoring and Decision Making: The bandwidth monitoring module continuously measures the uplink RTT and packet loss rate. During weekdays and nighttimes with ample bandwidth, the transmission control module uses full-frame transmission mode. On days with events causing network congestion and bandwidth falling below the threshold, the transmission control module automatically switches to keyframe priority mode. In this mode, only I-frames in the buffer and semantic keyframes marked with "person" / "vehicle" along with their metadata are uploaded, significantly reducing the data volume.

[0127] Metadata includes: uploaded metadata such as timestamps, previous and next keyframe indexes, motion vector cues (if any), semantic tags, etc., providing clues for server-side reconstruction.

[0128] Server-side operations:

[0129] Receiving and identifying missing intervals: The receiving and parsing module acquires the keyframe sequence (e.g., frame 1, frame 10, frame 15, etc.). The frame sequence analysis and missing interval detection module identifies the absence of non-keyframes (frames 2-9) between frame 1 and frame 10 by analyzing the timestamps and frame indices.

[0130] Two-stage intermediate frame reconstruction: The intermediate frame reconstruction module is triggered to reconstruct the missing intervals. Taking the generation of frame 5 as an example:

[0131] Phase 1 (Motion Estimation and Warping): Using frames 1 and 10 as references, optical flow methods (such as the RAFT algorithm) are used to calculate the dense motion field between consecutive frames, and a coarse version of frame 5 is initially generated through forward / backward warping. This phase mainly focuses on recovering large-scale motion structures.

[0132] Phase Two (Generative Model Restoration): The coarse 5th, 1st, and 10th frames are input together into a pre-trained generative model (such as a diffusion model based on the U-Net structure). Based on contextual information, this model performs detail restoration and texture generation on the artifacts and occluded areas generated by warping, outputting visually coherent and detailed high-quality intermediate frames.

[0133] Quality Verification: The quality verification module calculates the SSIM index of the generated sequence of frames 2-9. It calculates the SSIM index relative to the preceding and following keyframes and uses a lightweight network to check for obvious spatiotemporal flicker or semantic anomalies (such as a person suddenly disappearing). If the SSIM is below the threshold or an anomaly is detected, the generated frame sequence is marked, and a request may be made to the camera to retransmit more densely packed original frames within that time period (if the network temporarily improves).

[0134] Sequence Reassembly and Output: The video sequence reassembly module splices the received keyframes and successfully reconstructed intermediate frames in chronological order to form a continuous, smooth, high-quality video for use by the storage, playback, or intelligent analysis modules.

[0135] The present invention may also have the following embodiments:

[0136] In edge server scenarios with limited computing power, the intermediate frame reconstruction module can be simplified to single-stage interpolation. For example, using only high-performance optical flow algorithms (such as GMFlow) or adaptive filtering interpolation algorithms to generate intermediate frames may result in slightly less detail in complex scenes, but it can still guarantee basic smoothness and lower latency.

[0137] There are multiple alternative solutions to the technical solution of this invention. These solutions can also achieve the purpose of this invention, but may differ in effect, cost or complexity.

[0138] Alternatives to semantic keyframe annotation:

[0139] Alternative Solution 1: In addition to object detection, other modalities or algorithms such as scene change detection, abnormal motion detection (such as sudden running), and audio event detection (such as the sound of breaking glass) can be used to define and label "semantic keyframes".

[0140] Alternative Solution 2: The lightweight model can be replaced with other efficient network structures, such as MobileNet-SSD, SqueezeNet-based detectors, etc., as long as they can perform real-time semantic analysis on the edge.

[0141] Alternatives to bandwidth assessment and decision-making:

[0142] Alternative approaches: In addition to RTT and packet loss rate, bandwidth assessment can incorporate predictive bandwidth estimation (e.g., based on historical data) or more complex network state models. Transmission strategies can also be subdivided into multi-level degradation (e.g., full frame -> I-frame only -> high-confidence semantic keyframe only), rather than simple binary switching.

[0143] Alternatives to intermediate frame reconstruction algorithms:

[0144] Alternative Solution 1: Two-stage reconstruction can be simplified to a single stage, such as using only one end-to-end, high-performance generative video interpolation model (such as FILM or IFRNet) to directly generate intermediate frames.

[0145] Alternative Solution 2: For the motion estimation part, block matching or feature matching can be used instead of dense optical flow. For the generative inpainting part, generative models such as GAN or VAE can be used instead of diffusion models.

[0146] Alternative Solution 3: When server computing power is extremely limited, reconstruction can be completed using only traditional frame interpolation methods (such as adaptive filtering and motion compensation interpolation). Although the effect is not as good as AI methods, it can still play a certain role.

[0147] Alternatives to quality verification methods:

[0148] Alternative solutions: In addition to SSIM and lightweight network checks, other quality metrics such as PSNR and VMAF can be used, or the cosine similarity between the features of the generated frame and the features of the key frame can be calculated for consistency checks.

[0149] System architecture alternatives:

[0150] Alternative solution: The "server-side" functionality can be deployed on edge servers, cloud virtual machines, or dedicated hardware devices. Communication between the camera and the server can utilize various methods such as 5G, Wi-Fi, and wired networks.

[0151] To implement the above embodiments, such as Figure 4 As shown, this embodiment also provides a video adaptive transmission and reconstruction device 10 based on end-to-cloud collaboration. The device 10 includes a semantic keyframe annotation module 100, a bandwidth adaptive transmission strategy module 200, a two-stage intermediate frame reconstruction module 300, a quality verification module 400, a video sequence reconstruction module 500, and a multimodal semantic analysis module 600.

[0152] The semantic keyframe annotation module 100 is used to detect semantic events in video frames and annotate keyframes using a lightweight visual model at the camera end. The keyframes include coded keyframes and semantic keyframes.

[0153] The bandwidth adaptive transmission strategy module 200 is used to dynamically switch between full frame transmission mode and key frame priority transmission mode according to the real-time network bandwidth monitoring results. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded.

[0154] The two-stage intermediate frame reconstruction module 300 is used to identify missing intervals based on the received key frame timestamps and indices, and generate missing frames. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module.

[0155] The quality verification module 400 is used to evaluate the spatiotemporal consistency and semantic rationality of the reconstructed frame. If the evaluation result does not meet the preset threshold, a feedback mechanism is triggered to request the camera to retransmit the original frame in the missing interval.

[0156] The video sequence reconstruction module 500 is used to reconstruct the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis.

[0157] The multimodal semantic analysis module 600 is used to annotate video frames in conjunction with an audio event detection algorithm. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

[0158] Furthermore, the semantic keyframe annotation module 100 described above is also used for:

[0159] A target detection model is used to analyze video frames in real time. When the confidence of a preset target category exceeds a threshold, the corresponding frame is marked as a semantic keyframe.

[0160] A scene change detection algorithm is adopted to determine scene switching by calculating the structural similarity index (SSIM) between the current frame and the previous frame, and the scene switching frame is marked as a semantic keyframe.

[0161] Furthermore, the bandwidth adaptive transmission strategy module 200 described above is also used for:

[0162] The bandwidth fluctuation index is calculated based on RTT and packet loss rate. When the fluctuation index exceeds the first preset threshold, the system switches to key frame priority transmission mode. When the fluctuation index is lower than the second preset threshold, the system switches to full frame transmission mode.

[0163] Based on the bandwidth fluctuation index, the transmission modes are subdivided into three levels: full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and are downgraded in order of priority.

[0164] Furthermore, the aforementioned two-stage intermediate frame reconstruction module 300 is also used for:

[0165] The optical flow method is used to calculate the dense motion field of the preceding and following keyframes, and a preliminary version of the missing frames is generated by forward / backward warping.

[0166] The initial version is input into the generative model along with the preceding and following keyframes for detailed repair, generating the final missing frames.

[0167] Furthermore, the aforementioned quality verification module 400 is also used for:

[0168] Calculate the SSIM index of the reconstructed frame and the preceding and following keyframes, and detect the presence of spatiotemporal flicker or semantic anomalies using a lightweight neural network;

[0169] Calculate the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determine whether to trigger the feedback mechanism based on a preset similarity threshold.

[0170] This invention discloses an AI-based intelligent optimization device for video stream stuttering. Through a method for intelligent prediction and optimization of video stream stuttering based on multi-dimensional data fusion, it can achieve multi-dimensional data collaborative modeling of network status, video content, and user behavior, significantly improving the accuracy of stuttering prediction and the real-time performance of optimization strategies, effectively reducing video playback stuttering rate and improving user experience quality.

[0171] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0172] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

Claims

1. A video adaptive transmission and reconstruction method based on end-to-cloud collaboration, characterized in that, include: S1, using the semantic keyframe annotation module at the camera end, semantic events in video frames are detected and keyframes are labeled based on a lightweight visual model. The keyframes include coded keyframes and semantic keyframes. S2, based on the real-time network bandwidth monitoring results, dynamically switch between full-frame transmission mode and key frame priority transmission mode through the bandwidth adaptive transmission strategy module. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded. S3, through the missing frame identification module in the cloud, the missing interval is identified based on the received key frame timestamp and index, and the two-stage intermediate frame reconstruction module is triggered to generate the missing frame. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module. S4. The spatiotemporal consistency and semantic rationality of the reconstructed frame are evaluated by the quality verification module. If the evaluation result does not meet the preset threshold, the feedback mechanism is triggered to request the camera to retransmit the original frame in the missing interval. S5 reassembles the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis. S6 uses a multimodal semantic analysis module and an audio event detection algorithm to annotate video frames. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

2. The method as described in claim 1, characterized in that, S1 further includes: S11, a target detection model is used to analyze video frames in real time. When the confidence of a preset target category exceeds the threshold, the corresponding frame is marked as a semantic key frame. S12 employs a scene change detection algorithm, which calculates the structural similarity index (SSIM) between the current frame and the previous frame to determine scene switching, and marks the scene switching frame as a semantic keyframe.

3. The method as described in claim 1, characterized in that, The S2 further includes: S21, calculate the bandwidth fluctuation index based on RTT and packet loss rate. When the fluctuation index exceeds the first preset threshold, switch to key frame priority transmission mode. When the fluctuation index is lower than the second preset threshold, switch to full frame transmission mode. S22, based on the bandwidth fluctuation index, subdivides the transmission mode into three levels, including full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and degrades them in order of priority.

4. The method as described in claim 1, characterized in that, The S3 further includes: S31 uses optical flow to calculate the dense motion field of the preceding and following keyframes, and generates a preliminary version of the missing frames through forward / backward warping. S32, the initial version and the input keyframes of the preceding and following keyframes are used to refine the details of the generative model and generate the final missing frames.

5. The method as described in claim 1, characterized in that, The S4 further includes: S41, calculate the SSIM index of the reconstructed frame and the preceding and following keyframes, and detect the presence of spatiotemporal flicker or semantic anomalies through a lightweight neural network; S42, calculate the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determine whether to trigger the feedback mechanism based on the preset similarity threshold.

6. A video adaptive transmission and reconstruction device based on end-to-cloud collaboration, characterized in that, include: The semantic keyframe annotation module is used to detect semantic events in video frames and label keyframes using a lightweight visual model at the camera end. The keyframes include coded keyframes and semantic keyframes. The bandwidth adaptive transmission strategy module is used to dynamically switch between full frame transmission mode and key frame priority transmission mode based on real-time network bandwidth monitoring results. In the key frame priority transmission mode, only the marked key frames and their metadata are uploaded. A two-stage intermediate frame reconstruction module is used to identify missing intervals based on the received key frame timestamps and indices, and to generate missing frames. The two-stage intermediate frame reconstruction module includes a motion estimation module and a generative repair module. The quality verification module is used to evaluate the spatiotemporal consistency and semantic rationality of the reconstructed frames. If the evaluation result does not meet the preset threshold, a feedback mechanism is triggered to request the camera to retransmit the original frames in the missing interval. The video sequence reconstruction module is used to reconstruct the received keyframes and successfully reconstructed intermediate frames into a continuous video sequence in chronological order for storage or intelligent analysis. The multimodal semantic analysis module is used to annotate video frames in conjunction with audio event detection algorithms. When a preset audio event is detected, the corresponding frame is marked as a semantic key frame and uploaded.

7. The apparatus as claimed in claim 6, characterized in that, The semantic keyframe annotation module is also used for: A target detection model is used to analyze video frames in real time. When the confidence of a preset target category exceeds a threshold, the corresponding frame is marked as a semantic keyframe. A scene change detection algorithm is adopted to determine scene switching by calculating the structural similarity index (SSIM) between the current frame and the previous frame, and the scene switching frame is marked as a semantic keyframe.

8. The apparatus as claimed in claim 6, characterized in that, The bandwidth adaptive transmission strategy module is also used for: The bandwidth fluctuation index is calculated based on RTT and packet loss rate. When the fluctuation index exceeds the first preset threshold, the system switches to key frame priority transmission mode. When the fluctuation index is lower than the second preset threshold, the system switches to full frame transmission mode. Based on the bandwidth fluctuation index, the transmission modes are subdivided into three levels: full-frame transmission mode, I-frame-only transmission mode, and high-confidence semantic keyframe transmission mode, and are downgraded in order of priority.

9. The apparatus as claimed in claim 6, characterized in that, The two-stage intermediate frame reconstruction module is also used for: The optical flow method is used to calculate the dense motion field of the preceding and following keyframes, and a preliminary version of the missing frames is generated by forward / backward warping. The initial version is input into the generative model along with the preceding and following keyframes for detailed repair, generating the final missing frames.

10. The apparatus as claimed in claim 6, characterized in that, The quality verification module is also used for: Calculate the SSIM index of the reconstructed frame and the preceding and following keyframes, and detect the presence of spatiotemporal flicker or semantic anomalies using a lightweight neural network; Calculate the feature cosine similarity between the reconstructed frame and the preceding and following keyframes, and determine whether to trigger the feedback mechanism based on a preset similarity threshold.

Images