Encrypted video target segmentation method and system based on multi-scale features
By employing multi-scale feature extraction and attention mechanisms, the accuracy and robustness issues of encrypted video target segmentation are addressed. This enables high-precision segmentation of encrypted videos without decrypting the video content, adapting to complex scene changes and meeting the security and compliance requirements of cross-border cloud computing.
Patent Information
- Application Number
- CN202512029885.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-30
- Publication Date
- 2026-05-19
AI Technical Summary
Existing technologies struggle to achieve high-precision and robust target segmentation of encrypted videos without decrypting the video content, especially in complex scenarios where segmentation detection accuracy and adaptability to scene changes are insufficient.
A multi-scale feature extraction method is adopted, including macroblock coding bit quantity, motion vector difference aggregation quantity, residual sparsity and macroblock partitioning depth, combined with single-head or multi-head attention mechanism for context enhancement representation, to achieve encrypted video target segmentation.
Without decrypting the video content, it achieves high accuracy in encrypted video target segmentation and strong robustness to complex scene changes, meeting the security and compliance requirements of cross-border cloud computing.
Smart Images
Figure CN122067150A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target segmentation technology, and in particular to a method and system for segmenting encrypted video targets based on multi-scale features. Background Technology
[0002] With the rapid development of cloud computing technology, cross-border data flow, and video sensing applications, video data is being stored and processed on a massive scale in the cloud. For privacy and data security reasons, video content is typically protected with encryption during the acquisition or uploading stage. Especially in cross-border data processing scenarios, videos are often transmitted and stored in the cloud as encrypted streams. At the same time, privacy regulations such as GDPR and PIPL impose strict restrictions on the cross-border flow and plaintext processing of personal data and sensitive video content, making the direct processing of plaintext video in the cloud a compliance risk.
[0003] However, most existing mainstream video detection and segmentation algorithms are designed based on plaintext pixel domains, relying on complete decoding and access to video content, making them difficult to apply directly to encrypted video scenarios. Adopting a "cloud download—local decryption—reprocessing" approach not only introduces additional data transmission and computational overhead, reducing processing efficiency, but also significantly increases the risk of video content leakage, failing to meet the comprehensive requirements of security, real-time performance, and compliance in cross-border cloud computing environments. Therefore, completing cloud-based video understanding tasks without decrypting video content has become a crucial technical problem that urgently needs to be solved.
[0004] For encrypted video analysis, existing research and technical solutions mainly focus on target detection or target tracking tasks. They typically extract only limited motion-related features, such as motion vectors or their statistics, from the encrypted compressed bitstream for coarse-grained target localization or presence determination. These methods reduce reliance on plaintext video to some extent, but due to the limited feature dimensions, they lack the ability to perform pixel-level segmentation tasks, and their segmentation and detection accuracy and robustness to complex scene changes remain limited. Summary of the Invention
[0005] Therefore, it is necessary to address the problem that existing encrypted video target segmentation relies solely on a single motion feature, resulting in limited accuracy and robustness, without decrypting the video content. A method and system based on multi-scale features should be provided to achieve high accuracy and strong robustness to complex scene changes in encrypted video target segmentation and detection without decrypting the video content.
[0006] To achieve the above-mentioned technical effects, the technical solution of the present invention is as follows: A method for segmenting encrypted video targets based on multi-scale features, comprising: S1. Receive encrypted video stream; S2. Perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; S3. Use the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result.
[0007] Preferably, the encrypted video stream is a video stream that has been compressed and encoded and then selectively encrypted using a format-compatible method.
[0008] Preferably, the multi-scale feature set includes: macroblock coding bit quantity, motion vector difference aggregation quantity, residual sparsity, and macroblock partitioning depth.
[0009] Preferably, the macroblock coding bit quantity The calculation expression is as follows:
[0010] Where m is a macroblock. For macroblocks The set of syntactic elements; It is the i-th syntax element instance of a macroblock.
[0011] Preferably, the motion vector difference aggregation amount The calculation expression is as follows:
[0012] Where m is a macroblock. For a certain prediction partition, For macroblock m, the set of motion vector indices This is the difference between the predicted and actual motion vectors.
[0013] Preferably, the residual sparsity The calculation expression is as follows:
[0014] Where m is a macroblock. Let m be the set of indices of all transform residual coefficients within macroblock m. To transform the residual coefficients.
[0015] Preferably, the macroblock partitioning depth The calculation expression is as follows:
[0016] Where m represents a macroblock, and mb_type represents the macroblock type. for Macroblock type, kron(.) is Kronecker multiplication. For four Sub-macroblock partitioning depth, It is a 2×2 matrix of all ones.
[0017] Preferably, the step of using the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result includes: S31. Normalize the multi-scale feature set to obtain a multi-channel feature tensor; S32. Use single-head attention or multi-head attention to extract context-enhanced representations from the multi-channel feature tensor to obtain context-enhanced feature representations; S33. Decode the context-enhanced feature representation to obtain a decoding result, and perform upsampling processing on the decoding result to obtain a target segmentation result.
[0018] Preferably, the multi-channel feature tensor is a four-channel feature tensor. The expression is as follows:
[0019] in, Let be the normalized macroblock coding bits at time t. Let be the normalized motion vector difference aggregation at time t. Let be the normalized residual sparsity at time t. Let t be the normalized macroblock partitioning depth at time t, and concat(.) is the channel concatenation operation.
[0020] This invention also proposes an encrypted video target segmentation system based on multi-scale features, comprising: The video stream receiving module is used to receive encrypted video streams; The feature extraction module is used to perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; The target segmentation module is used to segment the encrypted video stream using the multi-scale feature set to obtain the target segmentation result.
[0021] Compared with the prior art, the beneficial effects of the technical solution of the present invention are: This invention provides a method and system for target segmentation in encrypted video based on multi-scale features. First, an encrypted video stream is received to obtain the target to be processed. Then, multi-scale features are extracted from the encrypted video stream to obtain a multi-scale feature set, yielding various aspects of the video information. Finally, the multi-scale feature set is used to segment the encrypted video stream to obtain the target segmentation result. This method avoids decryption and pixel reconstruction of the video content, thus preventing the risk of plaintext exposure at the source. Furthermore, the extracted multi-scale feature set enables accurate capture of the target's shape and boundaries without relying on plaintext pixels, and effectively suppresses interference from noise and sudden scene changes. Therefore, without decrypting the video content, it achieves high precision and robustness in target segmentation of encrypted video in complex scenes. Attached Figure Description
[0022] Figure 1 This is a flowchart of the encrypted video target segmentation method based on multi-scale features in an embodiment of the present invention; Figure 2 This is a feature map of the encrypted video target segmentation method based on multi-scale features in an embodiment of the present invention; Figure 3 This is a schematic diagram of the encrypted video target segmentation method based on multi-scale features in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating video target segmentation in an embodiment of the present invention; Figure 5 This is a schematic diagram of the algorithm for calculating multi-scale feature sets in an embodiment of the present invention; Figure 6 This is a schematic diagram of the video target segmentation result in an embodiment of the present invention; Figure 7 This is a structural block diagram of an encrypted video target segmentation system based on multi-scale features in an embodiment of the present invention. Detailed Implementation
[0023] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. It is understandable to those skilled in the art that some well-known details may be omitted from the accompanying drawings; The positional relationships depicted in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. To better illustrate this embodiment, some parts of the accompanying drawings may be omitted, enlarged, or reduced, and do not represent actual dimensions. The descriptions of directions such as "up" and "down" are not intended to limit this patent. To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments.
[0024] Example 1 like Figure 1 As shown, this embodiment proposes a method for segmenting encrypted video targets based on multi-scale features, including: S1. Receive encrypted video stream; In S1, the encrypted video stream is a video stream that has been compressed and encoded and then selectively encrypted in a format-compatible manner.
[0025] Specifically, the encrypted video stream is a bitstream obtained through video encoding. Predetermined syntax elements in the bitstream undergo format-compatible selective encryption. The encryption result is then backfilled into its original position without compromising the parsability of the bitstream's syntax structure. Specifically, for any selected syntax element bit string b, ciphertext is generated under key k:
[0026] This ensures that the decoder can still parse the structural information according to the standard even without plaintext.
[0027] Specifically, existing video format-compatible encryption methods can be used, or the following steps can be included: ① Select syntax elements from the video bitstream that do not corrupt the video format for encryption, such as residual coefficients, intra-frame prediction modes, motion vectors and their differences, etc. ②Use encryption algorithms such as AES-GCM and RC4 to encrypt the selected syntax elements; ③ Replace the encrypted syntax elements back in their original positions to obtain the encrypted video stream.
[0028] S2. Perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; In S2, after the cross-border cloud server receives the encrypted stream, such as Figure 2 and Figure 3 As shown, the encrypted video stream is parsed without decryption to obtain macroblocks; then, the compressed domain information is directly parsed according to the macroblock / subblock structure to construct multi-scale features: 16×16 macroblock coding bit quantity, 8×8 macroblock partitioning depth, 4×4 subblock non-zero coefficients to approximate residual sparsity, and motion vector difference aggregation quantity.
[0029] The macroblock encoding bit count is the number of encoded bits for a macroblock in the encrypted video bitstream; the macroblock encoding bit count The calculation expression is as follows:
[0030] Where m is a macroblock. For macroblocks The set of syntactic elements; It is the i-th syntax element instance of a macroblock.
[0031] The motion vector difference aggregation amount The calculation expression is as follows:
[0032] Where m is a macroblock. For a certain prediction partition, For macroblock m, the set of motion vector indices This is the difference between the predicted and actual motion vectors.
[0033] The residual sparsity The calculation expression is as follows:
[0034] Where m is a macroblock. Let m be the set of indices of all transform residual coefficients within macroblock m. To transform the residual coefficients.
[0035] Macroblock partitioning depth The calculation expression is as follows:
[0036] Where m represents a macroblock, and mb_type represents the macroblock type. for Macroblock type, kron(.) is Kronecker multiplication. For four Sub-macroblock partitioning depth, It is a 2×2 matrix of all ones.
[0037] in, For four The partitioning depth of submacroblocks, and
[0038] in, This is a mapping function from sub-block type to depth.
[0039] S3. Use the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result.
[0040] In S3, the step of using the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result includes: S31. Normalize the multi-scale feature set to obtain a multi-channel feature tensor; In S31, the macroblock coding bit quantity, motion vector difference aggregation quantity, residual sparsity, and macroblock partitioning depth are normalized at the block level as follows to obtain the normalized macroblock coding bit quantity at time t. The normalized motion vector difference aggregation at time t Normalized residual sparsity at time t Normalized macroblock partitioning depth at time t :
[0041] in, For sliding window statistics, It is the stability constant.
[0042] The above normalized features are concatenated to form a four-channel tensor. :
[0043] The `concat(.)` function performs channel concatenation.
[0044] S32. Use single-head attention or multi-head attention to extract context-enhanced representations from the multi-channel feature tensor to obtain context-enhanced feature representations; In S32, the characteristics of several historical moments are described. The corresponding segmentation mask is encoded as a memory key-value pair (K,V); where K is the memory feature and V is the value feature, which encodes the current... Encode the query feature Q and perform scaled dot product attention through a spatiotemporal Transformer to obtain a context-enhanced feature representation. :
[0045] in, For activation function, For feature dimensions.
[0046] Alternatively, multi-head attention can be used to obtain context-enhanced feature representations. :
[0047]
[0048] Where h represents the number of attention heads. , , Let be the linear projection matrices of the query, key, and value corresponding to the i-th attention head, respectively. The output is a mapping matrix. To balance accuracy and computational cost, the network can adopt a lightweight 2-layer, 2-head (2L-2H) Transformer structure.
[0049] S33. Decode the context-enhanced feature representation to obtain a decoding result, and perform upsampling processing on the decoding result to obtain a target segmentation result.
[0050] In S33, the retrieved and aggregated features are input into the decoder, and then combined with skip-connection step-by-step upsampling to the input frame resolution to obtain a pixel-level segmentation mask. Use a threshold when necessary. Obtain the binary mask:
[0051] The cloud receives the encrypted stream and simultaneously parses the compressed domain features and updates the segmentation results in real time. In other words, it processes each part as it is received until the video ends.
[0052] The cloud only outputs structured segmentation results:
[0053] in, To segment the mask, For confidence or quality metrics, t is a timestamp.
[0054] The entire process requires no decryption of the video content, and the cloud does not output or store any plaintext video; the terminal can combine the locally decrypted video for visualization or further processing.
[0055] The principle of this embodiment is as follows: After the video terminal device compresses and encodes the captured video, a format-compatible selective encryption strategy is used to encrypt certain syntax elements in the bitstream (such as motion vector residuals, intra-frame prediction modes, residual coefficients, etc.). This can hide pixel plaintext without compromising the parsability of the bitstream's syntax structure, while maintaining the parsability of the bitstream structure. The encrypted bitstream is then uploaded to the cloud server in real time in a streaming manner. Figure 4 and Figure 5As shown, the cloud directly parses the syntax and statistical information at the macroblock / subblock level without decryption, constructing multi-scale compression domain features: using macroblock coding bit quantity to characterize texture and coding complexity, using macroblock / subblock partitioning depth to provide structural priors, using 4×4 subblock non-zero transform coefficients to count approximate residual energy, and optionally pooling motion vector residuals as displacement priors; after normalizing each channel, it concatenates them into an encrypted domain feature map, which is then input into a memory-query segmentation network trained on the encrypted feature space. This network uses the features and masks of several historical frames as memory, performs spatiotemporal Transformer retrieval and context aggregation on the current frame, and obtains pixel-level probability maps through decoding and progressive upsampling, which are then thresholded to form target masks and confidence scores, achieving pixel-level object segmentation of encrypted videos. The entire process supports streaming processing with simultaneous transmission and recognition: the segmentation results are updated as soon as the bitstream arrives; the cloud only transmits structured metadata such as masks, confidence scores and timestamps, without outputting or storing plaintext video. This embodiment meets cross-border compliance requirements such as GDPR / PIPL and can be deployed across devices and network environments. Compared with encrypted domain methods that rely solely on a single motion feature, the target segmentation accuracy and robustness are significantly improved.
[0056] Example 2 This embodiment further illustrates the results of the above embodiments.
[0057] like Figure 6 As shown, objects in the video can be accurately segmented from the encrypted video using the methods described in the embodiments.
[0058] Table 1
[0059] As shown in Table 1, the detection algorithm in this embodiment can achieve high J&F scores on different datasets. Here, J measures the degree of overlap between the predicted mask and the ground truth mask in the region, and F measures the degree of matching between the boundary of the predicted mask and the boundary of the ground truth mask.
[0060] Example 3 like Figure 7 As shown, this embodiment also proposes an encrypted video target segmentation system based on multi-scale features, including: The video stream receiving module is used to receive encrypted video streams; The feature extraction module is used to perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; The target segmentation module is used to segment the encrypted video stream using the multi-scale feature set to obtain the target segmentation result.
[0061] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is impossible to exhaustively list all the implementation methods here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.
Claims
1. A method for segmenting encrypted video targets based on multi-scale features, characterized in that, include: S1. Receive encrypted video stream; S2. Perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; S3. Use the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result.
2. The encrypted video target segmentation method based on multi-scale features according to claim 1, characterized in that, The encrypted video stream is a video stream that has been compressed and encoded, and then selectively encrypted using a format-compatible method.
3. The encrypted video target segmentation method based on multi-scale features according to claim 1, characterized in that, The multi-scale feature set includes: macroblock coding bit quantity, motion vector difference aggregation quantity, residual sparsity, and macroblock partitioning depth.
4. The encrypted video target segmentation method based on multi-scale features according to claim 3, characterized in that, The amount of macroblock coding bits The calculation expression is as follows: Where m is a macroblock. For macroblocks The set of syntactic elements; It is the i-th syntax element instance of a macroblock.
5. The encrypted video target segmentation method based on multi-scale features according to claim 3, characterized in that, The motion vector difference aggregation amount The calculation expression is as follows: Where m is a macroblock. For a certain prediction partition, For macroblock m, the set of motion vector indices This is the difference between the predicted and actual motion vectors.
6. The encrypted video target segmentation method based on multi-scale features according to claim 3, characterized in that, The residual sparsity The calculation expression is as follows: Where m is a macroblock. Let m be the set of indices of all transform residual coefficients within macroblock m. To transform the residual coefficients.
7. The encrypted video target segmentation method based on multi-scale features according to claim 3, characterized in that, Macroblock partitioning depth The calculation expression is as follows: Where m represents a macroblock, and mb_type represents the macroblock type. for Macroblock type, kron(.) is Kronecker multiplication. For four Sub-macroblock partitioning depth, It is a 2×2 matrix of all ones.
8. The encrypted video target segmentation method based on multi-scale features according to claim 1, characterized in that, The step of using the multi-scale feature set to perform target segmentation on the encrypted video stream to obtain the target segmentation result includes: S31. Normalize the multi-scale feature set to obtain a multi-channel feature tensor; S32. Use single-head attention or multi-head attention to extract context-enhanced representations from the multi-channel feature tensor to obtain context-enhanced feature representations; S33. Decode the context-enhanced feature representation to obtain a decoding result, and perform upsampling processing on the decoding result to obtain a target segmentation result.
9. The encrypted video target segmentation method based on multi-scale features according to claim 8, characterized in that, The multi-channel feature tensor is a four-channel feature tensor. The expression is as follows: in, Let be the normalized macroblock coding bits at time t. Let be the normalized motion vector difference aggregation at time t. Let be the normalized residual sparsity at time t. Let t be the normalized macroblock partitioning depth at time t, and concat(.) is the channel concatenation operation.
10. A multi-scale feature-based encrypted video target segmentation system, characterized in that, include: The video stream receiving module is used to receive encrypted video streams; The feature extraction module is used to perform multi-scale feature extraction on the encrypted video stream to obtain a multi-scale feature set; The target segmentation module is used to segment the encrypted video stream using the multi-scale feature set to obtain the target segmentation result.