A privacy-preserving autoencoder for machine video coding

Abstract

This invention belongs to the technical field of video coding, and specifically relates to a privacy-preserving autoencoder for machine video coding. Video coding for machine (VCM) contains sensitive information that requires privacy protection to address security vulnerabilities. This approach balances VCM performance while protecting this sensitive data. This invention introduces an autoencoder integrated with a deep learning network, using the ResNet architecture. This design blurs private details while preserving outlines, providing a high-dimensional representation that maintains privacy and VCM performance. The split position between the encoder and decoder is crucial, as it influences the balance between compression efficiency and machine task performance. This invention designs a flexible, position-adjustable autoencoder setup that achieves a harmonious trade-off between bitrate and mean average performance (MAP), applicable to various deep learning networks. This adjustment demonstrates superior performance compared to existing models.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding, and in particular relates to an autoencoder with a privacy protection function for machine video coding. Background Art

[0002] Video coding standards such as High Efficiency Video Coding (HEVC) and Versatile Video Coding (VVC) are designed primarily to address the needs of high-resolution and high-frame-rate video. These standards have become mainstream in the field of video coding, providing high-quality video compression methods to meet the growing demand for high-definition video transmission and storage. However, traditional video coding methods focus primarily on pixel-level frame representations rather than feature-level representations required for machine tasks. Recent research has begun to emphasize feature-level enhancement in VCM (video coding for machines) to bridge the gap between human vision and machine vision. Gao et al. proposed using the Resnet architecture to extract coding units (CUs) instead of traditional rate-distortion optimization (RDO) to achieve a balance between compression and VCM.

[0003] Although VCM technology has made significant progress in video coding, it often involves datasets containing sensitive information, which may lead to serious misuse or leakage due to various vulnerabilities. Data transmission from the cloud to the device may allow attackers to grab the original data containing private information, such as facial details and license plate numbers, through model inversion attacks. To protect privacy under different attacks, many recent research works use deep learning network coding methods. Le et al. use end-to-end learning image coding to deal with the delicate problem of balancing competitive loss functions. Yuan et al. use a learnable feature compression method to propose a lightweight computational method for compressing intermediate features. However, these methods must include the privacy protection function of VCM. Features using mutual information aim to protect the privacy of machine video coding, but the degree of privacy protection is low and there is a risk of leakage.

[0004] A machine-specific feature encoding method was proposed, allowing object detection at high bitrates with little impact on default accuracy while also being resistant to model inversion attacks. However, the autoencoder used in their work is fixed, reducing the ability to balance performance between compression and machine learning tasks, resulting in a sharp drop in performance on either side. The position of the autoencoder in both the encoder and decoder is fixed.

[0005] This paper uses autoencoders to create appropriate representations, aiming to protect privacy by blurring specific details while keeping overall outlines and background sharp. Specifically, a flexible and tunable autoencoder is used to separate the encoder and decoder, generate latent representations, and achieve a balance between compression and performance in machine vision tasks. Furthermore, this tunable architecture is applied to various deep learning networks to confirm the broad applicability of our approach. Summary of the Invention

[0006] The purpose of the present invention is to provide an autoencoder with privacy protection function for machine video encoding to ensure real-time performance and reduce energy consumption, and provide efficient data processing and analysis services.

[0007] The system of the present invention combines the advantages of edge computing and cloud computing, and optimizes performance and resource utilization by dividing the single-shot detection multi-frame (SSD) model [1] into two parts. The front end running on the edge device is responsible for preliminary data processing and analysis, while the complex computing tasks are handled by the back end in the cloud. This division method fully considers the energy and computing power limitations of the edge device, while utilizing the powerful processing power of the cloud to complete advanced data analysis tasks. In addition, the system also considers the connection performance between the edge and the cloud to ensure efficient and stable data transmission. Through this edge-cloud collaboration method, the present invention can provide efficient data processing and analysis services while ensuring real-time performance and reducing energy consumption.

[0008] The neural network structure involved in the present invention is:

[0009] In the context of edge-cloud collaboration framework, the SSD model is divided into two parts: one is the front-end installed on the edge device, and the other is the back-end hosted in the cloud, e.g. Figure 1 Determining the optimal point for model partitioning is a nuanced and complex task that is influenced by factors including energy constraints, the computational power of edge devices, and the nature of the connection between edge and cloud infrastructure.

[0010] In this paper, the decision about the split point is primarily influenced by information theoretic perspectives

[11] . These considerations favor a split point that is as deep as possible in the model to optimize performance. However, this tendency is counterbalanced by the limited computational and energy resources available to edge devices. These limitations suggest that it is wise to choose a shallower split point to ensure efficient operation.

[0011] Furthermore, deeper split points will require encoding and transmitting multiple feature tensors, adding another layer of complexity and resource requirements. Therefore, determining the most effective split point for a model in an edge-cloud collaborative environment requires a balanced approach that considers both information theoretic benefits and resource constraints.

[0012] like Figure 2 As shown in Figures 1 and 2, (a) and (c) are input images. (b) and (d) are visualizations of feature extraction using the proposed network, including 64 different features. The feature visualization is at the encoder output. The visualization of feature extraction from the input image shows the details of the background contours, meeting the requirements of machine task performance. Furthermore, during the extraction process using the proposed network, the background details are blurred to protect privacy. Feature visualization meets the requirements of privacy protection and prevents privacy leaks.

[0013] Flexible and adjustable position autoencoder

[0014] This paper introduces an autoencoder integrated with a deep learning network, using the ResNet architecture. The autoencoder is embedded in an SSD network and, at the split point within the deep learning network, is divided into two independent encoder and decoder components. This strategic arrangement significantly impacts bitrate performance and machine task execution. To meet diverse performance requirements, a versatile configuration of this partitioning has been carefully designed. Figure 1 A visual description of the complexity of the method of the present invention is presented.

[0015] The encoder performs preliminary feature extraction on the input image. This extracted feature data is transmitted over the network to a cloud server. The cloud server then feeds this feature data into a decoder, which decodes and further processes the features to complete object detection. The results are then returned to the edge device.

[0016] As the network progresses, information becomes sparser, making the model more resilient to inversion attacks. This depth also enhances compression efficiency, but may compromise performance on machine learning tasks. The bottleneck feature tensor is replicated and converted to an 8-bit per-element structure through a sophisticated pre-quantization mechanism. Next, it undergoes meticulous encoding using VVC. In the following stage, the encoded bitstream is expertly decoded by the VVC decoder at the cloud-side intersection. After decoding, it seamlessly enters the SSD backend, guiding it to the next stage of processing.

[0017] Every aspect, from the partitioning of the deep learning network to the processing and transformation of bottleneck feature tensors, is executed with precision to ensure optimal performance in terms of both bitrate and machine tasks in this complex dance in the edge-cloud collaborative ecosystem.

[0018] The autoencoder proposed in this invention has a structure as follows Figure 1As shown in the figure, it is embedded in the SSD network. The encoder consists of three convolutional layers, where the first convolutional layer is Conv(512,3,1), the second convolutional layer is Conv(512,3,1), and the third convolutional layer is Conv(1024,3,1). The output of the first layer jumps to the input of the third layer. The decoder also consists of three convolutional layers, where the first convolutional layer is Conv(128,3,1), the second convolutional layer is Conv(64,3,1), and the third convolutional layer is Conv(192,3,1). The output of the first layer jumps to the input of the third layer. Here, Conv(x,y,z) represents a convolutional layer, x represents the output channel, y represents the convolution kernel size, and z represents the convolution stride. For example, a convolutional layer of Conv(512,3,1) indicates that the output channel of this layer is 512, the convolution kernel size is 3, and the convolution stride is 1.

[0019] The split point corresponds to the segmentation of the SSD network. The split point is determined based on the trade-off between bit rate and mean average precision (mAP). The lower the bit rate, the better; the higher the mAP, the better. These two indicators can be used to determine the split point of the current network. The specific split point is at the fifth convolutional layer of the SSD.

[0020] Loss Function

[0021] The loss function optimizes the encoder and decoder networks through the autoencoder. The loss types include local loss and task loss. Local loss L local There are two main parts: pixel loss L p and Structural Similarity Index (SSIM) [2] loss L ssim The global loss uses the perceptual loss of VGG-16 [3] for feature extraction.

[0022] Local loss L local It is used to optimize the encoder-decoder architecture network with local information, and further optimize the local feature information and CNN branches. The local loss has two important parts: pixel loss and SSIM loss. p Calculate the difference between the input image I(x,y) and the output image O(x,y). The calculation formula is:

[0023]

[0024] SSIM calculates the structural similarity between the original image I(x,y) and the compressed image O(x,y). The higher the value, the higher the similarity. SSIM loss L ssim The formula is:

[0025] L ssim =1-SSIM(O(x,y)-I(x,y)), (2)

[0026] The local loss is the pixel loss L p and SSIM loss L ssim The sum of:

[0027] L local =L p +L ssim , (3)

[0028] The loss function consists of two main parts: local loss and task loss L task The scale parameter β controls the balance between local loss and task loss to further obtain performance trade-offs. The overall loss function can be written as:

[0029] L o =L local +β.L task .

[0030] Beneficial effects of the present invention

[0031] The effectiveness of the present invention is evaluated by comparing it with the method of directly encoding features in the 5th layer of SSD, which serves as a comparison baseline. In addition, the method outlined in the related research [1] is also combined to provide a comprehensive performance comparison perspective. See the table below for details:

[0032]

[0033] Figure 3 A comparative visual analysis involving VTM14.0 and our proposed method is presented in Figure 2. Experimental results demonstrate the effectiveness of both the original image and the proposed network. The left side shows the detection results for the original image, while the right side shows the results for our proposed method. Our proposed method effectively segments objects while simultaneously protecting privacy by blurring texture details. This dual achievement of accurate segmentation and privacy protection highlights the sophisticated design and effectiveness of our proposed method in VCM. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is the overall system diagram proposed by the present invention.

[0035] Figure 2 A block partition decision diagram is proposed for the present invention.

[0036] Figure 3 The AP50 value of the detection performance is measured under the same bit rate. DETAILED DESCRIPTION

[0037] The present invention is further described below through embodiments in conjunction with the accompanying drawings.

[0038] (1) Data preprocessing:

[0039] Download and decompress the COCO dataset. Use the preprocessing script to scale the images to 300x300 and normalize them. Split the processed images and their corresponding annotation files into training and test sets.

[0040] (2) Model training

[0041] The model is trained on a cloud server using the preprocessed COCO dataset. The training parameters are configured as follows: learning rate 0.001, batch size 32, number of training rounds 100, and Adam optimizer. The loss function includes local loss (pixel loss and SSIM loss) and task loss (perceptual loss of VGG-16), and the balance is achieved by adjusting the β parameter. During training, the model weights are saved every 10 epochs and evaluated on the validation set. It can be observed that after the input image is extracted by the model, the features will blur the sensitive privacy information in the image, such as facial details, etc. Figure 2 .

[0042] (3) Model segmentation

[0043] Based on the trade-off between bitrate and mAP, the split point is determined to be the fifth convolutional layer of the SSD. The trained model is divided into the encoder and decoder at this split point. The encoder and decoder are exported and saved separately.

[0044] (4) Marginal Reasoning

[0045] The encoder is deployed on an NVIDIA Jetson TX2 edge device. The live video stream runs on the edge device, and the encoder performs preliminary feature extraction. The extracted feature data is transmitted over the network to a cloud server.

[0046] (5) Cloud decoding

[0047] The received feature data is input into the decoder. The decoder is used on the cloud server to decode the features and further process them to complete the object detection task. The results are returned to the edge device and the detection results are displayed on the edge device. Figure 3 As shown, the left side is the detection image obtained by the original SSD, and the right side is the result image obtained by the present invention. The detection effect remains basically unchanged, but the privacy-sensitive detail information can be blurred to a great extent.

[0048] References

[0049] [1].Liu,W.,Anguelov,D.,Erhan,D.,Szegedy,C.,Reed,S.,Fu,C.Y.,&Berg,A.C.(2016).SSD:Single Shot MultiBox Detector.In European Conference on ComputerVision(ECCV)(pp.21-37).Springer,Cham.

[0050] [2].Wang,Z.,Bovik,A.C.,Sheikh,H.R.,&Simoncelli,E.P.(2004).Imagequality assessment:From error visibility to structural similarity.IEEETransactions on Image Processing,13(4),600-612.

[0051] [3].Simonyan,K.,&Zisserman,A.(2015).Very Deep Convolutional Networksfor Large-Scale Image Recognition.In International Conference on LearningRepresentations(ICLR)。

Claims

1. An autoencoder with privacy protection for machine video coding, characterized in that The single-shot multi-frame SSD model is divided into two parts to optimize performance and resource utilization. The front-end running on the edge device is responsible for preliminary data processing and analysis, while the complex computing tasks are handled by the back-end in the cloud. In the edge-cloud collaboration framework, the SSD model is divided into two parts: one is the front-end installed on the edge device, and the other is the back-end hosted in the cloud; Introducing an autoencoder integrated by a deep learning network, using the ResNet architecture; The autoencoder is embedded in the SSD network, and the split point divides the autoencoder into two independent encoder and decoder parts; The encoder is used to perform preliminary feature extraction on the input image, and the extracted feature data is transmitted to the cloud server via the network; The cloud server inputs the received feature data into the decoder; the decoder decodes the features and further processes them to complete the object detection task; the results are returned to the edge device; The encoder consists of three convolutional layers, where the first convolutional layer is Conv(512,3,1), the second convolutional layer is Conv(512,3,1), and the third convolutional layer is Conv(1024,3,1), and the output of the first layer jumps to the input of the third layer; the decoder also consists of three convolutional layers, where the first convolutional layer is Conv(128,3,1), the second convolutional layer is Conv(64,3,1), and the third convolutional layer is Conv(192,3,1); the output of the first layer jumps to the input of the third layer; here, Conv(x,y,z) represents a convolutional layer, x represents the output channel, y represents the convolution kernel size, and z represents the convolution step size; The position of the split point is determined based on the trade-off between bit rate and mean average precision (mAP). The lower the bit rate, the better, and the higher the mAP, the better. These two indicators determine that the split point of the current network is at the fifth convolutional layer of SSD.

2. The autoencoder according to claim 1, wherein The loss function optimizes the encoder and decoder networks through the autoencoder; loss types include local loss and task loss; Local loss L local There are two main parts: pixel loss L p and Structural Similarity Index (SSIM) loss L ssim ; Local loss uses VGG-16 perceptual loss for feature extraction; Local loss L local It is used to optimize the encoder-decoder network with local information, and further optimize the local feature information and CNN branches; Pixel loss L p Calculate the difference between the input image I(x,y) and the output image O(x,y). The calculation formula is: SSIM calculates the structural similarity between the original image I(x,y) and the compressed image O(x,y); the higher the value, the higher the similarity; SSIM loss L ssim The formula is: L ssim =1-SSIM(O(x,y)-I(x,y)), (2) The local loss is the pixel loss L p and SSIM loss L ssim The sum of: L local =L p +L ssim , (3) The overall loss function is written as: L o =L local +β.L task ; L local is the local loss, L task is the task loss, and β is the scale parameter used to control the balance between local loss and task loss.

3. The autoencoder according to claim 2, wherein The specific operation process is as follows: (1) Data preprocessing: Download and decompress the COCO dataset; use the preprocessing script to scale the images to 300x300 and normalize them; split the processed images and corresponding annotation files into training and test sets; (2) Model training The model was trained on a cloud server using the preprocessed COCO dataset. Training parameters were configured as follows: a learning rate of 0.001, a batch size of 32, 100 epochs, and the Adam optimizer. The loss function included a local loss (i.e., pixel loss and SSIM loss) and a task loss (i.e., the perceptual loss of VGG-16), with the β parameter being adjusted to achieve a balance. During training, the model weights were saved every 10 epochs and evaluated on the validation set. (3) Model segmentation Based on the trade-off between bitrate and mAP, the split point is determined to be the fifth convolutional layer of the SSD. The trained model is divided into the encoder and decoder at the split point. The encoder and decoder are exported and saved separately. (4) Marginal Reasoning Deploy the encoder part to the edge device NVIDIA Jetson TX2; run the real-time video stream on the edge device and use the encoder to perform preliminary feature extraction; extract The characteristic data is transmitted to the cloud server through the network; (5) Cloud decoding The received feature data is input into the decoder; the decoder is used on the cloud server to decode the features and further process them to complete the object detection task; the results are returned to the edge device, and the detection results are displayed on the edge device.

Images