Multi-feature fusion vehicle re-identification method for privacy protection

Abstract

The invention provides a privacy protection-oriented multi-feature fusion vehicle re-identification method, aims to realize privacy protection and retrieval of vehicle data in a cloud-assisted Internet of Vehicles scene, and mainly comprises the following contents: (1) providing a new self-supervised learning scheme; (2) providing a multi-feature fusion vehicle feature extraction framework; and (3) proposing a privacy protection scheme in a cloud-assisted Internet of Vehicles scene.

Description

Technical Field

[0001] The present invention relates to Internet of Vehicles security, and in particular to a multi-feature fusion vehicle re-identification method for privacy protection. Background Art

[0002] Vehicles continuously collect driving data through on-board sensors. This data can be used for location tracking, traffic accident evidence collection, etc., which gave rise to the Internet of Vehicles. However, due to the limitations of storage capacity and computing power, it is difficult for on-board devices to process and store massive amounts of data. With the development of cloud technology, cloud services have been integrated into the Internet of Vehicles, and cloud-assisted Internet of Vehicles has become the main solution for collecting and processing data. However, cloud servers are generally considered to be an entity that is not fully trusted, so when driving data is sent to the cloud server, it is out of the control of the owner, which may lead to the leakage of the owner's privacy. For example, the owner's whereabouts can be analyzed through this data, resulting in the leakage of sensitive information such as the owner's home address or work address. How cloud-assisted Internet of Vehicles can protect vehicle data privacy while achieving data processing has become a challenge.

[0003] Encryption is an intuitive way to protect data. Vehicle data usually contains text, images, videos, etc. Taking image data as an example, if it is encrypted before uploading, the ability to retrieve the vehicle image data is lost, and the data cannot be processed and applied. There are still relatively few studies on vehicle re-identification methods with privacy protection. The document "Sultan A, Tahir S, Tahir H, et al. A novel image-based homomorphic approach for preserving the privacy of autonomous vehicles connected to the cloud [J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 24 (2): 1936-1948" first proposed a privacy protection scheme in the field of vehicle retrieval. By processing the driving image data taken by the vehicle, other vehicle images on the road are detected and segmented, and vehicle retrieval in the ciphertext domain is realized through pixel-level homomorphic encryption. However, this scheme has the problems of long encryption time, high storage cost, and low retrieval efficiency. Therefore, the present invention aims to propose a method for realizing efficient and low storage cost vehicle re-identification with privacy protection in the cloud-assisted vehicle network scenario, which can realize the retrieval of vehicle image data while protecting vehicle image data. Summary of the invention

[0004] This paper proposes a privacy-preserving multi-feature fusion vehicle re-identification method, which aims to realize the privacy protection and retrieval of vehicle data in the cloud-assisted Internet of Vehicles scenario. It mainly includes the following contents:

[0005] (1) Propose a new self-supervised learning scheme.

[0006] (2) A vehicle feature extraction framework with multi-feature fusion is proposed.

[0007] (3) Propose a privacy protection scheme in the cloud-assisted Internet of Vehicles scenario.

[0008] The specific contents are as follows:

[0009] (1) Propose a new self-supervised learning scheme: In order to train CNN to effectively extract local features of vehicles, a self-supervised learning scheme for predicting the position of image blocks is proposed. The input of CNN is a labeled tensor. For the input vehicle image X∈R H×W×C , split into image blocks of equal size Where H×W is the resolution of the original image, C is the number of channels of the image, P×P is the resolution of the image block, and n is the number of divided blocks, that is, X→{x1, x2, ..., x n}, and then convert the image block into a tensor to obtain X′→{x′1, x′2, ..., x′ n}, for the labels corresponding to the image blocks, they are marked from 1 to n in order from left to right and from top to bottom, that is,

[0010]

[0011] The tensor and the corresponding label are input into CNN for training. The specific process and network structure are as follows Figure 1 As shown. The output of CNN is the predicted image position. During the training process, the CNN model is trained on the input image block {x′1, x′2, ..., x′ n}Extract features, i.e. f i =F self (x i )i∈(1,n), and obtain the corresponding features {f1,f2,...,f n}, and then the position prediction is performed through the fully connected layer. The calculation of this process can be expressed as:

[0012]

[0013] where f i Extract x′ for CNN i (Tag is l i) feature representation. Some traditional self-supervised learning methods, such as jigsaw puzzles, cannot extract effective local features. The self-supervised scheme proposed in the present invention is simple to implement, and in the training of the vehicle feature extraction model, no additional vehicle local feature annotation is required. CNN shows a good ability to extract local features.

[0014] (2) Propose a vehicle feature extraction framework with multi-feature fusion: Based on the proposed self-supervision scheme, a vehicle feature extraction framework is implemented. The framework mainly consists of two branches: global feature branch and local feature branch. The specific framework structure is as follows: Figure 2 As shown. The input original vehicle image of this framework is defined as X∈R H×W×C , where H is the image height, W is the image width, and C is the number of channels of the image. Then the image X is input into both branches at the same time.

[0015] In the local branch part, it consists of a self-supervision module, a local feature extraction module and a self-attention module. In the self-supervision module, the input original image X is self-supervised to obtain the input image block X′→{x′1, x′2, ..., x′ n}, ResNet18 is used as a shared feature extractor to extract the corresponding feature f i =F self (x i )i∈(1,n), and perform model training according to Formula 3:

[0016] L=E X [CE{P(x i ), l i}] (3)

[0017] Among them, CE is the cross entropy loss function. During the training process, the ResNet18 is also used as a local feature extraction module to extract local features of the vehicle. For the input image X, data enhancement processing such as rotation and scaling is performed to obtain As the input of the local feature extraction module, the corresponding feature map is obtained by ResNet 18 In order to make the model focus on more critical information, the feature map f l After the self-attention module as local feature representation.

[0018] The global branch consists of a global feature extraction module. This module uses Vision Transformer to capture global dependencies and context information in the image and extract global features that effectively represent the vehicle. The input image S is passed through the global feature extraction module to obtain f g =F global (X) is represented as a global feature.

[0019] Finally, the features are fused by splicing, and the features extracted by this framework are expressed as

[0020]

[0021] In order to optimize the entire framework, the CE loss of local branches and global branches, the hard mining triplet loss (Tri), and the self-supervised smooth cross entropy loss (SCE) are combined as the final loss function, which is calculated as follows:

[0022]

[0023] In the absence of local information annotation of vehicle images, the features extracted by this framework can effectively characterize vehicle images for vehicle re-identification and vehicle image retrieval, while ensuring accuracy and reducing manual annotation costs.

[0024] (3) A privacy protection scheme for cloud-assisted Internet of Vehicles is proposed: a privacy protection scheme for cloud-assisted Internet of Vehicles is implemented by combining vehicle re-identification with encrypted retrieval. The system framework is as follows: Figure 3 As shown in the figure, the system architecture mainly consists of three entities: data owner, cloud server and data user. The data owner owns the driving data recorded by the vehicle-mounted device, which includes the road conditions and surrounding environment when the vehicle is driving. Before uploading these data to the cloud server, the pre-trained vehicle feature extraction model extracts features and uses these features to build an index. Then, the encrypted index and image are uploaded to the cloud server for query. At the same time, the data owner needs to assign keys to data users for querying and decrypting images. When the image user needs to retrieve and locate a specific vehicle, the query vehicle image can be encrypted with the key provided by the owner to generate a query trap, and submitted to the cloud server to obtain the encrypted query result. After decryption with the decryption key, the vehicle image sorted by similarity can be obtained for locating the query vehicle. The cloud server mainly implements the storage and retrieval functions, storing encrypted data from the owner.

[0025] The specific process is as follows Figure 4 As shown, the vehicle, as the owner of the image, generates a secret key on the vehicle terminal Used to encrypt images and generate security indexes, while generating secret keys Sent to authorized users to generate query trapdoors and decrypt retrieval results. O 、M U is a (2d)×(2d)-dimensional random reversible matrix, and M O ×M U=I, I is the unit matrix, A, U and M are all random reversible matrices, the secret key pk is the image encryption key, sk is the image decryption key, and B is the defined matrix.

[0026] The vehicle obtains the original image X while driving u The vehicle terminal uses the trained vehicle feature extraction model to extract the feature F that effectively represents the vehicle image. u , convert it to a d×d dimensional matrix, and expand it to a d×2d dimensional matrix according to the formula Finally, use the random reversible matrix M and random matrix A to encrypt, that is Get the encrypted features. At the same time, use the secret key pk to encrypt the original image to get The encrypted features and images Stored together in the cloud for retrieval and processing of vehicle images.

[0027] When an authorized user needs to query a specific vehicle X q When the information is obtained, the pre-trained vehicle feature extraction model is first used to extract the retrieved vehicle feature vector F q To protect the query image, the authorized user needs to encrypt the query vehicle features. First, the retrieved vehicle feature vector is normalized, then converted into a d×d dimensional matrix, and then expanded into a d×2d dimensional matrix. Encrypt using random reversible matrix M and matrix B, that is The query trap can be obtained and uploaded to the cloud server for retrieval.

[0028] When the cloud server receives a search request from a search user, it Compare the stored image features, calculate the similarity between the encrypted query vehicle features and the stored encrypted vehicle features, sort the similarities using Formula 6, and return the k most similar encrypted images to the authorized user.

[0029]

[0030] After the authorized user gets the returned result, he decrypts it with the key sk to obtain the plaintext image The specific vehicle images being queried can be obtained. These images are taken by different vehicles on the same or different road sections and can be used for subsequent traffic accident evidence collection or location tracking.

[0031] This solution can not only protect vehicle images, that is, only users with query permissions can obtain decrypted vehicle images, but also cannot obtain plaintext images or reconstruct images through features even if the cloud is unreliable or there are malicious attackers trying to obtain information stored in the cloud.

[0032] Compared with other solutions, the advantages of the present invention are as follows:

[0033] 1. The present invention proposes a new self-supervised learning method, which is simple to implement and can improve the model's ability to extract local features of vehicles.

[0034] 2. Compared with the existing solutions, the privacy protection scheme in the cloud-assisted Internet of Vehicles scenario proposed in this paper can shorten the retrieval time and reduce the storage cost while ensuring the retrieval accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 Self-supervised learning scheme

[0036] Figure 2 Vehicle feature extraction model architecture

[0037] Figure 3 System framework diagram

[0038] Figure 4 Solution flow chart DETAILED DESCRIPTION

[0039] Before elaborating on the embodiments of the present invention, it is necessary to clarify that the purpose of these examples is only to provide a deep understanding of the essence of the invention and to demonstrate its possible application. It should be recognized that the examples provided are for the reference and follow-up of technicians. They are intended to reveal the core concepts and operating principles of the present invention and are not intended to limit the wide applicability of the invention. Through these embodiments, we hope to clearly convey the innovation and practical application potential of the present invention while retaining the protection of various equivalent implementations.

[0040] The present invention is a privacy-preserving multi-feature fusion vehicle re-identification method. The development language is Python 3.9, the deep learning framework is Pytorch 1.13.1, and the operating system is Windows 11. The specific steps are as follows:

[0041] Step 1: Prepare model training and datasets. The VeRi-776 dataset contains 51,035 images of 776 cars, of which 37,778 images of 576 cars form the training set and 13,257 images of 200 cars form the test set. The training set in the VehicleID dataset contains 221,763 images of 26,267 cars. There are three numbers of gallery images for testing, namely 800, 1,600 and 2,400. The training set is used for model training, and the test set is split into the original gallery and the query gallery in an 8:2 ratio. The original gallery is the vehicle image uploaded to the cloud server for storage, and the query gallery is the vehicle image that the authorized user wants to query.

[0042] Step 2: Build a network model. Figure 2 A network model is built, and Pytorch is used to implement a vehicle feature extraction framework according to the present invention, which mainly includes a self-supervision module, a local feature extraction module, a self-attention module, and a global feature extraction module. Its input is the original vehicle image, and its output is a vehicle feature representation, that is, a vehicle image feature that combines local and global features.

[0043] Step 3: Network model training. During the training process, the model was optimized by the Adam optimizer, with parameter values ​​β1 = 0.9, β2 = 0.999, and weight decay 5e-4. The initial learning rate was 0.0001. A batch size of 28 was used, and a total of 120 rounds of learning were performed. The images were all resized to 256×256, and the training data samples were increased by random cropping, horizontal flipping, and erasing. The loss function described in Formula 5 was optimized by Adam, and the coefficients of the loss function were simply Set to 1, where the hard mining triplet loss and CE loss are widely used in re-ID to optimize identity feature learning.

[0044] Step 4: Image owner uploads the image. Image owner generates a key And the secret key The image features in the dataset are extracted through the model trained in step 3, and then the vehicle features are normalized to obtain F′ u , which is then converted to a d×d matrix and expanded to a d×2d matrix Finally, use the random reversible matrix M and the random matrix A to encrypt and obtain the encrypted features At the same time, the original image is encrypted using the secret key pk and uploaded to the cloud server together with the encrypted features.

[0045] Step 5: The authorized user queries the image. The authorized user uses the pre-trained vehicle feature extraction model to extract the retrieved vehicle feature vector F q In order to protect the query image, the retrieval user needs to encrypt the query vehicle features. First, the retrieved vehicle feature vector is normalized, then converted into a d×d dimensional matrix, and then expanded into a d×2d dimensional matrix. Encrypt using random reversible matrix M and matrix B, that is The query trapdoor can be obtained and uploaded to the cloud server for retrieval. When receiving a retrieval request from a retrieval user, the cloud server needs to calculate the similarity between the encrypted query vehicle features and the stored encrypted vehicle features, sort the similarity using formula 5, and return the k encrypted images with the highest similarity. After receiving the retrieval results returned from the cloud server, the retrieval user uses the key sk to decrypt and obtain the plaintext image, and then obtains the image of the query vehicle, which can be used for subsequent traffic accident evidence collection or location tracking.

[0046] In summary, for the cloud-assisted Internet of Vehicles scenario, the present invention designs a solution to achieve vehicle re-identification in the ciphertext domain. While protecting vehicle privacy, vehicle data can be used for subsequent applications such as traffic accident evidence collection and location tracking.

[0047] In the above embodiments, we have described in detail the specific implementation methods of the present invention, and these embodiments are intended to help technicians better understand the principles and applications of the present invention. However, it should be emphasized that these examples are only possible implementation methods provided by the present invention and should not be regarded as limiting the scope of protection of the present invention. The core of the present invention lies in innovative ideas and technical solutions, not just these specific examples. We must recognize that the scope of protection of the present invention should be determined by the declared claims, which are intended to protect the core ideas and innovations of the present invention, rather than specific implementation methods. Therefore, any equivalent changes and modifications made on the basis of the technical solution of the present invention and in accordance with the above technical inspiration should be deemed to be included in the scope of protection of the present invention.

Claims

1. A new self-supervised learning scheme: In order to train CNN to effectively extract local features of vehicles, a self-supervised learning scheme for predicting the position of image blocks is proposed. The input of CNN is a labeled tensor. For the input vehicle image X∈R H ×W×C , split into image blocks of equal size Where H×W is the resolution of the original image, C is the number of channels of the image, P×P is the resolution of the image block, and n is the number of divided blocks. Then X→{x1,x2,…,x n }, and then convert the image block into a tensor, that is, X′→{x′1,x′2,...,x′ n }, for the labels corresponding to the image blocks, they are marked from 1 to n in order from left to right and from top to bottom, i.e. L:(l1,l2,...,l n )={1,2,...,n}, input the tensor and the corresponding label into CNN for training, and perform position prediction through the fully connected layer.

2. A vehicle feature extraction framework for multi-feature fusion according to claim 1: the framework mainly includes two branches: a global feature branch and a local feature branch, and the input original vehicle image of the framework is defined as X∈R H×W×C , where H is the image height, W is the image width, and C is the number of channels of the image. Then the image X is input into the two branches at the same time. In the local branch part, it is composed of a self-supervision module, a local feature extraction module, and a self-attention module. In the self-supervision module, the input original image X is self-supervised to obtain the input image block, and the corresponding feature f is extracted by ResNet18 as a shared feature extractor. i =F self (x i )i∈(1,n), during the training process, the ResNet18 is also used as a local feature extraction module to extract local features of the vehicle. For the input image X, data enhancement processing such as rotation and scaling is performed to obtain As the input of the local feature extraction module, the corresponding feature map is obtained by ResNet18 In order to make the model focus on more critical information, the feature map f l After the self-attention module is used as a local feature representation, the global branch part is composed of a global feature extraction module. This module uses Vision Transformer, which can be used to capture the global dependencies and contextual information in the image and extract global features that effectively represent the vehicle. The input image X is obtained as a global feature representation after passing through the global feature extraction module. Finally, the features are fused by splicing. In order to optimize the entire framework, the CE loss of the local branch and the global branch, the hard mining triplet loss (Tri) and the self-supervised smooth cross entropy loss (SCE) are combined as the final loss function. In the absence of local information annotation of vehicle images, the features extracted by this framework can effectively represent vehicle images for vehicle re-identification and vehicle image retrieval, while ensuring accuracy and reducing the cost of manual annotation.

3. A privacy protection scheme in cloud-assisted Internet of Vehicles scenario: The vehicle acts as the image owner and generates a secret key on the vehicle terminal Used to encrypt images and generate security indexes, while generating secret keys Sent to authorized users to generate query trapdoors and decrypt retrieval results, where M O 、M U is a (2d)×(2d)-dimensional random reversible matrix, and M O ×M U =I, I is the unit matrix, A, U and M are all random reversible matrices, the secret key pk is the image encryption key, sk is the image decryption key, B is the defined matrix, and the vehicle obtains the original image X during driving. u The vehicle terminal uses the trained vehicle feature extraction model to extract the feature F that effectively represents the vehicle image. u , convert it to a d×d dimensional matrix, and expand it to a d×2d dimensional matrix according to the formula Finally, use the random reversible matrix M and random matrix A to encrypt, that is, Get the encrypted features and use the secret key pk to encrypt the original image The encrypted features and images The images are stored together in the cloud for retrieval and processing of vehicle images. When an authorized user needs to query a specific vehicle X q When the information is obtained, the pre-trained vehicle feature extraction model is first used to extract the retrieved vehicle feature vector F q ,In order to protect the query image, the authorized user needs to encrypt the query vehicle features.,First, the retrieved vehicle feature vector is normalized, then converted into a d×d dimensional matrix, and then expanded into a d×2d dimensional matrix. Encrypt using random reversible matrix M and matrix B, that is The query trap can be obtained and uploaded to the cloud server for retrieval. When the cloud server receives the retrieval request from the retrieval user, it will Compare the stored image features, calculate the similarity between the encrypted query vehicle features and the stored encrypted vehicle features, sort the similarities, and return the k most similar encrypted images to the authorized user. After the authorized user gets the returned result, he decrypts it with the key sk to obtain the plaintext image The specific vehicle image being queried can be obtained. These images are taken by different vehicles on the same or different road sections and can be used for subsequent traffic accident evidence collection or location tracking. This solution can not only protect the vehicle image, that is, only users with query permissions can obtain the decrypted vehicle image, but also cannot obtain the plaintext image or reconstruct the image through features even if the cloud is unreliable or there are malicious attackers trying to obtain information stored in the cloud.

Images