A method, device and equipment for extracting facial features

Abstract

The embodiments of this specification disclose a facial feature extraction method, device and apparatus for privacy protection. The scheme includes: inputting a facial image of a user to be identified into an encoder to obtain a coding vector of the facial image output by the encoder. After receiving the coding vector using a decoder in a facial feature extraction model, reconstructing facial image data is output to a feature extraction model in the facial feature extraction model, so that the feature extraction model performs feature processing on the reconstructed facial image data and outputs the facial feature vector of the user to be identified.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of computer technology, and in particular, to a method, device, and apparatus for extracting facial features. Background Art

[0002] With the development of computer technology and optical imaging technology, user identification methods based on face recognition technology are becoming increasingly popular. At present, it is usually necessary to send the face image of the user to be identified collected by the client device to the server device, so that the server device can extract the face feature vector from the face image of the user to be identified, and then generate the user identification result based on the face feature vector. Since the face image of the user to be identified belongs to the user's sensitive information, this method of sending the face image of the user to be identified to other devices for feature extraction has the risk of leaking the user's sensitive information.

[0003] Based on this, how to extract user facial features while ensuring the privacy of user facial information has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a facial feature extraction method, apparatus, and device for extracting user facial features while ensuring the privacy of user facial information.

[0005] To solve the above technical problems, the embodiments of this specification are implemented as follows:

[0006] A method for extracting facial features provided by an embodiment of the present specification uses a user feature extraction model for privacy protection, wherein the user feature extraction model includes: an encoder and a facial feature extraction model, wherein the facial feature extraction model is a model obtained by locking a decoder and a feature extraction model based on a convolutional neural network, wherein the encoder and the decoder constitute an autoencoder;

[0007] The encoder is connected to the decoder in the face feature extraction model, and the decoder is connected to the feature extraction model; the method comprises:

[0008] Inputting a facial image of a user to be identified into the encoder, obtaining a coding vector of the facial image output by the encoder, wherein the coding vector is vector data obtained after characterization processing of the facial image;

[0009] After receiving the encoding vector, the decoder in the facial feature extraction model outputs the reconstructed facial image data to the feature extraction model, so that the feature extraction model can perform feature processing on the reconstructed facial image data and then output the facial feature vector of the user to be identified.

[0010] The embodiment of this specification provides a training method for a user feature extraction model for privacy protection, the method comprising:

[0011] Acquire a first training sample set, wherein the training samples in the first training sample set are face images;

[0012] Using the first training sample set to train the initial autoencoder to obtain a trained autoencoder;

[0013] Acquire a second training sample set, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the encoder in the trained autoencoder performs feature processing on the face image;

[0014] Inputting the training samples in the second training sample set into the decoder of the initial facial feature extraction model, so as to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model using the reconstructed facial image data output by the decoder, and obtain a trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder;

[0015] A user feature extraction model for privacy protection is generated according to the encoder and the trained facial feature extraction model.

[0016] The embodiment of the present specification provides a facial feature extraction device, the device uses a user feature extraction model for privacy protection, the user feature extraction model includes: an encoder and a facial feature extraction model, the facial feature extraction model is a model obtained by locking a decoder and a feature extraction model based on a convolutional neural network, wherein the encoder and the decoder form an autoencoder; the encoder is connected to the decoder in the facial feature extraction model, and the decoder is connected to the feature extraction model; the device includes:

[0017] An input module, used to input a facial image of a user to be identified into the encoder, and obtain a coding vector of the facial image output by the encoder, wherein the coding vector is vector data obtained after characterization processing of the facial image;

[0018] The facial feature vector generation module is used to enable the decoder in the facial feature extraction model to output the reconstructed facial image data to the feature extraction model after receiving the encoding vector; so that the feature extraction model can output the facial feature vector of the user to be identified after performing feature processing on the reconstructed facial image data.

[0019] The embodiment of this specification provides a training device for a user feature extraction model for privacy protection, the device comprising:

[0020] A first acquisition module, used to acquire a first training sample set, wherein the training samples in the first training sample set are face images;

[0021] A first training module, used to train the initial autoencoder using the first training sample set to obtain a trained autoencoder;

[0022] A second acquisition module is used to acquire a second training sample set, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained by performing feature processing on the face image using the encoder in the trained autoencoder;

[0023] A second training module is used to input the training samples in the second training sample set into the decoder of the initial facial feature extraction model, so as to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model by using the reconstructed facial image data output by the decoder, so as to obtain a trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder;

[0024] A user feature extraction model generation module is used to generate a user feature extraction model for privacy protection based on the encoder and the trained face feature extraction model.

[0025] A client device provided in an embodiment of the present specification includes:

[0026] at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores an image encoder and instructions executable by the at least one processor, the image encoder is an encoder in an autoencoder, and the instructions are executed by the at least one processor so that the at least one processor can:

[0027] Inputting a facial image of a user to be identified into the image encoder, and obtaining a coding vector of the facial image output by the image encoder, wherein the coding vector is vector data obtained after characterization processing of the facial image;

[0028] The encoding vector is sent to the server device so that the server device generates the facial feature vector of the user to be identified according to the encoding vector using a facial feature extraction model, wherein the facial feature extraction model is a model obtained by locking the decoder in the autoencoder and a feature extraction model based on a convolutional neural network.

[0029] A server device provided in an embodiment of this specification includes:

[0030] At least one processor; and a memory in communication with the at least one processor; wherein the memory stores a face feature extraction model, the face feature extraction model is a model obtained by locking a decoder in an autoencoder and a feature extraction model based on a convolutional neural network, and the memory also stores instructions executable by the at least one processor, the instructions are executed by the at least one processor so that the at least one processor can:

[0031] Obtaining a coding vector of a facial image of a user to be identified, wherein the coding vector is vector data obtained by performing feature processing on the facial image using an encoder in the autoencoder;

[0032] After the encoding vector is input into the decoder in the facial feature extraction model, the decoder outputs the reconstructed facial image data to the feature extraction model; so that the feature extraction model can perform feature processing on the reconstructed facial image data and then output the facial feature vector of the user to be identified.

[0033] The embodiment of this specification provides a training device for a face feature extraction model for privacy protection, including:

[0034] at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0035] Acquire a first training sample set, wherein the training samples in the first training sample set are face images;

[0036] Using the first training sample set to train the initial autoencoder to obtain a trained autoencoder;

[0037] Acquire a second training sample set, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the encoder in the trained autoencoder performs feature processing on the face image;

[0038] Inputting the training samples in the second training sample set into the decoder of the initial facial feature extraction model, so as to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model using the reconstructed facial image data output by the decoder, and obtain a trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder;

[0039] A user feature extraction model for privacy protection is generated according to the encoder and the trained facial feature extraction model.

[0040] An embodiment of this specification can achieve the following beneficial effects:

[0041] Since the transmission, storage or use of the encoding vector of the face image generated by the encoder in the autoencoder will not affect the privacy and security of the user's face information, the service provider can obtain and process the encoding vector of the face image of the user to be identified to generate the face feature vector of the user to be identified without obtaining the original face image of the user to be identified, thereby extracting the user's face feature vector on the basis of ensuring the privacy and security of the user's face information.

[0042] And because the facial feature extraction model used to extract facial feature vectors is a model obtained by locking the decoder in the autoencoder and the feature extraction model based on the convolutional neural network, the facial feature extraction model will not leak the reconstructed facial image data generated by the decoder in the autoencoder during the process of extracting the user's facial feature vector, thereby ensuring the privacy and security of the user's facial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of one or more embodiments of this specification and constitute a part of one or more embodiments of this specification. The exemplary embodiments of this specification and their descriptions are used to explain one or more embodiments of this specification and do not constitute an improper limitation on one or more embodiments of this specification. In the drawings:

[0044] Figure 1 A flowchart of a facial feature extraction method provided in an embodiment of this specification;

[0045] Figure 2 A schematic diagram of the structure of a face feature extraction model for privacy protection provided in an embodiment of this specification;

[0046] Figure 3A flowchart of a training method for a facial feature extraction model for privacy protection provided in an embodiment of this specification;

[0047] Figure 4 The embodiments of this specification provide corresponding to Figure 1 A structural schematic diagram of a facial feature extraction device;

[0048] Figure 5 The embodiments of this specification provide corresponding to Figure 3 A training device for a facial feature extraction model for privacy protection. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of one or more embodiments of this specification clearer, the technical solutions of one or more embodiments of this specification will be clearly and completely described below in combination with the specific embodiments of this specification and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this description, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of one or more embodiments of this specification.

[0050] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0051] In the prior art, when performing user identification based on face recognition technology, it is usually necessary to send the face image of the user to be identified to a service provider, so that the service provider can extract the face feature vector from the face image of the user to be identified, and perform user identification based on the face feature vector. Since this method requires the service provider to obtain, store or process the face image of the user, it is easy to affect the privacy and security of the user's face information.

[0052] At present, when extracting facial feature vectors from user face images, the user face images are usually preprocessed before extracting facial feature vectors. For example, in a face recognition method based on principal component analysis (PCA), the principal component information is first extracted from the user face image, and some detail information is discarded to generate a facial feature vector based on the principal component information. The facial feature vector generated based on this method has the problem of facial feature information loss. It can be seen that the accuracy of the facial feature vector currently extracted is also poor.

[0053] In order to solve the defects in the prior art, this solution provides the following embodiments:

[0054] Figure 1A flowchart of a facial feature extraction method provided in an embodiment of this specification is provided. The method uses a facial feature extraction model for privacy protection to extract facial feature vectors.

[0055] Figure 2 A schematic diagram of a facial feature extraction model for privacy protection provided in an embodiment of this specification is shown in FIG. Figure 2 As shown, the user feature extraction model 201 for privacy protection includes: an encoder 202 and a face feature extraction model 203, the face feature extraction model is a model obtained by locking a decoder 204 and a feature extraction model 205 based on a convolutional neural network, wherein the encoder 202 and the decoder 204 form an autoencoder. The encoder 202 is connected to the decoder 204 in the face feature extraction model 201, and the decoder 204 is connected to the feature extraction model 205.

[0056] From a procedural perspective, Figure 1 The execution subject of the process shown in can be a user face feature extraction system or a program installed on the user face feature extraction system. The user face feature extraction system can include a client device and a server device. The client device can be equipped with an encoder in a face feature extraction model for privacy protection, and the server device can be equipped with a face feature extraction model in a face feature extraction model for privacy protection.

[0057] like Figure 1 As shown, the process may include the following steps:

[0058] Step 102: Input the facial image of the user to be identified into the encoder to obtain the encoding vector of the facial image output by the encoder, where the encoding vector is vector data obtained after characterization processing is performed on the facial image.

[0059] In the embodiments of this specification, when using various applications, users usually need to register accounts at each application. When the user logs in, unlocks the registered account, or uses the registered account to make payments, etc., it is usually necessary to identify the operating user of the registered account (i.e., the user to be identified), and only after determining that the user to be identified is the authenticated user of the registered account (i.e., the designated user), the user to be identified is allowed to perform subsequent operations. Alternatively, for the scenario where the user needs to pass through the access control system, it is usually necessary to identify the user, and only after determining that the user (i.e., the user to be identified) is a whitelist user (i.e., the designated user) of the access control system, the user is allowed to pass through the access control system.

[0060] When performing user identification on a user to be identified based on face recognition technology, the client device usually needs to collect the face image of the user to be identified and use the encoder it carries to extract the encoding vector of the face image. The client device can send the encoding vector to the server device so that the server device can generate a face feature vector of the user to be identified based on the encoding vector, and then perform user identification based on the generated face feature vector of the user to be identified.

[0061] The encoder in step 102 may be an encoder in an auto encoder (AE). An auto encoder is a network model structure in deep learning, which is characterized in that the input image itself can be used as supervisory information to perform network training with the goal of reconstructing the input image, thereby achieving the purpose of encoding the input image. Since the auto encoder does not require other information other than the input image as supervisory information in network training, the training cost of the auto encoder is low and economical and practical.

[0062] An autoencoder usually includes two parts: an encoder and a decoder. The encoder in the autoencoder can be used to encode a face image to obtain a coding vector of the face image, while the decoder in the autoencoder can reconstruct the face image according to the coding vector to obtain a reconstructed face image.

[0063] Since the encoding vector of the facial image generated by the encoder in the autoencoder is vector data obtained after characterization processing of the facial image, and the encoding vector of the facial image cannot reflect the appearance information of the user to be identified, the service provider's transmission, storage and processing of the encoding vector of the facial image will not affect the security and privacy of the facial information of the user to be identified.

[0064] In the embodiments of this specification, since the autoencoder is an artificial neural network that can learn input data through unsupervised learning and can efficiently and accurately represent the input data, the facial feature information contained in the encoding vector of the facial image generated by the encoder in the autoencoder is more comprehensive and has less noise, so when extracting the facial feature vector based on the encoding vector of the facial image generated by the encoder in the autoencoder, the accuracy of the obtained facial feature vector can be improved, which is conducive to improving the accuracy of the user recognition result generated based on the facial feature vector.

[0065] In the embodiments of this specification, the facial image of the user to be identified may be a multi-channel facial image. In practical applications, when the facial image of the user to be identified collected by the user device is a single-channel facial image, the single-channel image data of the user to be identified may be determined first; a multi-channel image of the user to be identified may be generated based on the single-channel image data, so that the encoder in the autoencoder may be used to process the multi-channel facial image of the user to be identified, wherein the image data of each channel of the multi-channel facial image of the user to be identified is the same as the single-channel image data.

[0066] Step 104: After receiving the encoding vector, the decoder in the facial feature extraction model outputs the reconstructed facial image data to the feature extraction model, so that the feature extraction model performs feature processing on the reconstructed facial image data and outputs the facial feature vector of the user to be identified.

[0067] In the embodiments of the present specification, since the training goal of the autoencoder is to minimize the difference between the reconstructed face image and the original face image, and it is not used to classify the user's face, if the encoding vector of the face image extracted by the encoder in the autoencoder is directly used as the face feature vector of the user to be identified to perform user identification, the accuracy of the user identification result will be poor.

[0068] In an embodiment of the present specification, a facial feature extraction model obtained by locking a decoder in an autoencoder and a feature extraction model based on a convolutional neural network can be deployed on a server-side device. Since the decoder in the autoencoder can be used to generate reconstructed facial image data based on the encoding vector of the facial image of the user to be identified, and the feature extraction model based on the convolutional neural network can classify the reconstructed facial image data, the output vector of the feature extraction model based on the convolutional neural network can be used as the facial feature vector of the user to be identified, so as to improve the accuracy of the user identification result generated based on the facial feature vector of the user to be identified.

[0069] In the embodiment of this specification, since the feature extraction model based on the convolutional neural network in the face feature extraction model is used to extract the face feature vector from the reconstructed face image, the feature extraction model based on the convolutional neural network can be implemented by using the existing face recognition model based on the convolutional neural network, such as DeepFace, FaceNet, MTCNN, RetinaFace, etc. It can be seen that the face feature extraction model has good compatibility.

[0070] Furthermore, since the reconstructed facial image data obtained by the decoder in the facial feature extraction model after decrypting the encoded vector of the facial image of the user to be identified has a high degree of similarity with the facial image of the user to be identified, the facial feature vector of the user to be identified extracted by the feature extraction model based on the convolutional neural network has better accuracy.

[0071] In the embodiments of this specification, encryption software can be used to lock the decoder in the autoencoder and the feature extraction model based on the convolutional neural network, or the decoder in the autoencoder and the feature extraction model based on the convolutional neural network can be stored in the secure hardware module of the device, so that the user cannot read the reconstructed facial image data output by the decoder, thereby ensuring the privacy of the user's facial information. In the embodiments of this specification, there are many ways to lock the decoder in the autoencoder and the feature extraction model, which are not specifically limited. It is only necessary to ensure the security of the use of the reconstructed facial image data output by the decoder in the autoencoder.

[0072] In actual applications, when a service provider or other user obtains the read permission for the reconstructed facial image data of the user to be identified, they can also obtain the reconstructed facial image data output by the decoder in the facial feature extraction model based on the read permission, which is conducive to improving data utilization.

[0073] It should be understood that the order of some steps in the methods described in one or more embodiments of this specification can be interchanged according to actual needs, or some steps can be omitted or deleted.

[0074] Figure 1 In the method, since the service provider can extract the facial feature vector from the encoding vector of the facial image of the user to be identified, there is no need to obtain the facial image of the user to be identified, which avoids the service provider from transmitting, storing and using the facial image of the user to be identified, thereby ensuring the privacy and security of the facial information of the user to be identified.

[0075] And because the reconstructed face image data generated by the decoder in the face feature extraction model has a high similarity with the face image of the user to be identified, the face feature vector of the user to be identified extracted from the reconstructed face image by the feature extraction model based on the convolutional neural network can have better accuracy.

[0076] based on Figure 1 The method, the examples of this specification also provide some specific implementation plans of the method, which are described below.

[0077] In the embodiment of this specification, the encoder may include: an input layer, a first hidden layer and a bottleneck layer in the autoencoder, and the decoder may include: a second hidden layer and an output layer in the autoencoder.

[0078] Among them, the input layer of the encoder is connected to the first hidden layer, the first hidden layer is connected to the bottleneck layer, the bottleneck layer of the encoder is connected to the second hidden layer of the decoder, the second hidden layer is connected to the output layer, and the output layer is connected to the feature extraction model.

[0079] The input layer can be used to receive the facial image of the user to be identified.

[0080] The first hidden layer can be used to encode the face image to obtain a first feature vector.

[0081] The bottleneck layer can be used to perform dimensionality reduction processing on the first feature vector to obtain an encoding vector of the face image, and the number of dimensions of the encoding vector is smaller than the number of dimensions of the first feature vector.

[0082] The second hidden layer can be used to decode the encoded vector to obtain a second feature vector.

[0083] The output layer can be used to generate reconstructed facial image data according to the second feature vector.

[0084] In the embodiment of the present specification, since the encoder in the autoencoder needs to encode the image, and the decoder in the autoencoder needs to generate a reconstructed face image, in order to ensure the encoding effect and the decoding effect, the first hidden layer and the second hidden layer may include multiple convolutional layers, and the first hidden layer and the second hidden layer may also include a pooling layer and a fully connected layer. The bottleneck layer can be used to reduce the feature dimension. The dimension of the feature vector output by the hidden layer connected to the bottleneck layer is higher than the dimension of the feature vector output by the bottleneck layer.

[0085] In an embodiment of the present specification, the feature extraction model based on convolutional neural network may include: an input layer, a convolutional layer, a fully connected layer and an output layer; wherein the input layer is connected to the output of the decoder, the input layer is also connected to the convolutional layer, the convolutional layer is connected to the fully connected layer, and the fully connected layer is connected to the output layer.

[0086] The input layer may be used to receive the reconstructed face image data output by the decoder;

[0087] The convolution layer can be used to extract local features from the reconstructed facial image data to obtain a local facial feature vector of the user to be identified;

[0088] The fully connected layer can be used to generate the facial feature vector of the user to be identified based on the local facial feature vector.

[0089] The output layer can be used to generate a face classification result according to the face feature vector of the user to be identified output by the fully connected layer.

[0090] In an embodiment of the present specification, the facial feature vector of the user to be identified may be the output vector of a fully connected layer adjacent to the output layer; or, when there are multiple fully connected layers in a feature extraction model based on a convolutional neural network, the facial feature vector of the user to be identified may also be the output vector of a fully connected layer that is N network layers away from the output layer; no specific limitation is made to this.

[0091] In the embodiment of this specification, since the facial feature vector of the user to be identified generated in step 104 can be used in the user identification scenario, the facial feature extraction model may also include a user matching model, and the input of the user matching model may be connected to the output of the feature extraction model based on the convolutional neural network in the facial feature extraction model.

[0092] After step 104, the method may further include: allowing the user matching model to receive the facial feature vector of the user to be identified and the facial feature vector of the designated user, and generating output information indicating whether the user to be identified is the designated user based on the vector distance between the facial feature vector of the user to be identified and the facial feature vector of the designated user, wherein the facial feature vector of the designated user is obtained by processing the facial image of the designated user using the encoder and the facial feature extraction model.

[0093] In the embodiments of this specification, the vector distance between the facial feature vector of the user to be identified and the facial feature vector of the designated user can be used to represent the similarity between the facial feature vector of the user to be identified and the facial feature vector of the designated user. Specifically, when the vector distance is less than or equal to a threshold, it can be determined that the user to be identified and the designated user are the same user. When the vector distance is greater than a threshold, it can be determined that the user to be identified and the designated user are different users. The threshold can be determined according to actual needs and is not specifically limited thereto.

[0094] In the embodiments of this specification, it is possible to use Figure 1 The method in generates the face feature vector of the user to be identified and the face feature vector of the specified user. Figure 1 The user face feature vector generated by the method has good accuracy, so it is helpful to improve the accuracy of user recognition results.

[0095] Figure 3 The following is a flow chart of a method for training a face recognition model provided in an embodiment of the present specification. From a program perspective, the execution subject of the process can be a server or a program installed on the server. Figure 3 As shown, the process may include the following steps:

[0096] Step 302: Obtain a first training sample set, where the training samples in the first training sample set are face images.

[0097] In the embodiment of this specification, the training samples in the first training sample set are facial images that have been authorized to use, such as facial images in a public facial database or facial images authorized by the user, to ensure that the facial recognition model training process does not affect the privacy of the user's facial information.

[0098] In an embodiment of the present specification, the training samples in the first training sample set may be multi-channel face images. When a face image in a public face database or a face image authorized by a user is a single-channel face image, the single-channel image data of the face image may be determined first; a multi-channel image is generated based on the single-channel image data, and the multi-channel image is used as a training sample in the first training sample set. The image data of each channel of the multi-channel face image is the same as the single-channel image data, thereby ensuring the consistency of the training samples in the first training sample set.

[0099] Step 304: Train the initial autoencoder using the first training sample set to obtain a trained autoencoder.

[0100] In an embodiment of the present specification, step 304 may specifically include: for each training sample in the first training sample set, inputting the training sample into the initial autoencoder to obtain reconstructed facial image data, optimizing the model parameters of the initial autoencoder with the goal of minimizing image reconstruction loss to obtain a trained autoencoder; the image reconstruction loss is the difference value between the reconstructed facial image data and the training sample.

[0101] In the embodiment of this specification, the input layer, the first hidden layer and the bottleneck layer in the autoencoder constitute an encoder, and the second hidden layer and the output layer in the autoencoder constitute a decoder. The encoder can be used to encode a face image to obtain a coding vector of the face image. The decoder can decode the coding vector generated by the encoder to obtain a reconstructed face image. The functions of each layer of the autoencoder are similar to Figure 1 The functions of the various layers of the autoencoder mentioned in the embodiment of the method may be the same, and will not be described in detail.

[0102] Step 306: Obtain a second training sample set, where the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the encoder in the trained autoencoder performs feature processing on the face image.

[0103] In the embodiments of this specification, the training samples in the second training sample set can be vector data obtained by using the encoder in the trained autoencoder to perform feature processing on the face image of the user whose privacy needs to be protected. Among them, the users who need to be protected can be determined according to actual needs. For example, the operating users and authenticated users of the registered accounts at the application. Or, the users to be identified and the whitelist users at the access control based on face recognition technology, etc.

[0104] In the embodiment of the present specification, the encoder in the trained autoencoder can be used in advance to generate and store the training samples in the second training sample set. When executing step 306, it is only necessary to extract the training samples in the pre-generated second training sample set from the database. Since the training samples in the second training sample set stored in the database are encoding vectors of the user's facial image, and the encoding vector cannot reflect the appearance information of the user to be identified, the service provider transmits, stores and processes the training samples in the second training sample set, which will not affect the privacy of the user's facial information.

[0105] Step 308: Input the training samples in the second training sample set into the decoder of the initial facial feature extraction model, so as to use the reconstructed facial image data output by the decoder to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model to obtain the trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder.

[0106] In the embodiments of this specification, when training the initial facial feature extraction model, it is not necessary to optimize the model parameters of the decoder in the initial facial feature extraction model, but only the model parameters of the initial feature extraction model based on the convolutional neural network need to be optimized.

[0107] The step of training the initial feature extraction model based on a convolutional neural network in the initial face feature extraction model by using the reconstructed face image data output by the decoder may specifically include:

[0108] The reconstructed facial image data is classified by using the initial feature extraction model to obtain a category label prediction value of the reconstructed facial image data; a category label preset value for the reconstructed facial image data is obtained; and the model parameters of the initial feature extraction model are optimized with the goal of minimizing the classification loss, wherein the classification loss is the difference between the category label prediction value and the category label preset value.

[0109] Step 310: Generate a user feature extraction model for privacy protection based on the encoder and the trained facial feature extraction model.

[0110] In an embodiment of the present specification, the input of the encoder is used to receive a facial image of a user to be identified, the output of the encoder is connected to the input of a decoder in the trained facial feature extraction model, the output of the decoder is connected to the input of a feature extraction model based on a convolutional neural network in the trained facial feature extraction model, and the output of the feature extraction model based on a convolutional neural network is a facial feature vector of the user to be identified.

[0111] In the embodiments of this specification, the autoencoder and the initial facial feature extraction model are trained to build a facial feature extraction model for privacy protection based on the trained autoencoder and the trained initial facial feature extraction model. Since the autoencoder does not require other information other than the input image as supervision information in network training, the training cost of the facial feature extraction model for privacy protection can be reduced, which is economical and practical.

[0112] based on Figure 3 The method, the examples of this specification also provide some specific implementation plans of the method, which are described below.

[0113] In the embodiments of this specification, Figure 3 The user feature extraction model generated by the method can be applied to user recognition scenarios. After the user face feature vector is extracted using the user feature extraction model, it is usually necessary to compare the user face feature vectors to generate the final user recognition result.

[0114] Therefore, before the facial feature extraction model for privacy protection is generated in step 310, the method may further include: establishing a user matching model, wherein the user matching model is used to generate an output result indicating whether the user to be identified is the designated user according to a vector distance between a first facial feature vector of the user to be identified and a second facial feature vector of the designated user, wherein the first facial feature vector is obtained by processing the facial image of the user to be identified using the encoder and the trained facial feature extraction model, and the second facial feature vector is obtained by processing the facial image of the designated user using the encoder and the trained facial feature extraction model;

[0115] Step 310 may specifically include: generating a user feature extraction model for privacy protection composed of the encoder, the trained face feature extraction model and the user matching model.

[0116] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Figure 4 The embodiments of this specification provide corresponding to Figure 1 A structural diagram of a facial feature extraction device. The device uses a user feature extraction model for privacy protection, and the user feature extraction model may include: an encoder and a facial feature extraction model, and the facial feature extraction model is a model obtained by locking a decoder and a feature extraction model based on a convolutional neural network, wherein the encoder and the decoder form an autoencoder; the encoder is connected to the decoder in the facial feature extraction model, and the decoder is connected to the feature extraction model; the device may include:

[0117] An input module 402 may be used to input a facial image of a user to be identified into the encoder, and obtain a coding vector of the facial image output by the encoder, wherein the coding vector is vector data obtained after characterization processing of the facial image;

[0118] The facial feature vector generation module 404 can be used to enable the decoder in the facial feature extraction model to receive the encoding vector and then output the reconstructed facial image data to the feature extraction model; so that the feature extraction model can output the facial feature vector of the user to be identified after performing feature processing on the reconstructed facial image data.

[0119] Optionally, the encoder may include: an input layer, a first hidden layer and a bottleneck layer in an autoencoder, and the decoder may include: a second hidden layer and an output layer in the autoencoder; wherein the input layer of the encoder is connected to the first hidden layer, the first hidden layer is connected to the bottleneck layer, the bottleneck layer of the encoder is connected to the second hidden layer of the decoder, the second hidden layer is connected to the output layer, and the output layer is connected to the feature extraction model.

[0120] The input layer in the autoencoder can be used to receive the facial image of the user to be identified; the first hidden layer can be used to encode the facial image to obtain a first feature vector; the bottleneck layer can be used to reduce the dimension of the first feature vector to obtain an encoding vector of the facial image, and the number of dimensions of the encoding vector is smaller than the number of dimensions of the first feature vector; the second hidden layer can be used to decode the encoding vector to obtain a second feature vector; the output layer can be used to generate reconstructed facial image data based on the second feature vector.

[0121] Optionally, the feature extraction model based on convolutional neural network may include: an input layer, a convolutional layer and a fully connected layer; wherein the input layer is connected to the output of the decoder, the input layer is also connected to the convolutional layer, and the convolutional layer is connected to the fully connected layer.

[0122] The input layer of the feature extraction model based on convolutional neural network can be used to receive the reconstructed facial image data output by the decoder; the convolution layer can be used to extract local features of the reconstructed facial image data to obtain the local facial feature vector of the user to be identified; the fully connected layer is used to generate the facial feature vector of the user to be identified based on the local facial feature vector.

[0123] Optionally, the user feature extraction model may further include a user matching model, and the user matching model is connected to the feature extraction model; the device may further include:

[0124] A user matching module is used to enable the user matching model to receive the facial feature vector of the user to be identified and the facial feature vector of the designated user, and then generate output information indicating whether the user to be identified is the designated user according to the vector distance between the facial feature vector of the user to be identified and the facial feature vector of the designated user, wherein the facial feature vector of the designated user is obtained by processing the facial image of the designated user using the encoder and the facial feature extraction model.

[0125] Based on the same idea, the embodiments of this specification also provide a device corresponding to the above method. Figure 5The embodiments of this specification provide corresponding to Figure 3 A training device for a face feature extraction model for privacy protection. Figure 5 As shown, the device may include:

[0126] A first acquisition module 502 is used to acquire a first training sample set, wherein the training samples in the first training sample set are face images;

[0127] A first training module 504 is used to train the initial autoencoder using the first training sample set to obtain a trained autoencoder;

[0128] A second acquisition module 506 is used to acquire a second training sample set, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained by performing feature processing on the face image using the encoder in the trained autoencoder;

[0129] A second training module 508 is used to input the training samples in the second training sample set into the decoder of the initial facial feature extraction model, so as to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model using the reconstructed facial image data output by the decoder, and obtain a trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder;

[0130] The user feature extraction model generation module 510 is used to generate a user feature extraction model for privacy protection based on the encoder and the trained face feature extraction model.

[0131] Optionally, the first training module 502 may be specifically used for:

[0132] For each training sample in the first training sample set, the training sample is input into the initial autoencoder to obtain reconstructed facial image data; with the goal of minimizing image reconstruction loss, the model parameters of the initial autoencoder are optimized to obtain a trained autoencoder; the image reconstruction loss is the difference value between the reconstructed facial image data and the training sample.

[0133] Optionally, the reconstructed facial image data output by the decoder is used to train an initial feature extraction model based on a convolutional neural network in the initial facial feature extraction model, which may specifically include: using the initial feature extraction model to classify the reconstructed facial image data to obtain a category label prediction value for the reconstructed facial image data; obtaining a category label preset value for the reconstructed facial image data; and optimizing model parameters of the initial feature extraction model with the goal of minimizing classification loss, wherein the classification loss is the difference between the category label prediction value and the category label preset value.

[0134] Optional, Figure 5 The device may also include: a user matching model establishment module, used to establish a user matching model, the user matching model is used to generate an output result indicating whether the user to be identified is the designated user based on the vector distance between the first facial feature vector of the user to be identified and the second facial feature vector of the designated user, the first facial feature vector is obtained by processing the facial image of the user to be identified using the encoder and the trained facial feature extraction model, and the second facial feature vector is obtained by processing the facial image of the designated user using the encoder and the trained facial feature extraction model.

[0135] The user feature extraction model generation module 510 can be specifically used to: generate a user feature extraction model for privacy protection composed of the encoder, the trained face feature extraction model and the user matching model.

[0136] Based on the same idea, the embodiment of this specification also provides a client device corresponding to the above method. The client device may include:

[0137] at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores an image encoder and instructions executable by the at least one processor, the image encoder is an encoder in an autoencoder, and the instructions are executed by the at least one processor so that the at least one processor can:

[0138] The facial image of the user to be identified is input into the image encoder to obtain the encoding vector of the facial image output by the image encoder, where the encoding vector is vector data obtained after characterization processing is performed on the facial image.

[0139] The encoding vector is sent to the server device so that the server device generates the facial feature vector of the user to be identified according to the encoding vector using a facial feature extraction model, wherein the facial feature extraction model is a model obtained by locking the decoder in the autoencoder and a feature extraction model based on a convolutional neural network.

[0140] In the embodiments of the present specification, by allowing the client device to use the encoder in the autoencoder it carries to generate the encoding vector of the facial image of the user to be identified, the client device can send the encoding vector of the facial image of the user to be identified to the server device for user identification without sending the facial image of the user to be identified to the server device, thereby avoiding the transmission of the facial image of the user to be identified, thereby ensuring the privacy and security of the facial information of the user to be identified.

[0141] Based on the same idea, the embodiment of this specification also provides a server device corresponding to the above method. The server device may include:

[0142] at least one processor; and a memory in communication with the at least one processor; wherein,

[0143] The memory stores a face feature extraction model, wherein the face feature extraction model is a model obtained by locking a decoder in an autoencoder and a feature extraction model based on a convolutional neural network. The memory also stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so that the at least one processor can:

[0144] Obtaining a coding vector of a facial image of a user to be identified, wherein the coding vector is vector data obtained by performing feature processing on the facial image using an encoder in the autoencoder;

[0145] After the encoding vector is input into the decoder in the facial feature extraction model, the decoder outputs the reconstructed facial image data to the feature extraction model; so that the feature extraction model can perform feature processing on the reconstructed facial image data and then output the facial feature vector of the user to be identified.

[0146] In the embodiments of the present specification, the server-side device is enabled to generate a facial feature vector of the user to be identified based on the encoding vector of the facial image of the user to be identified based on the facial feature extraction model installed thereon, thereby allowing the server-side device to perform user identification without obtaining the facial image of the user to be identified. This not only avoids the transmission operation of the facial image of the user to be identified, but also avoids the server-side device from storing and processing the facial image of the user to be identified, thereby improving the privacy and security of the facial information of the user to be identified.

[0147] Based on the same idea, the embodiments of this specification also provide Figure 3 A training device for a face feature extraction model for privacy protection corresponding to the method. The device may include:

[0148] at least one processor; and a memory in communication with the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to:

[0149] A first training sample set is obtained, where training samples in the first training sample set are face images.

[0150] The initial autoencoder is trained using the first training sample set to obtain a trained autoencoder.

[0151] A second training sample set is obtained, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the encoder in the trained autoencoder performs feature processing on the face image.

[0152] The training samples in the second training sample set are input into the decoder of the initial facial feature extraction model, so as to use the reconstructed facial image data output by the decoder to train the initial feature extraction model based on the convolutional neural network in the initial facial feature extraction model to obtain the trained facial feature extraction model; the initial facial feature extraction model is obtained by locking the decoder and the initial feature extraction model, and the decoder is the decoder in the trained autoencoder.

[0153] A user feature extraction model for privacy protection is generated according to the encoder and the trained facial feature extraction model.

[0154] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0155] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0156] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0157] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0158] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing one or more embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0159] Those skilled in the art will appreciate that one or more embodiments of this specification may be provided as methods, systems, or computer program products. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0160] One or more embodiments of the present specification are described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to one or more embodiments of the present specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0163] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0164] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0165] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0166] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0167] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0168] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0169] The above description is only an embodiment of the present specification and is not intended to limit one or more embodiments of the present specification. For those skilled in the art, one or more embodiments of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of one or more embodiments of the present specification shall be included in the scope of the claims of one or more embodiments of the present specification.

Claims

1. A method for extracting facial features, comprising: Inputting a facial image of a user to be identified into an encoder, obtaining an encoding vector of the facial image of the user to be identified output by the encoder, wherein the encoding vector is vector data obtained after characterization processing of the facial image of the user to be identified; After receiving the coding vector, the decoder outputs the reconstructed facial image data to the feature extraction model; so that the feature extraction model can perform feature processing on the reconstructed facial image data and then output the facial feature vector of the user to be identified; the decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of the device; the encoder is connected to the decoder in the facial feature extraction model; the decoder is connected to the feature extraction model; the facial feature extraction model is a model obtained by encrypting and locking the decoder and the feature extraction model.

2. The method according to claim 1, wherein the method uses a user feature extraction model for protecting the privacy of the user's facial image; The user feature extraction model includes: The encoder and face feature extraction model; The face feature extraction model is a model obtained by encrypting and locking the decoder and the feature extraction model based on the convolutional neural network; The encoder and the decoder constitute an autoencoder; The encoder is connected to the decoder in the face feature extraction model, and the decoder is connected to the feature extraction model.

3. The method according to claim 1 or 2, wherein the encoder comprises: An input layer, a first hidden layer and a bottleneck layer, the decoder comprising: a second hidden layer and an output layer; The input layer of the encoder is connected to the first hidden layer, the first hidden layer is connected to the bottleneck layer, the bottleneck layer of the encoder is connected to the second hidden layer of the decoder, the second hidden layer is connected to the output layer, and the output layer is connected to the feature extraction model; The input layer is used to receive the facial image of the user to be identified; The first hidden layer is used to encode the face image to obtain a first feature vector; The bottleneck layer is used to perform dimensionality reduction processing on the first feature vector to obtain a coding vector of the face image, where the number of dimensions of the coding vector is smaller than the number of dimensions of the first feature vector; The second hidden layer is used to decode the encoding vector to obtain a second feature vector; The output layer is used to generate reconstructed facial image data according to the second feature vector.

4. The method according to claim 3, wherein the feature extraction model based on a convolutional neural network comprises: Input layer, convolutional layer and fully connected layer; The input layer is connected to the output of the decoder, the input layer is also connected to the convolutional layer, and the convolutional layer is connected to the fully connected layer; The input layer is used to receive the reconstructed face image data output by the decoder; The convolution layer is used to extract local features from the reconstructed facial image data to obtain a local facial feature vector of the user to be identified; The fully connected layer is used to generate the facial feature vector of the user to be identified based on the local facial feature vector.

5. The method according to claim 4, wherein the feature extraction model based on the convolutional neural network further comprises an output layer, wherein the output layer is connected to the fully connected layer; the output layer is used to generate a face classification result according to the face feature vector of the user to be identified output by the fully connected layer; The facial feature vector of the user to be identified is the output vector of the fully connected layer adjacent to the output layer.

6. The method according to claim 2, wherein the user feature extraction model further comprises a user matching model, and the user matching model is connected to the feature extraction model; the method further comprises: The user matching model receives the facial feature vector of the user to be identified and the facial feature vector of the designated user, and generates output information indicating whether the user to be identified is the designated user based on the vector distance between the facial feature vector of the user to be identified and the facial feature vector of the designated user, wherein the facial feature vector of the designated user is obtained by processing the facial image of the designated user using the encoder and the facial feature extraction model.

7. The method according to claim 1, wherein the facial image of the user to be identified is a multi-channel facial image, and the image data of each channel of the multi-channel facial image of the user to be identified is the same as the single-channel image data.

8. A method for training a user feature extraction model for privacy protection, the method comprising: Using the first training sample set to train the initial autoencoder to obtain a trained autoencoder; The training samples in the first training sample set are face images; Acquire a second training sample set, wherein the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the encoder in the trained autoencoder performs feature processing on the face image; Inputting the training samples in the second training sample set into the decoder and the initial feature extraction model to obtain a trained feature extraction model; Generating a user feature extraction model for privacy protection according to the trained encoder, decoder and the trained feature extraction model; The decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of the device; the output of the encoder is connected to the input of the decoder; the output of the decoder is connected to the input of the trained feature extraction model in the trained facial feature extraction model; the trained facial feature extraction model is obtained by encrypting and locking the decoder and the trained feature extraction model.

9. The method as claimed in claim 8, wherein the facial images of the first training sample set are multi-channel facial images, and the image data of each channel of the multi-channel facial images are the same as the single-channel image data.

10. The method of claim 8, further comprising: The initial face feature extraction model is obtained by encrypting and locking the decoder and the initial feature extraction model, wherein the decoder is a decoder in the trained autoencoder; or, the decoder and the initial feature extraction model are stored in a secure hardware module; The trained facial feature extraction model is obtained by encrypting and locking the decoder and the trained feature extraction model, where the decoder is the decoder in the trained autoencoder; or the decoder and the trained feature extraction model are stored in a secure hardware module.

11. The method according to claim 8, wherein the step of training the initial autoencoder using the first training sample set to obtain the trained autoencoder comprises: For each training sample in the first training sample set, input the training sample into the initial autoencoder to obtain reconstructed facial image data; The model parameters of the initial autoencoder are optimized with the goal of minimizing the image reconstruction loss to obtain a trained autoencoder; the image reconstruction loss is the difference between the reconstructed face image data and the training sample.

12. The method according to claim 8, wherein the step of inputting the training samples in the second training sample set into the initial decoder and feature extraction model to obtain the trained decoder and feature extraction model comprises: The reconstructed facial image data output by the decoder is used to train an initial feature extraction model based on a convolutional neural network in the initial facial feature extraction model.

13. The method according to claim 12, wherein the using the reconstructed facial image data output by the decoder to train the initial feature extraction model based on a convolutional neural network in the initial facial feature extraction model specifically comprises: Using the initial feature extraction model to classify the reconstructed face image data, and obtain a category label prediction value of the reconstructed face image data; Obtaining a preset value of a category label for the reconstructed face image data; The model parameters of the initial feature extraction model are optimized with the goal of minimizing the classification loss, where the classification loss is the difference between the predicted value of the category label and the preset value of the category label.

14. The method according to claim 8, wherein the training samples in the first training sample set are facial images for which use permission has been obtained.

15. The method as claimed in claim 8, wherein the training samples in the second training sample set are vector data obtained by using the encoder to perform feature processing on facial images of users who need privacy protection.

16. The method according to claim 8, before generating the user feature extraction model for privacy protection, further comprising: Establishing a user matching model, the user matching model is used to generate an output result indicating whether the user to be identified is the designated user according to a vector distance between a first facial feature vector of the user to be identified and a second facial feature vector of the designated user, the first facial feature vector being obtained by processing a facial image of the user to be identified using the encoder and the trained facial feature extraction model, and the second facial feature vector being obtained by processing a facial image of the designated user using the encoder and the trained facial feature extraction model; The generating of the user feature extraction model for privacy protection specifically includes: A user feature extraction model for privacy protection is generated, which is composed of the encoder, the trained face feature extraction model and the user matching model.

17. A facial feature extraction device, comprising: An input module, used for inputting a facial image of a user to be identified into an encoder, and obtaining a coding vector of the facial image of the user to be identified output by the encoder, wherein the coding vector is vector data obtained after characterization processing is performed on the facial image of the user to be identified; A facial feature vector generation module is used to enable the decoder to output reconstructed facial image data to the feature extraction model after receiving the coding vector; so that the feature extraction model can output the facial feature vector of the user to be identified after characterizing the reconstructed facial image data; use encryption software to lock the decoder and the feature extraction model, or store the decoder and the feature extraction model in a secure hardware module of the device; the encoder is connected to the decoder in the facial feature extraction model; the decoder is connected to the feature extraction model; the facial feature extraction model is a model obtained by encrypting and locking the decoder and the feature extraction model.

18. A training device for a user feature extraction model for privacy protection, the device comprising: A first training module is used to train the initial autoencoder using the first training sample set to obtain a trained autoencoder; A second acquisition module, used to acquire a second training sample set; The training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the trained encoder performs feature processing on the face image; A second training module, used for inputting the training samples in the second training sample set into the decoder and the initial feature extraction model to obtain a trained decoder and feature extraction model; A user feature extraction model generation module, used to generate a user feature extraction model for privacy protection based on the trained encoder, the trained decoder and the feature extraction model; The decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of the device; the output of the encoder is connected to the input of the decoder; the output of the decoder is connected to the input of the trained feature extraction model in the trained facial feature extraction model; the trained facial feature extraction model is obtained by encrypting and locking the decoder and the trained feature extraction model.

19. A client device comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores an image encoder and instructions executable by the at least one processor, wherein the image encoder is an encoder in an autoencoder, and the instructions are executed by the at least one processor so that the at least one processor can: Inputting a facial image of a user to be identified into an encoder, obtaining an encoding vector of the facial image of the user to be identified output by the encoder, wherein the encoding vector is vector data obtained after characterization processing of the facial image of the user to be identified; Sending the encoding vector to a server device so that the server device generates a facial feature vector of the user to be identified based on the encoding vector using a facial feature extraction model; The facial feature vector is obtained by the facial feature extraction model based on the reconstructed facial image data generated according to the coding vector based on the decoder input; the decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of the device; the encoder is connected to the decoder in the facial feature extraction model; the decoder is connected to the feature extraction model; the facial feature extraction model is a model obtained by encrypting and locking the decoder and the feature extraction model.

20. A server device, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores a facial feature extraction model, and the memory also stores instructions executable by the at least one processor, wherein the instructions are executed by the at least one processor so that the at least one processor can: Obtaining a coding vector of a facial image of a user to be identified, wherein the coding vector is vector data obtained by performing feature processing on the facial image using an encoder in an autoencoder; After the coding vector is input into the decoder in the facial feature extraction model, the decoder outputs the reconstructed facial image data to the feature extraction model; so that the feature extraction model can perform feature processing on the reconstructed facial image data and then output the facial feature vector of the user to be identified; the decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of the device; the encoder is connected to the decoder in the facial feature extraction model; the decoder is connected to the feature extraction model; the facial feature extraction model is a model obtained by encrypting and locking the decoder and the feature extraction model.

21. A training device for a face feature extraction model for privacy protection, comprising: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to: Using a first training sample set to train an initial autoencoder to obtain a trained autoencoder, wherein the training samples in the first training sample set are face images; Obtain a second training sample set; the training samples in the second training sample set are encoding vectors, and the encoding vectors are vector data obtained after the trained encoder performs feature processing on the face image; Inputting the training samples in the second training sample set into the decoder and the initial feature extraction model to obtain a trained decoder and feature extraction model; A user feature extraction model for privacy protection is generated according to the trained encoder and the trained decoder and feature extraction model; the decoder and the feature extraction model are locked using encryption software, or the decoder and the feature extraction model are stored in a secure hardware module of a device; the output of the encoder is connected to the input of the decoder; the output of the decoder is connected to the input of the trained feature extraction model in the trained facial feature extraction model; the trained facial feature extraction model is obtained by encrypting and locking the decoder and the trained feature extraction model.

Images