A method and device for optimizing face recognition model

Abstract

A method for optimizing a face recognition model, the face recognition model comprises a backbone network and a recognition network; the method comprises: obtaining a face image sample set for continuous learning and training of a pre-trained face recognition model; the face image sample set comprises face image samples authorized by a user and unavailable face image samples; in response to the recognition network contained in the face recognition model being replaced by a classification network, the face image samples in the face image sample set are input into the face recognition model; the task objective of the recognition network is different from the task objective of the classification network; a global model loss is calculated based on the classification results corresponding to the face image samples output by the face recognition model; the global model loss comprises a first model loss corresponding to the face image samples authorized by the user, a second model loss corresponding to the unavailable face image samples, and a penalty term; and parameter adjustment is performed on each model parameter contained in the backbone network based on the global model loss.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular, to a method and device for optimizing a face recognition model. Background Art

[0002] The face recognition model is a computer vision technology that uses machine learning algorithms, especially deep learning methods, to identify and verify faces. For example, the extracted facial features can be compared with the facial features in the database to identify the individual's identity, or to verify whether the individual's identity matches the pre-registered information.

[0003] In practical applications, face recognition models usually need to rely on a large amount of real face data when training, and this face data mainly comes from public data sets and face data authorized by users.

[0004] However, if the user cancels the authorization for at least part of these facial data due to privacy concerns, it may affect the availability of the facial recognition model trained based on these facial data. Summary of the invention

[0005] This specification discloses an optimization method for a face recognition model, wherein the face recognition model comprises a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the method comprises:

[0006] Acquire a face image sample set for continuous learning and training of the pre-trained face recognition model; wherein the face image sample set includes face image samples authorized by the user and unavailable face image samples;

[0007] In response to the recognition network included in the face recognition model being replaced by a preset classification network, the face image samples in the face image sample set are input as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network; the task goal of the recognition network is different from the task goal of the classification network;

[0008] Obtain the classification result corresponding to the face image sample output by the face recognition model, and calculate the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty term constructed based on an importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the importance index is an index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter to the task objective; the penalty term is used to adjust the parameter adjustment amplitude of each model parameter based on the importance of each model parameter to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment amplitude of each model parameter;

[0009] Based on the global model loss, each model parameter included in the backbone network is adjusted to complete continuous learning and training for the face recognition model.

[0010] Optionally, the unavailable facial image samples include facial image samples whose authorization is revoked by the user; and / or facial image samples specified from facial image samples authorized by the user that need to be forgotten by the facial recognition model.

[0011] Optionally, the task objective of the recognition network is to identify whether the at least two face image samples inputted belong to the same user based on the face features extracted by the backbone network from the at least two face image samples.

[0012] Optionally, before inputting the face image samples in the face image sample set as training samples into the face recognition model, the method further includes:

[0013] In response to the recognition network included in the face recognition model being replaced by a preset classification network, model parameters of the classification layer of the recognition network are copied to the classification network.

[0014] Optionally, the importance index includes Fisher information;

[0015] Before replacing the identification network with the preset classification network, the method further includes:

[0016] The user-authorized face image samples contained in the face image sample set are input into the face recognition model for training, and based on the gradients of the various parameters contained in the backbone network that are back-propagated during the training process, the Fisher information matrix corresponding to the various parameters contained in the backbone network is calculated; wherein the Fisher information matrix is ​​a matrix composed of Fisher information quantities corresponding to the various model parameters contained in the backbone network.

[0017] Optionally, the global model loss is expressed by the following formula:

[0018]

[0019] in, represents the global model loss; represents the first model loss; represents the second model loss; represents the penalty term; represents a constant used as a weighting factor; represents the initial value of the i-th model parameter contained in the backbone network; represents the current value of the i-th model parameter before the i-th model parameter included in the backbone network is adjusted in each round of iteration of continuous learning training; represents the Fisher information corresponding to the i-th model parameter contained in the backbone network; N represents the total number of model parameters contained in the backbone network; and n is an even number greater than or equal to 2.

[0020] Optionally, the first model loss and the second model loss are represented by cross entropy loss.

[0021] Optionally, the face recognition model is a face recognition model based on ArcFace architecture.

[0022] This specification also proposes an optimization device for a face recognition model, wherein the face recognition model comprises a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the device comprises:

[0023] An acquisition module, which acquires a face image sample set used for continuous learning and training of the pre-trained face recognition model; wherein the face image sample set includes user-authorized face image samples and unavailable face image samples;

[0024] An input module, in response to the recognition network included in the face recognition model being replaced by a preset classification network, inputs the face image samples in the face image sample set as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network;

[0025] A calculation module, which obtains the classification result corresponding to the face image sample output by the face recognition model, and calculates the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty item constructed based on a preset parameter index corresponding to each model parameter contained in the backbone network; the preset parameter index is a parameter index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the penalty item is used to determine the parameter adjustment range for each model parameter based on the importance of each model parameter contained in the backbone network to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment range for each model parameter; the task objective of the recognition network is different from the task objective of the classification network;

[0026] An adjustment module adjusts the parameters of each model contained in the backbone network based on the global model loss to complete continuous learning and training for the face recognition model.

[0027] In the above embodiment, on the one hand, by replacing the recognition network of the pre-trained face recognition model with a classification network that has a completely different task objective from the recognition network, and performing continuous learning and training on the face recognition model based on a set of face image samples authorized by the user and unavailable face image samples, and in the process of continuous learning and training, introducing a second model loss corresponding to the unavailable face image samples and negatively correlated with the size of the global model loss into the global model loss, not only can the face recognition model be avoided from being retrained as a whole, but also the classification accuracy of the classification network for the unavailable face image samples can be reduced as much as possible, thereby achieving the purpose of making the face recognition model forget these unavailable face image samples;

[0028] On the other hand, a penalty item for adjusting the parameter adjustment amplitude of each model parameter is introduced into the above-mentioned global model loss based on the importance of each model parameter contained in the backbone network of the face recognition network to the task objective of the face recognition model during continuous learning and training, so that the importance of each model parameter contained in the backbone network to the task objective of the face recognition model is negatively correlated with the parameter adjustment amplitude of each model parameter contained in the backbone network. In this way, on the basis of the face recognition model forgetting the unavailable face image samples, the model parameters contained in the backbone network of the face recognition model, which are learned through training based on the face image samples authorized by the user, are ensured as much as possible without large parameter value updates, thereby maintaining the backbone network of the face recognition model's ability to parse and generalize face features to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0030] Figure 1 is a flow chart of a method for optimizing a face recognition model shown in an embodiment of this specification;

[0031] Figure 2 is a schematic structural diagram of an electronic device shown in an embodiment of this specification;

[0032] Figure 3 It is a block diagram of an optimization device for a face recognition model shown in one embodiment of this specification. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this specification.

[0034] In actual applications, for a face recognition model based on a deep learning network trained based on face data authorized by the user, if within a period of time, the user cancels the authorization for at least part of these face data due to privacy concerns, it may affect the availability of the face recognition model; for example, the face recognition model may become unavailable.

[0035] In this case, the usual practice is to use some facial data that has been authorized by the user to retrain the entire facial recognition model.

[0036] However, retraining the entire face recognition model may cause the following two problems:

[0037] On the one hand, retraining the entire face recognition model requires a huge amount of time and resources because the model parameters of the face recognition model need to be readjusted according to the new data. On the other hand, after the face recognition model is retrained, if the user changes his mind again in the future and cancels the authorization of part of the authorized face data, it will be necessary to remove that part of the data and retrain the entire face recognition model.

[0038] Based on this, this specification proposes a technical solution based on elastic weight consolidation to continuously learn the face recognition model, so as to prompt the face recognition model to forget unavailable face image samples and ensure as much as possible that the model parameters contained in the backbone network of the face recognition model do not undergo large parameter value updates.

[0039] During implementation, for a pre-trained face recognition model consisting of a backbone network and a recognition network, the recognition network can be replaced with a preset classification network; wherein the task objective of the classification network can be to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network.

[0040] After the replacement is completed, the face recognition model can continue to be trained through continuous learning based on a face image sample set consisting of face image samples authorized by the user and unavailable face image samples. The face image samples in the face image sample set are input into the face recognition model as training samples, and the classification results corresponding to the face sample set output by the face recognition model are obtained. Then, the global model loss for continuous learning training of the face recognition model is calculated based on the classification results.

[0041] Among them, the global model loss may specifically include a first model loss corresponding to a face image sample authorized by the user and a second model loss corresponding to an unavailable face image sample.

[0042] In addition, a penalty term can be introduced into the global model loss. The penalty term can be constructed based on the importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the importance index can be calculated based on the face image samples authorized by the user, and is used to evaluate the importance of each model parameter in the backbone network of the face recognition model to the task objective of the face recognition model.

[0043] The first model loss may be positively correlated with the global model loss; the second model loss may be negatively correlated with the global model loss; the penalty term is specifically used to adjust the parameter adjustment amplitude for each model parameter included in the backbone network based on the importance of each model parameter included in the backbone network to the task objective of the face recognition model during the continuous learning and training process. The importance of each model parameter included in the backbone network to the task objective of the face recognition model may be negatively correlated with the parameter adjustment amplitude for each model parameter included in the backbone network.

[0044] After calculating the global model loss for continuous learning and training of the face recognition model, the various model parameters included in the backbone network can be adjusted based on the global model loss to complete the continuous learning and training of the face recognition model.

[0045] In the above technical solution, on the one hand, by replacing the recognition network of the pre-trained face recognition model with a classification network that has a completely different task objective from the recognition network, and continuously learning and training the face recognition model based on a set of face image samples authorized by the user and unavailable face image samples, and in the process of continuous learning and training, introducing a second model loss corresponding to the unavailable face image samples and negatively correlated with the size of the global model loss into the global model loss, not only can the face recognition model be avoided from being retrained as a whole, but also the classification accuracy of the classification network for the unavailable face image samples can be reduced as much as possible, thereby achieving the purpose of making the face recognition model forget these unavailable face image samples;

[0046] On the other hand, a penalty item for adjusting the parameter adjustment amplitude of each model parameter is introduced into the above-mentioned global model loss based on the importance of each model parameter contained in the backbone network of the face recognition network to the task objective of the face recognition model during continuous learning and training, so that the importance of each model parameter contained in the backbone network to the task objective of the face recognition model is negatively correlated with the parameter adjustment amplitude of each model parameter contained in the backbone network. In this way, on the basis of the face recognition model forgetting the unavailable face image samples, the model parameters contained in the backbone network of the face recognition model, which are learned through training based on the face image samples authorized by the user, are ensured as much as possible without large parameter value updates, thereby maintaining the backbone network of the face recognition model's ability to parse and generalize face features to the greatest extent.

[0047] See also Figure 1 , Figure 1 This is a flowchart of a method for optimizing a face recognition model shown in this specification, wherein the face recognition model includes a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the method includes the following execution process:

[0048] Step 102: obtaining a face image sample set for continuous learning and training of the pre-trained face recognition model; wherein the face image sample set includes face image samples authorized by the user and unavailable face image samples;

[0049] The above-mentioned face recognition model can specifically be a face recognition model that adopts a deep learning network structure; for example, in one example, the above-mentioned face recognition model can specifically be a face recognition model based on the ArcFace network architecture.

[0050] A pre-trained face recognition model using a deep learning network structure usually includes a backbone network and a recognition network.

[0051] Among them, the above-mentioned backbone network can specifically be a network architecture based on deep learning, which can be used to extract facial features from the facial image input into the face recognition model.

[0052] For example, in practical applications, the backbone network may specifically be a network structure composed of a multi-layer convolutional neural network, through which facial feature representations can be learned and extracted from input face images.

[0053] It should be noted that the specific type of convolutional neural network used to implement the backbone network structure is not specifically limited in this specification; for example, in one example, ResNet (residual network) can be used to implement the backbone network of the face recognition model.

[0054] The recognition network may be a network architecture based on deep learning. In practical applications, the recognition network may be connected to the backbone network, and the output of the backbone network may be used as input to perform face recognition based on the facial features extracted by the backbone network. For example, in one example, the recognition network may be a multi-layer network structure including a fully connected layer and a classification layer.

[0055] In this specification, after completing the model training for the above-mentioned face recognition model based on the face data authorized by the user, in order to avoid the user canceling the authorization of at least part of these face data within a period of time, thereby affecting the availability of the face recognition model, a face image sample set for continuous learning and training of the face recognition model can be prepared in advance.

[0056] The facial image sample set may specifically include facial image samples authorized by the user and unavailable facial image samples.

[0057] In one embodiment shown, the unavailable face image samples may specifically include face image samples whose authorization is revoked by the user. In addition, in practical applications, the unavailable face image samples may also include face image samples that are specified from the face image samples authorized by the user and need to be forgotten by the face recognition model.

[0058] It should be noted that, in actual applications, the unavailable face image samples included in the above-mentioned face image sample set may include face image samples for which the user has revoked authorization, or may include face image samples specified from the face image samples authorized by the user and need to be forgotten by the face recognition model, or may also include both face image samples for which the user has revoked authorization and face image samples specified from the face image samples authorized by the user and need to be forgotten by the face recognition model.

[0059] Step 104: In response to the recognition network included in the face recognition model being replaced with a preset classification network, the face image samples in the face image sample set are input as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network;

[0060] In this specification, in order to prompt the trained face recognition model to forget unusable face image samples, the task objective of the face recognition model can be modified, and the recognition network contained in the trained face recognition model can be replaced with a preset classification network that has a completely different task objective from the recognition network.

[0061] For example, in actual applications, human intervention can be performed to manually replace the recognition network contained in the trained face recognition model with a preset classification network that has a completely different task objective from the recognition network. Alternatively, an automated configuration tool can be used to reconfigure the network structure of the face recognition model to automatically replace the recognition network contained in the trained face recognition model with a preset classification network that has a completely different task objective from the recognition network.

[0062] The task goal of the classification network may be to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network. The task goal of the recognition network before replacement included in the face recognition model may be completely different from the task goal of the classification network.

[0063] For example, in one example, as a general face recognition model, the task objective of the recognition network included in the face recognition model can usually be to identify whether the at least two face image samples inputted belong to the same user based on the face features extracted by the backbone network from the at least two face image samples.

[0064] It should be noted that the task objective of the recognition network included in the face recognition model, in addition to identifying whether the at least two face image samples belong to the same user based on the face features extracted by the backbone network from the at least two face image samples input, can also be other forms of task objectives that are completely different from the classification network after replacement, and this is not specifically limited in this specification.

[0065] After the recognition network contained in the pre-trained face recognition model is replaced with the above-mentioned preset classification network, the elastic weight solidification method can be adopted to input the face image samples obtained in the above-mentioned face image sample set as training samples into the face recognition model to perform continuous learning and training on the face recognition model.

[0066] By adopting the elastic weight solidification method for the face recognition model, it is possible to avoid retraining the entire face recognition model.

[0067] It should be explained that elastic weight solidification is a method of protecting the model parameters that have been learned by the old training tasks in the neural network when performing incremental learning on the neural network. In this specification, the training method of elastic weight solidification is used to enable the face recognition model to retain the learned model parameters as much as possible, so that the learned model parameters do not undergo a large parameter value update.

[0068] In one embodiment shown, since the recognition network before being replaced is usually also a multi-layer network structure including a classification layer, before the face image samples in the face image sample set are input into the face recognition model as training samples, in response to the recognition network included in the face recognition model being replaced by a preset classification network, the model parameters of the classification layer of the recognition network can be copied to the classification network.

[0069] In this way, the trained model parameters in the classification layer of the original recognition network can be directly transplanted to the classification network after replacement, thereby avoiding the need to adjust the model parameters in the classification network during continuous learning and training of the face recognition model, which helps to reduce the amount of parameter updates during continuous learning and training of the face recognition model.

[0070] Step 106: Obtain the classification result corresponding to the face image sample output by the face recognition model, and calculate the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty term constructed based on an importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the importance index is an index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter to the task objective; the penalty term is used to adjust the parameter adjustment amplitude of each model parameter based on the importance of each model parameter to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment amplitude of each model parameter;

[0071] In this specification, after the face image samples in the above-mentioned face image sample set are used as training samples and input into the face recognition model, and the face recognition model is continuously learned and trained, the classification results corresponding to the face image samples output by the face recognition model can be obtained, and the global model loss of the continuous learning and training of the face recognition model can be calculated based on the classification results.

[0072] Among them, the above-mentioned global model loss may specifically include a first model loss corresponding to the face image samples authorized by the user and a second model loss corresponding to the unavailable face image samples.

[0073] In one embodiment shown, the first model loss and the second model loss can be specifically represented by cross-entropy loss. Cross-entropy loss is a commonly used loss function, which is widely used in multi-category classification tasks in deep learning. Cross-entropy loss is usually used to describe the difference between the probability distribution of the classification label predicted by the model and the probability distribution of the actual classification label. In the training of the classification model, it can be used to measure the difference between the classification output by the model and the actual classification label.

[0074] Among them, the first model loss corresponding to the face image sample authorized by the user can be positively correlated with the above-mentioned global model loss; that is, when the above-mentioned first model loss increases or decreases, the above-mentioned global model loss also increases or decreases accordingly.

[0075] Since the training goal of the face recognition model is usually to minimize the global model loss; therefore, if the above-mentioned first model loss is positively correlated with the above-mentioned global model loss, the face recognition model will often produce a smaller global model loss when classifying user-authorized face image samples, thereby ensuring the classification accuracy of the face recognition model for these user-authorized face image samples.

[0076] The second model loss corresponding to the unavailable face image samples may be negatively correlated with the global model loss; that is, when the first model loss increases or decreases, the global model loss will change in the opposite direction, decreasing or increasing accordingly.

[0077] Since the training goal of the face recognition model is usually to minimize the global model loss, the first model loss is negatively correlated with the global model loss. Then, when the face recognition model classifies unavailable face image samples, a larger global model loss will often be generated. This can reduce the classification accuracy of the face recognition model for these unavailable face image samples as much as possible, so as to achieve the purpose of forgetting these unavailable face image samples.

[0078] In addition, a penalty term may be introduced into the global model loss. The penalty term may be a penalty term constructed based on the importance index of each model parameter included in the backbone network to the task objective of the face recognition model.

[0079] Among them, the importance index can specifically be an index calculated based on face image samples authorized by the user, and is used to evaluate the importance of each model parameter in the backbone network of the face recognition model to the task objective of the face recognition model.

[0080] For example, in one embodiment shown, the Fisher information of the model parameters can be used to characterize the importance of the model parameters to the task objective of the face recognition model.

[0081] In this case, before replacing the recognition network contained in the face recognition model with the above-mentioned preset classification network, the user-authorized face image samples contained in the above-mentioned face image sample set can be input into the face recognition model for training, and the gradients of the various model parameters contained in the backbone network back-propagated during the training process are used as calculation parameters to calculate the Fisher information matrix corresponding to the various parameters contained in the backbone network; wherein the Fisher information matrix is ​​a matrix composed of Fisher information quantities corresponding to the various model parameters contained in the backbone network, and when it is necessary to obtain the Fisher information quantity of a certain model parameter, the Fisher information quantity of the model parameter can be directly queried from the Fisher information matrix.

[0082] It should be noted that the specific process of using the gradients of the various model parameters contained in the backbone network as calculation parameters and calculating the Fisher information matrix corresponding to the various parameters contained in the backbone network will not be described in detail in this specification.

[0083] It should be emphasized that when calculating the Fisher information matrix, only one round of training iteration (epoch) is needed based on the user-authorized face image samples, and in the back-propagation gradient stage, the propagated gradient value is obtained, and then the Fisher information matrix is ​​calculated based on the obtained gradient value. During the whole process, there is no need to adjust the model parameters contained in the backbone network, that is, the learnable parameters in the model have not changed, and the back-propagation gradient is only for obtaining the parameters required to calculate the Fisher information matrix.

[0084] The above-mentioned penalty term can be specifically used to adjust the parameter adjustment range for each model parameter contained in the backbone network based on the importance of each model parameter to the task objective of the face recognition model during the continuous learning and training process.

[0085] Among them, it should be emphasized that in the process of using the above-mentioned penalty term to adjust the parameter adjustment range of each model parameter contained in the backbone network based on the importance of the above-mentioned model parameters to the task objective of the face recognition model, the importance of each model parameter contained in the backbone network to the task objective of the face recognition model and the parameter adjustment range of the above-mentioned model parameters will be negatively correlated; that is, when the importance of the above-mentioned model parameters increases or decreases, the parameter adjustment range of the above-mentioned model parameters will change in the completely opposite direction, decreasing or increasing accordingly.

[0086] In this way, based on the introduced penalty term, the parameter value adjustment amplitude of the model parameters contained in the backbone network that are more important to the task objectives of the face recognition model can be intervened to ensure as much as possible that these model parameters that are more important to the task objectives of the face recognition model do not undergo large parameter value updates.

[0087] In one embodiment shown, the above global model loss can be specifically expressed by the following formula:

[0088]

[0089] Among them, in the above formula, represents the global model loss; represents the first model loss; represents the second model loss; represents the penalty term; represents a constant used as a weighting factor; Represents the initial value of the i-th model parameter contained in the backbone network; Indicates the current value of the i-th model parameter contained in the backbone network before the parameter adjustment is performed on the i-th model parameter in each round of continuous learning training (that is, the parameter value after the adjustment in the previous round of iteration); represents the importance index corresponding to the i-th model parameter contained in the backbone network; N represents the total number of model parameters contained in the backbone network; where n is an even number greater than or equal to 2.

[0090] From the above formula, we can know that and Positive correlation, and Negative correlation.

[0091] In one embodiment shown, if the Fisher information of the model parameters is used to characterize the importance of the model parameters to the task objective of the face recognition model, then the above global model loss can be expressed by the following formula:

[0092]

[0093] in, Represents the Fisher information corresponding to the i-th model parameter contained in the backbone network.

[0094] It should be noted that in practical applications, the above formula of global model loss can be flexibly adjusted based on demand.

[0095] For example, in one example, It can be taken as 1 / 2, and the value of n can be taken as 2. In this case, the above global model loss formula can be expressed as:

[0096]

[0097] It should be emphasized that in practical applications, the above The values ​​of and n can be flexibly set based on requirements and will not be listed one by one in this manual.

[0098] Step 108: Adjust the parameters of each model included in the backbone network based on the global model loss to complete continuous learning and training for the face recognition model.

[0099] After calculating the global model loss for continuous learning and training of the face recognition model, the various model parameters included in the backbone network can be adjusted based on the global model loss to complete the continuous learning and training of the face recognition model. The specific implementation process of adjusting the various model parameters included in the backbone network based on the model loss will not be described in detail in this specification, and those skilled in the art can refer to the records in the relevant technology.

[0100] For example, in practical applications, in a round of training iterations, after calculating the global model loss of this round of iterations, the gradients for adjusting each model parameter can be calculated based on the global model loss, and then the calculated gradients can be back-propagated, and then each model parameter can be adjusted based on the calculated gradients.

[0101] It should be noted that, since the above-mentioned penalty term is introduced in the global model loss, in the process of parameter adjustment, the parameter value adjustment amplitude of the model parameters contained in the backbone network that are more important to the task objectives of the face recognition model can be intervened based on the introduced penalty term, so as to ensure as much as possible that these model parameters that are more important to the task objectives of the face recognition model do not undergo large parameter value updates.

[0102] For example, the above formula shows that the penalty term middle, As This means that in each round of continuous learning and training, if a model parameter is more important to the task objective of the face recognition model, the value of the penalty term corresponding to the model parameter will usually be larger.

[0103] Since the penalty term introduced in the global model loss is the cumulative value of the penalty terms corresponding to all model parameters contained in the backbone network, in the model training with minimizing the global model loss as the training goal, in order to reduce the global model loss as much as possible, for the model parameters with larger penalty terms, in this round of iterations, the value of the model parameter is usually adjusted in the direction of reducing the value of the penalty term corresponding to the model parameter.

[0104] Taking the i-th model parameter in the backbone network as an example, the penalty term of the model parameter is , since the i-th parameter Value as The coefficient of the i-th parameter If the value is higher (indicating that the model parameter is more important), then in this round of iterations, we can try to reduce the i-th parameter as much as possible. The direction of the value, that is, the direction of reducing the parameter adjustment amplitude for the i-th parameter, is used to adjust the parameter value of the model parameter.

[0105] In this way, for parameters with higher importance, under the constraint of the penalty term introduced in the global model loss, a lower parameter adjustment amplitude is usually used to adjust the parameter value of the model parameter, so as to ensure as much as possible that the model parameters with higher importance to the task objectives of the face recognition model do not undergo large parameter value updates.

[0106] In the above technical solution, on the one hand, by replacing the recognition network of the pre-trained face recognition model with a classification network that has a completely different task objective from the recognition network, and continuously learning and training the face recognition model based on a set of face image samples authorized by the user and unavailable face image samples, and in the process of continuous learning and training, introducing a second model loss corresponding to the unavailable face image samples and negatively correlated with the size of the global model loss into the global model loss, not only can the face recognition model be avoided from being retrained as a whole, but also the classification accuracy of the classification network for the unavailable face image samples can be reduced as much as possible, thereby achieving the purpose of making the face recognition model forget these unavailable face image samples;

[0107] On the other hand, a penalty item for adjusting the parameter adjustment amplitude of each model parameter is introduced into the above-mentioned global model loss based on the importance of each model parameter contained in the backbone network of the face recognition network to the task objective of the face recognition model during continuous learning and training, so that the importance of each model parameter contained in the backbone network to the task objective of the face recognition model is negatively correlated with the parameter adjustment amplitude of each model parameter contained in the backbone network. In this way, on the basis of the face recognition model forgetting the unavailable face image samples, the model parameters contained in the backbone network of the face recognition model, which are learned through training based on the face image samples authorized by the user, are ensured as much as possible without large parameter value updates, thereby maintaining the backbone network of the face recognition model's ability to parse and generalize face features to the greatest extent.

[0108] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of an apparatus, an electronic device, and a storage medium.

[0109] Figure 2 is a schematic structural diagram of an electronic device provided by an exemplary embodiment. Figure 2 At the hardware level, the device includes a processor 202, an internal bus 204, a network interface 206, a memory 208, and a non-volatile memory 210, and may also include hardware required for other services. One or more embodiments of this specification may be implemented based on software, such as the processor 202 reading the corresponding computer program from the non-volatile memory 210 into the memory 208 and then running it. Of course, in addition to the software implementation, one or more embodiments of this specification do not exclude other implementations, such as logic devices or a combination of software and hardware, etc., that is, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0110] like Figure 3 As shown, Figure 3 is a block diagram of an optimization device for a face recognition model according to an exemplary embodiment of the present specification. The device can be applied to Figure 2 In the electronic device shown in the figure, the technical solution of this specification is implemented. The face recognition model includes a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the device 300 includes:

[0111] An acquisition module 301 is configured to acquire a face image sample set for continuous learning and training of the face recognition model that has been pre-trained; wherein the face image sample set includes user-authorized face image samples and unavailable face image samples;

[0112] Input module 302, in response to the recognition network included in the face recognition model being replaced by a preset classification network, inputs the face image samples in the face image sample set as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network; the task goal of the recognition network is different from the task goal of the classification network;

[0113] The calculation module 303 obtains the classification result corresponding to the face image sample output by the face recognition model, and calculates the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty term constructed based on an importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the importance index is an index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter to the task objective; the penalty term is used to adjust the parameter adjustment amplitude for each model parameter based on the importance of each model parameter to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment amplitude for each model parameter;

[0114] The adjustment module 304 adjusts the parameters of each model included in the backbone network based on the global model loss to complete the continuous learning and training of the face recognition model.

[0115] Accordingly, the present specification also provides an electronic device, which includes a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps in all the method flows described above.

[0116] Accordingly, the present specification also provides a computer-readable storage medium on which executable computer program instructions are stored; wherein, when the instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0117] Accordingly, the present specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, the steps in all the method flows described above are implemented.

[0118] In the 1990s, it was very clear whether the improvement of a technology was hardware improvement (for example, improvement of the circuit structure of diodes, transistors, switches, etc.) or software improvement (improvement of the method flow). However, with the development of technology, many improvements of the method flow today can be regarded as direct improvements of the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that the improvement of a method flow cannot be implemented with hardware entity modules. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask chip manufacturers to design and make dedicated integrated circuit chips. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one kind of HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0119] The controller may be implemented in any suitable manner, for example, the controller may take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (e.g., software or firmware) executable by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C2051F320. The memory controller may also be implemented as part of the control logic of the memory. It is also known to those skilled in the art that, in addition to implementing the controller in a purely computer-readable program code manner, the controller may be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, such a controller may be considered as a hardware component, and the devices included therein for implementing various functions may also be considered as structures within the hardware component. Or even, the devices for implementing various functions may be considered as both software modules for implementing the method and structures within the hardware component.

[0120] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a server system. Of course, it is not excluded that with the development of computer technology in the future, the computer that implements the functions of the above embodiments may be, for example, a personal computer, a laptop computer, a vehicle-mounted human-computer interaction device, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0121] Although one or more embodiments of the present specification provide method operation steps as described in the embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative means. The order of steps listed in the embodiments is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiments or the drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The terms "include", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For example, if the words first, second, etc. are used to represent the name, they do not represent any specific order.

[0122] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing one or more of the present specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0123] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0127] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0128] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage, graphene storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0129] It should be understood by those skilled in the art that one or more embodiments of the present specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, one or more embodiments of the present specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0130] One or more embodiments of this specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of this specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0131] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples without contradiction.

[0132] The above description is only an example of one or more embodiments of this specification and is not intended to limit one or more embodiments of this specification. For those skilled in the art, one or more embodiments of this specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this specification shall be included in the scope of the claims.

Claims

1. A method for optimizing a face recognition model, wherein the face recognition model comprises a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the method comprises: Acquire a face image sample set for continuous learning and training of the pre-trained face recognition model; wherein the face image sample set includes face image samples authorized by the user and unavailable face image samples; In response to the recognition network included in the face recognition model being replaced by a preset classification network, the face image samples in the face image sample set are input as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network; the task goal of the recognition network is different from the task goal of the classification network; Obtain the classification result corresponding to the face image sample output by the face recognition model, and calculate the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty term constructed based on an importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the importance index is an index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter to the task objective; the penalty term is used to adjust the parameter adjustment amplitude of each model parameter based on the importance of each model parameter to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment amplitude of each model parameter; Based on the global model loss, each model parameter included in the backbone network is adjusted to complete continuous learning and training for the face recognition model.

2. According to the method as claimed in claim 1, the unavailable facial image samples include facial image samples whose authorization has been revoked by the user; and / or facial image samples specified from the facial image samples authorized by the user that need to be forgotten by the facial recognition model.

3. According to the method as claimed in claim 2, the task objective of the recognition network is to identify whether the at least two face image samples belong to the same user based on the face features extracted by the backbone network from the at least two face image samples input.

4. The method according to claim 1, before inputting the face image samples in the face image sample set as training samples into the face recognition model, further comprising: In response to the recognition network included in the face recognition model being replaced by a preset classification network, model parameters of the classification layer of the recognition network are copied to the classification network.

5. The method of claim 1, wherein the importance index comprises Fisher information; Before replacing the identification network with the preset classification network, the method further includes: The user-authorized face image samples contained in the face image sample set are input into the face recognition model for training, and based on the gradients of the various model parameters contained in the backbone network that are back-propagated during the training process, the Fisher information matrix corresponding to the various parameters contained in the backbone network is calculated; wherein the Fisher information matrix is ​​a matrix composed of Fisher information quantities corresponding to the various model parameters contained in the backbone network.

6. The method of claim 5, wherein the global model loss is expressed by the following formula: in, represents the global model loss; represents the first model loss; represents the second model loss; represents the penalty term; represents a constant used as a weighting factor; represents the initial value of the i-th model parameter contained in the backbone network; represents the current value of the i-th model parameter before the i-th model parameter included in the backbone network is adjusted in each round of iteration of continuous learning training; represents the Fisher information corresponding to the i-th model parameter contained in the backbone network; N represents the total number of model parameters contained in the backbone network; and n is an even number greater than or equal to 2.

7. In the method as claimed in claim 6, the first model loss and the second model loss are represented by cross entropy loss.

8. The method according to claim 1, wherein the face recognition model is a face recognition model based on ArcFace architecture.

9. An optimization device for a face recognition model, the face recognition model comprising a backbone network for extracting face features from an input face image, and a recognition network for performing face recognition based on the face features extracted by the backbone network; the device comprises: An acquisition module, which acquires a face image sample set used for continuous learning and training of the pre-trained face recognition model; wherein the face image sample set includes user-authorized face image samples and unavailable face image samples; An input module, in response to the recognition network included in the face recognition model being replaced by a preset classification network, inputs the face image samples in the face image sample set as training samples into the face recognition model; wherein the task goal of the classification network is to determine the image classification corresponding to the face image sample based on the face features extracted from the input face image sample by the backbone network; the task goal of the recognition network is different from the task goal of the classification network; A calculation module, which obtains the classification result corresponding to the face image sample output by the face recognition model, and calculates the global model loss for continuous learning and training of the face recognition model based on the classification result; wherein the global model loss includes a first model loss corresponding to the face image sample authorized by the user, a second model loss corresponding to the unavailable face image sample, and a penalty term constructed based on an importance index of each model parameter contained in the backbone network to the task objective of the face recognition model; the first model loss is positively correlated with the global model loss; the second model loss is negatively correlated with the global model loss; the importance index is an index calculated based on the face image sample authorized by the user, and is used to evaluate the importance of each model parameter to the task objective; the penalty term is used to adjust the parameter adjustment amplitude for each model parameter based on the importance of each model parameter to the task objective during the continuous learning and training process; the importance of each model parameter to the task objective is negatively correlated with the parameter adjustment amplitude for each model parameter; An adjustment module adjusts the parameters of each model contained in the backbone network based on the global model loss to complete continuous learning and training for the face recognition model.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of any one of the methods of claims 1 to 8 when executed by a processor.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images