Single-mode face recognition and model training method, electronic device and storage medium

Abstract

Embodiments of the present application relate to the field of face recognition, and disclose a single-modal face recognition and model training method, an electronic device and a storage medium, wherein the model training method comprises: inputting multi-modal data samples of a face into a multi-modal recognition model to obtain multi-modal face features; inputting single-modal data samples of any specified modal in the multi-modal data samples into a single-modal recognition model to obtain single-modal face features; taking the multi-modal recognition model as a teacher model, taking the single-modal recognition model as a student model, constructing a loss based on distances between feature vectors of each two multi-modal face features and distances between feature vectors of each two single-modal face features, and training the single-modal recognition model. According to the scheme, in the case that dimensions of face feature vectors output by the multi-modal recognition model and the single-modal recognition model are inconsistent, multi-modal multi-dimensional features can be compressed into single-modal single-dimensional features, and the recognition performance of a single-modal small-scale model can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of face recognition, in particular to a single-modal face recognition method and model training method, electronic equipment and storage medium. BACKGROUND

[0002] At present, it has been proved that the multi-modal face recognition model is superior to the single-modal face recognition model in general cases, and the generalization ability of the large-scale model with more parameters is superior to that of the small-scale model. However, due to the limitation of data acquisition equipment and chip processing capacity, some products can only deploy a small-scale model in a single mode, which greatly affects the recognition ability of the model. How to promote the small-scale model in a single mode to be as close as possible to the recognition ability of the large-scale model in a multi-modal mode is a difficulty.

[0003] To realize the fusion of multi-modal information in a single-modal model, the following methods can be used:

[0004] One method is to estimate other modal information using single-modal information first, and then fuse the original single-modal information and the estimated other modal information to realize multi-modal recognition. For example, in the case of only color images, the depth map (or infrared image, etc.) is estimated using the color image first, and then the color image is used in combination with the estimated depth map to realize multi-modal face recognition in a single mode. This method depends on the accuracy of depth estimation, and it is difficult to estimate the depth information from the color image; in addition, this method is a two-step method, that is, depth estimation is needed first, and then face recognition is needed, which is relatively complex.

[0005] Another method is to distill a single-modal recognition model using a multi-modal recognition model. The purpose of the knowledge distillation method is to transfer the knowledge of a large and complex model (teacher model) to a small and compact model (student model). This method generally constrains the face features extracted by the teacher model and the student model to make their features closer. This method usually requires the feature dimensions of the teacher model and the student model to be the same, which limits the distillation of the multi-modal recognition model, because the multi-modal recognition model may be composed of multiple models with different network structures and different feature dimension outputs. SUMMARY

[0006] The purpose of the embodiments of the present application is to provide a single-modal face recognition method and model training method, electronic equipment and storage medium, which can realize the compression of multi-modal multi-dimensional features to single-modal single-dimensional features in the case that the face feature vector dimensions output by the multi-modal recognition model and the single-modal recognition model are inconsistent, and improve the recognition performance of the single-modal small-scale model.

[0007] To solve the above technical problems, the embodiments of the present application provide a model training method for single-modal face recognition, comprising:

[0008] inputting the multi-modal data sample of the human face into a multi-modal recognition model to obtain multi-modal human face features;

[0009] inputting the single-modal data sample of any specified modality in the multi-modal data sample into a single-modal recognition model to obtain single-modal human face features;

[0010] taking the multi-modal recognition model as a teacher model and the single-modal recognition model as a student model, constructing a loss by using a loss function based on metric learning based on distances between feature vectors of each two multi-modal human face features and distances between feature vectors of each two single-modal human face features, and training the single-modal recognition model by knowledge distillation to obtain a trained single-modal recognition model.

[0011] The embodiment of the present application further provides a single-modal human face recognition method, comprising:

[0012] obtaining single-modal data of a human face to be recognized in a target modality;

[0013] using the single-modal human face recognition model training method as described above, using a single-modal recognition model trained in the target modality as the specified modality to perform feature extraction on the single-modal data to obtain first single-modal human face features;

[0014] comparing the first single-modal human face features with registered human face features of a registered human face in the target modality to realize identity recognition of the human face to be recognized.

[0015] The embodiment of the present application further provides an electronic device, comprising:

[0016] at least one processor; and

[0017] a memory in communication connection with the at least one processor; wherein

[0018] the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the single-modal human face recognition model training method as described above or the single-modal human face recognition method as described above.

[0019] The embodiment of the present application further provides a computer readable storage medium storing a computer program, and the computer program is executed by a processor to implement the single-modal human face recognition model training method as described above or the single-modal human face recognition method as described above.

[0020] Compared with the prior art, the embodiment of the present application transmits information of a multi-modal large-scale model to a single-modal small-scale model through knowledge distillation, and the knowledge distillation is realized through a method of metric learning, that is, metric learning is performed by calculating distances between feature vectors of each two multi-modal face features output by the multi-modal recognition model and distances between feature vectors of each two single-modal face features output by the single-modal recognition model, so that the face feature vector dimensions output by the two models do not need to be consistent, and compression of multi-modal multi-dimensional features to single-modal single-dimensional features can be realized, and the recognition performance of the single-modal small-scale model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 is a specific flowchart of a model training method for single-modal face recognition according to the embodiment of the present application;

[0022] Figure 2 is a specific flowchart of a single-modal face recognition method according to the embodiment of the present application;

[0023] Figure 3 is a structural schematic diagram of an electronic device according to the embodiment of the present application. DETAILED DESCRIPTION

[0024] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that, in the embodiments of the present application, many technical details are proposed in order to make the readers better understand the present application. However, the technical solutions claimed by the present application can be realized even without these technical details and various changes and modifications based on the following embodiments.

[0025] An embodiment of the present application relates to a model training method for single-modal face recognition, as shown in Figure 1 The model training method for single-modal face recognition provided by the embodiment includes the following steps.

[0026] Step 101: input multi-modal data samples of a face into a multi-modal recognition model to obtain multi-modal face features.

[0027] In this embodiment, the multi-modal data of the face includes, but is not limited to, any combination of different modal data in the color image, depth image, and infrared image of the face. The multi-modal recognition model is a pre-trained model for extracting features from multi-modal data of the face, and the output of the model is multi-modal face features. The multi-modal recognition model can be composed of one or more recognition models. For example, when the multi-modal recognition model is a single model, the input of the multi-modal recognition model is the data fused from multiple modal data, and the output is the multi-modal face features. When the multi-modal recognition model is a combination of multiple recognition models (also referred to as “sub-models”), each sub-model corresponds to input of feature data of one modality, and the output is the face features of the corresponding modality. It is worth mentioning that when the multi-modal recognition model is a combination of multiple sub-models, the dimensions of the face feature vectors output by the sub-models can be different. The multi-modal recognition model can be constructed using a convolutional neural network, including but not limited to a very high resolution test sequence network (VGG-Net), a residual network (ResNet), a dense network (DenseNet), etc.

[0028] In this step, the multi-modal data samples used to train the single-modal recognition model are first input into the multi-modal recognition model to obtain multi-modal face features.

[0029] In one example, when the multi-modal recognition model is a single model, inputting the multi-modal data samples into the multi-modal recognition model to obtain multi-modal face features can include: fusing each modal data sample in the multi-modal data sample to obtain a hybrid modal data sample; and inputting the hybrid modal data sample into the multi-modal recognition model to obtain the multi-modal face features.

[0030] In this example, the specific method of fusing each modal data sample is not limited. For example, multiple multi-modal face images can be fused into one hybrid modal face image by superimposition at the pixel level. Since the multi-modal recognition model is a single model, data fusion is required before the multi-modal data sample enters the multi-modal recognition model to form a data form suitable for processing by the single model, so that the output of the multi-modal recognition model is the multi-modal face features.

[0031] In another example, when the multi-modal recognition model includes multiple sub-models corresponding to each modality in the multi-modal data sample; inputting the multi-modal data sample into the multi-modal recognition model to obtain multi-modal face features can include: inputting each modal data sample in the multi-modal data sample into the sub-model corresponding to the modality to obtain discrete modal face features corresponding to each modal data sample; and obtaining the multi-modal face features by taking the discrete modal face features corresponding to each modality as a set.

[0032] In this example, each modality data sample in the multi-modal data sample does not need to be fused before entering the multi-modal recognition model, but is processed by the sub-model corresponding to each modality to obtain the face feature of the corresponding modality, also known as the discrete modality face feature. Then, without complex fusion operation of each discrete modality face feature, only the face feature vectors are regarded as a face feature set as the multi-modal face feature, thereby reducing the complexity of the model.

[0033] Step 102: inputting the single modality data sample of any specified modality in the multi-modal data sample into the single modality recognition model to obtain the single modality face feature.

[0034] Specifically, the single modality recognition model is the model to be trained in this embodiment, and the input is single modality data, which can be any one of color image, depth image, infrared image and the like, and the output is single modality face feature. This embodiment adopts convolutional neural network as the single modality recognition model.

[0035] For the data modality to be trained in the single modality recognition model to be trained, the modality data corresponding to the modality is extracted from the above multi-modal data sample as the single modality data sample of the specified modality, which is input into the single modality recognition model to obtain the single modality face feature.

[0036] That is, for a multi-modal data sample, all modality data in the sample enters the multi-modal recognition model to obtain the multi-modal face feature, and the single modality data of the specified modality in the sample enters the single modality recognition model to obtain the single modality face feature.

[0037] Step 103: taking the multi-modal recognition model as the teacher model and the single modality recognition model as the student model, constructing the loss based on the distance between the feature vectors of each two multi-modal face features and the distance between the feature vectors of each two single modality face features by using the loss function based on metric learning, and training the single modality recognition model by knowledge distillation to obtain the trained single modality recognition model.

[0038] Specifically, the model size of the student model is generally not more than that of the teacher model. In the case of taking the multi-modal recognition model as the teacher model and the single-modal recognition model as the student model, the recognition features of the multi-modal recognition model are compressed and transmitted to the single-modal recognition model, so as to improve the recognition performance of the single-modal recognition model. In this embodiment, the loss used in the process of training the single-modal recognition model is not directly based on the feature distance between the multi-modal face features and the single-modal face features, but based on the distance between the feature vectors of each two multi-modal face features and the distance between the feature vectors of each two single-modal face features. The loss is constructed by using the two distances and a loss function based on metric learning, so as to train the single-modal recognition model by using the constructed loss through knowledge distillation. That is, the distance between the feature vectors of each two single-modal face features output by the single-modal recognition model is forced to approximate the distance between the feature vectors of each two multi-modal face features output by the multi-modal recognition model. The approximation is that, the closer the distance between the feature vectors of the multi-modal face features corresponding to any two multi-modal data samples, the closer the distance between the feature vectors of the single-modal face features corresponding to the two single-modal data samples of the specified modality in the two multi-modal data samples; the farther the distance between the feature vectors of the multi-modal face features corresponding to any two multi-modal data samples, the farther the distance between the feature vectors of the single-modal face features corresponding to the two single-modal data samples of the specified modality in the two multi-modal data samples.

[0039] Since the loss used in the training process is based on the distance between the feature vectors of each two multi-modal face features and the distance between the feature vectors of each two single-modal face features, the face feature vector dimensions output by the two models do not need to be consistent. In this embodiment, the face feature vector dimensions output by the multi-modal recognition model and the single-modal recognition model can be the same or different.

[0040] The loss function used to construct the loss can be a loss function based on metric learning, such as contrastive loss, triplet loss, quadruplet loss, etc. In the following, the loss used to train the single-modal recognition model in this embodiment will be described by taking the contrastive loss and the triplet loss as examples.

[0041] In one example, the loss used to train the single-modal recognition model can be constructed by contrastive loss, that is, taking the multi-modal recognition model as the teacher model and the single-modal recognition model as the student model, and based on the distance between the feature vectors of each two multi-modal face features and the distance between the feature vectors of each two single-modal face features, a loss function based on metric learning is used to construct the loss, which can include:

[0042] The contrastive loss L1 is constructed by using the following loss function:

[0043] ……(1)

[0044] wherein (b1, b2) is a sample pair constructed by any two multi-modal data samples b1 and b2; y is a label of whether b1 and b2 belong to the same person, y = 1 represents belonging to the same person (positive sample pair), and y = 0 represents not belonging to the same person (negative sample pair); f s () is a single-modal face feature extracted by a single-modal recognition model; f t () is a multi-modal face feature extracted by a multi-modal recognition model; d() is a distance between feature vectors; and N is a number of sample pairs.

[0045] Specifically, f t (b1, b2) represents two multi-modal face features extracted by inputting the multi-modal data sample pair (b1, b2) into the multi-modal recognition model; d(f t (b1, b2)) represents a distance between the two multi-modal face feature vectors; f s (b1, b2) represents two single-modal face features extracted by inputting a sample pair composed of single-modal data samples of the specified modal from the multi-modal data sample pair (b1, b2) into the single-modal recognition model; d(f s (b1, b2)) represents a distance between the two single-modal face feature vectors.

[0046] In the embodiment, d(f t (b1, b2)) is used to replace the constant margin (interval parameter) in the contrast loss, to realize dynamic guidance of the teacher model to the student model, and the item can also be other deformation formulas based on d(f t (b1, b2).

[0047] In another example, a loss used for training the single-modal recognition model can be constructed by a triplet loss, that is, the multi-modal recognition model is used as the teacher model, the single-modal recognition model is used as the student model, and the loss is constructed by using a loss function based on metric learning based on a distance between feature vectors of each two multi-modal face features and a distance between feature vectors of each two single-modal face features, and can include:

[0048] The triplet loss L2 is constructed by using the following loss function:

[0049] ……… (2)

[0050] wherein in a triplet constructed based on multi-modal data samples, a is an anchor sample, p is a positive sample, and n is a negative sample; f s () is a single-modal face feature extracted by a single-modal recognition model; f tis the multi-modal face feature extracted by the multi-modal recognition model; d() is the distance between the feature vectors; margin is a constant; and N is the number of triplets.

[0051] Specifically, a triplet (a, p, n) is constructed based on the multi-modal data samples, where a is an anchor sample, p is a positive sample, and n is a negative sample. t (a, n) represents two multi-modal face features extracted by inputting the multi-modal data sample pair (a, n) in the triplet to the multi-modal recognition model, d(f t (a, n)) represents the distance between the two multi-modal face feature vectors. t (a, p) represents two multi-modal face features extracted by inputting the multi-modal data sample pair (a, p) in the triplet to the multi-modal recognition model, d(f t (a, p)) represents the distance between the two multi-modal face feature vectors. s (a, n) represents two single-modal face features extracted by inputting the single-modal data sample pair of the specified modality from the multi-modal data sample pair (a, n) in the triplet to the single-modal recognition model, d(f s (a, n)) represents the distance between the two single-modal face feature vectors. s (a, p) represents two single-modal face features extracted by inputting the single-modal data sample pair of the specified modality from the multi-modal data sample pair (a, p) in the triplet to the single-modal recognition model, d(f s (a, p)) represents the distance between the two single-modal face feature vectors.

[0052] In this embodiment, max(d(f t (a, n))- d(f t (a, p)), margin) is used instead of the constant margin in the triplet loss, to realize dynamic guidance of the teacher model to the student model. This term can also be other deformations based on max(d(f t (a, n))- d(f t (a, p)), margin), such as completely removing the margin and only using d(f t (a, n))- d(f t (a, p)).

[0053] For metric learning, the basic idea is to pull the distance of positive sample pairs and push the distance of negative sample pairs. Conventional methods use fixed boundaries for sample pairs of different difficulty levels, which makes it difficult for the model to converge. The present application assigns dynamic boundaries to data of different difficulty levels through a teacher model, so that the feature style of the student model is closer to the teacher model, thereby improving the recognition effect of the student model.

[0054] In addition, when the multi-modal recognition model includes a plurality of sub-models corresponding to each modality in the multi-modal data sample, the distance between the feature vectors of any two multi-modal face features can be calculated using the following formula:

[0055] ……(3)

[0056] where (b1, b2) is a sample pair constructed by any two multi-modal data samples b1 and b2; (b1 i ,b2 i ) is the i-th modality data sample pair in (b1, b2); f t () is the multi-modal face feature extracted by the multi-modal recognition model; f ti () is the discrete modal face feature extracted by the i-th modality corresponding sub-model in the multi-modal recognition model; d() is the distance between the feature vectors; C i is the weight coefficient corresponding to the i-th modality, and the sum of the weight coefficients corresponding to all modalities is 1; m is the number of modalities contained in the multi-modal data sample.

[0057] Specifically, for each modality data sample contained in any two multi-modal data samples, a data sample pair (b1 i ,b2 i ) under each modality can be formed, and the distance d(f ti (b1 i ,b2 i ) between the feature vectors of the corresponding discrete modal face feature f ti (b1 i ,b2 i ) of each modality data sample pair is calculated first, and the distances d(f ti (b1 i ,b2 i ) of each modality are weighted and summed by setting the weight coefficient C i corresponding to each modality, to finally obtain the distance between the feature vectors of any two multi-modal face features as in formula (3).

[0058] For example, for triplet loss, if the teacher model (here referring to the multi-modal recognition model) is a combined model of three modalities, then:

[0059] d(f t (a,p)) = C1 d(f t1 (a1,p1))+ C2 d(f t2 (a2,p2))+ C3 d(f t3 (a2,p3)) ……(4)

[0060] Wherein, d(f t (a,p)) represents the distance between the feature vectors of the two multi-modal face features corresponding to the anchor sample and the positive sample; C i (i=1,2,3) is the feature distance weight, and C1+C2+C3=1, f t1 () can represent the face feature extracted by the color modal sub-model for the color image sample, f t2 () can represent the face feature extracted by the depth modal sub-model for the depth image sample, and f t3 () can represent the face feature extracted by the infrared modal sub-model for the infrared image sample; a1 is the anchor sample of the color modal, and p1 is the positive sample of the color modal; a2 is the anchor sample of the depth modal, and p2 is the positive sample of the depth modal; a3 is the anchor sample of the infrared modal, and p3 is the positive sample of the infrared modal.

[0061] Similarly, for the sample pair formed by the anchor sample and the negative sample in the triplet loss, the distance between the feature vectors of the two multi-modal face features corresponding to the sample pair can also be calculated by the similar formula (4), which will not be repeated here.

[0062] Compared with the related art, the embodiment transfers the information of the multi-modal large-scale model to the single-modal small-scale model through knowledge distillation, and the knowledge distillation is realized by the method of metric learning, that is, the metric learning is realized by calculating the distance between the feature vectors of every two multi-modal face features output by the multi-modal recognition model, and the distance between the feature vectors of every two single-modal face features output by the single-modal recognition model, so that the multi-modal multi-dimensional features to single-modal single-dimensional features can be compressed without the face feature vector dimensions output by the two models being consistent, and the recognition performance of the single-modal small-scale model is improved.

[0063] Another embodiment of the application relates to a single-modal face recognition method, which is realized based on the single-modal recognition model trained in the foregoing embodiment. As shown in the figure, the single-modal face recognition method comprises the following steps. Figure 2

[0064] Step 201: acquiring single-modal data of a face to be recognized in a target modal.

[0065] ​The single-modal data of the face includes, but is not limited to, any one of color images, depth images, and infrared images of the face. The target mode can be a specified data mode (color, depth, infrared) from the single-modal data.

[0066] Specifically, the single-modal data of the face in the target mode can be captured by a corresponding functional camera, for example, color images, depth images, and infrared images of the face are captured as single-modal data in the corresponding target mode.

[0067] In step 202, the single-modal recognition model trained by taking the target mode as the specified mode in the model training method of the single-modal face recognition is used to perform feature extraction on the single-modal data to obtain first single-modal face features.

[0068] In the foregoing embodiments, the single-modal recognition model is trained by using any one of the single-modal data samples in the specified mode of the multi-modal data samples, and therefore, when the single-modal recognition model is selected for feature extraction in this embodiment, the single-modal recognition model suitable for the single-modal data of the face to be recognized in the target mode should be selected. That is, the single-modal recognition model trained by taking the target mode of the face to be recognized as the specified mode for training the single-modal recognition model is used to perform feature extraction on the single-modal data of the face to be recognized in the target mode to obtain first single-modal face features.

[0069] In step 203, the first single-modal face features and the registered face features of the registered face in the target mode are compared to achieve identity recognition of the face to be recognized.

[0070] After obtaining the first single-modal face features, the first single-modal face features and the registered face features of the registered face in the target mode in the registered database can be compared, for example, the similarity between the feature vectors is calculated to achieve identity recognition of the face to be recognized.

[0071] Compared with the related art, the single-modal recognition model is used to perform face recognition on the single-modal data of the face to be recognized in the target mode, and the single-modal recognition model is obtained by knowledge distillation to transfer the information of the multi-modal large-scale model to the single-modal small-scale model, and the knowledge distillation is achieved by the method of metric learning, that is, the distance between the feature vectors of each two multi-modal face features output by the multi-modal recognition model and the distance between the feature vectors of each two single-modal face features output by the single-modal recognition model are calculated for metric learning, so that the face feature vector dimensions output by the two models do not need to be consistent, that is, the compression of multi-modal multi-dimensional features to single-modal single-dimensional features can be achieved, and the recognition performance of the single-modal small-scale model is improved.

[0072] Another embodiment of the present application relates to an electronic device, such as Figure 3As shown, the apparatus includes at least one processor 302; and a memory 301 connected with the at least one processor 302; wherein the memory 301 stores instructions executable by the at least one processor 302, and the instructions are executed by the at least one processor 302 to enable the at least one processor 302 to perform any of the above method embodiments.

[0073] The memory 301 and the processor 302 are connected in a bus manner, the bus can include any number of interconnected buses and bridges, and the bus connects one or more processors 302 and various circuits of the memory 301 together. The bus can also connect various other circuits such as peripheral devices, voltage stabilizers, and power management circuits together, which are well known in the art, and therefore, further description thereof will not be given herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements such as multiple receivers and transmitters, which provide a unit for communicating with various other devices on a transmission medium. The data processed by the processor 302 is transmitted on a wireless medium through an antenna, and further, the antenna also receives data and transmits the data to the processor 302.

[0074] The processor 302 is responsible for managing the bus and general processing, and can also provide various functions including timing, peripheral interface, voltage regulation, power management, and other control functions. The memory 301 can be used to store data used by the processor 302 in performing operations.

[0075] Another embodiment of the present application relates to a computer readable storage medium storing a computer program. The computer program is executed by a processor to implement any of the above method embodiments.

[0076] That is, those skilled in the art can understand that all or part of the steps of the above-mentioned embodiment methods can be completed by programs instructing related hardware, the programs are stored in a storage medium, and include a plurality of instructions for causing a device (which can be a single chip microcomputer, a chip, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0077] Those skilled in the art can understand that the above embodiments are specific embodiments for implementing the present application, and in actual applications, various changes can be made in form and details without departing from the spirit and scope of the present application.

Claims

1. A method for training a single-modal face recognition model, characterized in that, include: Multimodal facial data samples are input into a multimodal recognition model to obtain multimodal facial features; Inputting a single-modal data sample of any specified modality from the multimodal data samples into a single-modal recognition model yields single-modal face features; Using the multimodal recognition model as the teacher model and the unimodal recognition model as the student model, a loss is constructed based on the distance between feature vectors of every two multimodal face features and the distance between feature vectors of every two unimodal face features. The unimodal recognition model is then trained using knowledge distillation to obtain a trained unimodal recognition model. The contrastive loss L1 is constructed using the following loss function: Where (b1, b2) is a sample pair constructed from any two multimodal data samples b1 and b2; y is the label for whether b1 and b2 belong to the same person, y = 1 represents belonging to the same person, and y = 0 represents belonging to different people; f s () represents the single-modal face feature extracted by the single-modal recognition model; f t () represents the multimodal face features extracted by the multimodal recognition model; d() represents the distance between feature vectors; and N represents the number of sample pairs.

2. The method according to claim 1, characterized in that, The method uses the multimodal recognition model as the teacher model and the unimodal recognition model as the student model. Based on the distance between feature vectors of every two multimodal facial features and the distance between feature vectors of every two unimodal facial features, a loss function based on metric learning is constructed, including: The triplet loss L2 is constructed using the following loss function: In the triples constructed based on the multimodal data samples, a is the anchor sample, p is the positive sample, and n is the negative sample; f s () represents the single-modal face feature extracted by the single-modal recognition model; f t () represents the multimodal face features extracted by the multimodal recognition model; d() represents the distance between feature vectors; margin is a constant; N represents the number of triples.

3. The method according to any one of claims 1-2, characterized in that, The multimodal recognition model is a single model, and the step of inputting multimodal data samples into the multimodal recognition model to obtain multimodal facial features includes: The modal data samples in the multimodal data samples are fused to obtain mixed modal data samples; The mixed-modal data samples are input into the multimodal recognition model to obtain the multimodal facial features.

4. The method according to any one of claims 1-2, characterized in that, The multimodal recognition model includes multiple sub-models that correspond one-to-one with each modality in the multimodal data samples; the step of inputting the multimodal facial data samples into the multimodal recognition model to obtain multimodal facial features includes: Each modal data sample in the multimodal data sample is input into the sub-model corresponding to that modality to obtain the discrete modal face features corresponding to each modal data sample; The discrete modal facial features corresponding to each modality are combined into a set to obtain the multimodal facial features.

5. The method according to claim 4, characterized in that, The distance between the feature vectors of every two multimodal face features is calculated using the following formula: Where (b1, b2) is a sample pair constructed from any two multimodal data samples b1 and b2; (b1 i b2 i ) represents the data sample pair of the i-th mode in (b1, b2); f t () represents the multimodal facial features extracted by the multimodal recognition model; f ti () represents the discrete modal face feature extracted by the sub-model corresponding to the i-th modality in the multimodal recognition model; d() represents the distance between feature vectors; C i is the weight coefficient corresponding to the i-th mode, and the sum of the weight coefficients corresponding to all modes is 1; m is the number of modes contained in the multimodal data sample.

6. The method according to any one of claims 1-2, characterized in that, Both the multimodal recognition model and the unimodal recognition model are constructed using convolutional neural networks.

7. A single-modal face recognition method, characterized in that, include: Acquire single-modal data of the face to be identified in the target modality; In the single-modal face recognition model training method as described in any one of claims 1-6, the single-modal recognition model trained with the target modality as the specified modality is used to extract features from the single-modal data to obtain the first single-modal face features; The first unimodal face features and the registered face features of the registered face in the target modality are compared to achieve the identity recognition of the face to be identified.

8. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the model training method for unimodal face recognition as described in any one of claims 1 to 6, or the unimodal face recognition method as described in claim 7.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the model training method for single-modal face recognition as described in any one of claims 1 to 6, or the single-modal face recognition method as described in claim 7.

Images