A face retrieval method and device

Abstract

The present application provides a face retrieval method and device, which relate to the fields of artificial intelligence and computer vision. The method includes: obtaining a face image to be retrieved; extracting features from the face image through a first feature extraction model to obtain a first face feature; inputting the face image and the face feature into a first feature mapping model to obtain a standard feature corresponding to the output face feature, wherein the first feature mapping model is trained based on the target feature corresponding to the face image; and performing face retrieval on the face image based on the standard feature. In the present application, by splicing the features extracted by multiple feature extraction models and using the spliced ​​features as the basis for constructing standard features, the face retrieval system can utilize the combined effects of multiple feature extraction models to select an appropriate feature space and improve the accuracy of face retrieval.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and computer vision, and in particular to a face retrieval method and device. Background Art

[0002] With the development of science and technology, face retrieval is an emerging biometric technology that integrates computer image processing knowledge and biostatistics knowledge. Currently, face retrieval is widely used in identity recognition, identity verification and other related scenarios (such as security monitoring and access control gates, etc.).

[0003] In face retrieval technology, a face image to be retrieved is usually given, and the face retrieval system compares it with multiple face images in a specified face library to find the most similar face image or multiple face images. The face retrieval system does not directly calculate the similarity between the face image to be retrieved and the face images in the face library, but represents all images as features, and uses these features to calculate the similarity with each other, and then finds the most similar face image or multiple face images. Usually, different feature extraction models can extract different face features, but face features from different feature extraction models cannot be directly compared. In order to make face features from different feature extraction models directly comparable, it is necessary to map the face features extracted by all feature extraction models to the same feature space, and perform feature comparison in the feature space. At this time, what feature space to select for feature mapping becomes a problem, and since different face features are ultimately mapped to the same feature space, how to give full play to the comprehensive role of multiple feature extraction models is also a problem. Summary of the invention

[0004] The present application provides a face retrieval method and device, which utilizes the combined effect of multiple feature extraction models to select an appropriate feature space and improve the accuracy of face retrieval.

[0005] In a first aspect, the present application provides a face retrieval method, which can be applied to related scenarios such as identity recognition and identity authentication. The above-mentioned face retrieval method may include: obtaining a face image to be retrieved, which may be an image taken by a camera or an image manually uploaded by a user; extracting features from the face image through a first feature extraction model to obtain a first face feature; inputting the face image and the first face feature into a first feature mapping model to obtain a standard feature corresponding to the output first face feature, the first feature mapping model being trained based on the target feature corresponding to the face sample image; the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model; performing face retrieval on the face image based on the standard feature.

[0006] In the present application, the first facial feature in the facial image can be divided into structured features and unstructured features, wherein the structured features may include features used to characterize facial attributes, which may refer to some specific physical meanings of the facial image, such as age, gender and / or angle, etc., which are extracted from the facial image through a structured feature extraction model; and the unstructured features may include vectors used to represent facial features, which may refer to features that have no specific physical meaning in the facial image, and are composed of a string of numbers, and can also be called feature vectors, which are extracted from the facial image through an unstructured feature extraction model, and the similarity between feature vectors can be used to represent the similarity between the facial image to be retrieved and the facial template image.

[0007] In the present application, by taking facial features and facial images together as inputs of a feature mapping model, when it is difficult to obtain appropriate standard features using only facial features, more appropriate standard features are obtained through the additional information provided by facial images, thereby improving the accuracy of face retrieval.

[0008] In the present application, by splicing the features extracted by multiple feature extraction models and using the spliced ​​features as the basis for constructing standard features, the face retrieval system can utilize the combined effects of multiple feature extraction models to select appropriate feature spaces and improve the accuracy of face retrieval. Furthermore, since each face image only needs to go through one feature extraction model and one feature mapping model to obtain standard features, the system's computational load does not increase exponentially with the number of models, thereby reducing the system's computational load. Furthermore, since the feature mapping model corresponds one-to-one to the feature extraction model, the number of feature mapping models is consistent with the number of feature extraction models, thereby eliminating the need for the face retrieval system to train a huge number of feature mapping models, thereby reducing the system's computational load.

[0009] Based on the first aspect, in some possible implementations, the target features corresponding to the face sample image are obtained by splicing multiple face sample features, and the face sample features are obtained by performing feature extraction on the face sample image by multiple second feature extraction models; the multiple second feature extraction models include the first feature extraction model; the multiple second feature extraction models have at least different training samples, model structures, training strategies or feature dimensions.

[0010] In the present application, the multiple second feature extraction models may be the second feature extraction models of the manufacturer or the second feature extraction models from other different manufacturers; the multiple second feature extraction models may include the first feature extraction model, that is, the second feature extraction model may also have other feature extraction models on the basis of including the first feature extraction model; the training samples, model structures, training strategies or feature dimensions of the multiple second feature extraction models are different.

[0011] Based on the first aspect, in some possible implementations, the method further includes: obtaining a facial sample image; inputting the facial sample image into a first feature extraction model to obtain an output first facial sample feature; training a second feature mapping model based on the facial sample image, the first facial sample feature, and the target feature corresponding to the facial sample image to obtain a first feature mapping model, wherein the second feature mapping model corresponds to the first feature extraction model.

[0012] In the present application, the second feature mapping model can be a feature mapping model obtained by the manufacturer through sample training; it can also be a feature mapping model obtained by the manufacturer and other manufacturers through collaborative training. Specifically, the input of the second feature mapping model is a face image and a first face feature, and the output of the second feature mapping model is a standard feature. The optimization purpose of the training is to make the standard feature fit the target feature as much as possible, and the target feature is obtained by splicing multiple face sample features. The multiple face sample features are obtained by extracting features from the face sample image by the first feature extraction model from the manufacturer and multiple second feature extraction models from the manufacturer or other different manufacturers. The target feature is spliced ​​from multiple face sample features extracted by the manufacturer and other manufacturers. When training the second feature mapping model, it is necessary to make the output of the second feature mapping model fit the target feature as much as possible. After the second feature mapping model is trained, it becomes the first feature mapping model; face retrieval is performed on the face image according to the standard feature.

[0013] Based on the first aspect, in some possible implementations, before training the second feature mapping model according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model that satisfies the objective function, the method further includes: obtaining the face sample image; inputting the face sample image into N second feature extraction models to obtain output N second face sample features, where N is a positive integer greater than or equal to 2; multiplying the N second face sample features by N preset coefficients one by one; splicing the multiplied N second face sample features to obtain spliced ​​face sample features; obtaining the target feature corresponding to the face sample image based on the spliced ​​face sample features, wherein the dimension of the target feature is less than or equal to the sum of the dimensions of the N second feature extraction models.

[0014] Based on the first aspect, in some possible implementations, the above method also includes: obtaining a face sample image, the face sample image having identity information; inputting the face sample image into N second feature extraction models to obtain output N second face sample features; performing face recognition on the face sample image based on the N second face sample features and identity information to obtain N preset coefficients.

[0015] Based on the first aspect, in some possible implementations, the above method also includes: configuring N preset coefficients for N second feature extraction models, and the N preset coefficients are equal; or, configuring N preset coefficients for N second feature extraction models according to preset evaluation criteria.

[0016] Based on the first aspect, in some possible implementations, the above method also includes: obtaining N second feature extraction model corresponding coefficient combinations within a preset coefficient range; multiplying the N second face sample features by the coefficient combinations accordingly; splicing the multiplied N second face sample features to obtain spliced ​​face sample features; performing face retrieval on the face sample image based on the spliced ​​face sample features to obtain preset coefficients in the coefficient combination that meet preset conditions.

[0017] Based on the first aspect, in some possible implementations, obtaining target features corresponding to the face sample image based on the spliced ​​face sample features includes: reducing the dimension of the spliced ​​face sample features; and determining the face sample features after dimensionality reduction as the target features corresponding to the face sample image.

[0018] Based on the first aspect, in some possible implementations, the second feature mapping model includes a unique module and a shared module;

[0019] Correspondingly, the second feature mapping model is trained according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model, including: inputting the face sample image and multiple first face sample features into a unique module to obtain the output third face sample feature, the multiple first face sample features are extracted from the face sample image through different multiple first feature extraction models; inputting the third face sample feature into a shared module to obtain the standard feature corresponding to the multiple first face sample features; according to the face sample image, multiple first face sample features, the standard feature corresponding to the multiple first face sample features and the target feature corresponding to the face sample image, the unique module and the shared module are trained to obtain the first feature mapping model.

[0020] Based on the first aspect, in some possible implementations, the second feature mapping model includes an image branch module, a feature branch module, and a comprehensive module;

[0021] Correspondingly, the second feature mapping model is trained according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model, including: inputting the face sample image into the image branch module to obtain the fourth face sample feature after output; inputting the first face sample feature into the feature branch module to obtain the fifth face sample feature after output, and the first face sample feature is extracted from the face sample image through the first feature extraction model; inputting the fourth face sample feature and the fifth face sample feature into the comprehensive module to obtain the standard feature corresponding to the first face sample feature; according to the face sample image, the first face sample feature, the standard feature corresponding to the first face sample feature and the target feature corresponding to the face sample image, the image branch module, the feature branch module and the comprehensive module are trained to obtain the first feature mapping model.

[0022] Based on the first aspect, in some possible implementations, face retrieval is performed on face images according to the standard features, including: determining the similarity between the standard features and the standard features of a first face sample image, where the first face template image is any face sample image among multiple face sample images; when the similarity is greater than a first threshold, the first face sample image is a target for face image retrieval.

[0023] In a second aspect, the present application provides a face retrieval device, comprising: an interface module, used to obtain a face image to be retrieved, and the face image to be retrieved can be an image taken by a camera or an image manually uploaded by a user; a feature extraction module, used to extract features of the face image through a first feature extraction model to obtain a first face feature; a feature mapping module, used to input the face image and the first face feature into a first feature mapping model to obtain a standard feature corresponding to the output first face feature, the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model, and the first feature mapping model is trained according to the target feature corresponding to the face sample image; a face retrieval module, used to perform face retrieval on the face image according to the standard feature.

[0024] Based on the second aspect, in some possible implementations, the target features corresponding to the face sample images are obtained by concatenating multiple face sample features, and the face sample features are obtained by extracting features from the face sample images by multiple second feature extraction models.

[0025] Based on the second aspect, in some possible embodiments, the above-mentioned device also includes: a mapping model training module, used to obtain a face sample image; input the face sample image into the first feature extraction model to obtain an output first face sample feature; train the second feature mapping model according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain a first feature mapping model, and the second feature mapping model corresponds one-to-one to the first feature extraction model. Based on the second aspect, in some possible embodiments, the above-mentioned device also includes: a target feature acquisition module, which is used to acquire a face sample image before the mapping model training module obtains the first feature mapping model that satisfies the objective function; inputs the face sample image into N second feature extraction models to obtain output N second face sample features, where N is a positive integer greater than or equal to 2; multiplies the N second face sample features with N preset coefficients one by one; splices the multiplied N second face sample features to obtain spliced ​​face sample features; and obtains the target feature corresponding to the face sample image based on the spliced ​​face sample features, wherein the dimension of the target feature is less than or equal to the sum of the dimensions of the N second feature extraction models.

[0026] Based on the second aspect, in some possible implementations, the target feature acquisition module is also used to acquire a face sample image, which has identity information; the face sample image is input into N second feature extraction models to obtain N second face sample features as output; and face recognition is performed on the face sample image based on the N second face sample features and the identity information to obtain N preset coefficients.

[0027] Based on the second aspect, in some possible implementations, the target feature acquisition module is specifically used to configure N preset coefficients for N second feature extraction models, and the N preset coefficients are equal; or, according to preset evaluation criteria, configure N preset coefficients for the N second feature extraction models.

[0028] Based on the second aspect, in some possible implementations, the target feature acquisition module is specifically used to obtain N second feature extraction model corresponding coefficient combinations within a preset coefficient range; multiply the N second face sample features by the coefficient combinations accordingly; splice the multiplied N second face sample features to obtain spliced ​​face sample features; perform face retrieval on the face sample image based on the spliced ​​face sample features to obtain preset coefficients in the coefficient combination that meet preset conditions.

[0029] Based on the second aspect, in some possible implementations, the target feature acquisition module is further used to reduce the dimension of the spliced ​​face sample features; and determine the face sample features after dimensionality reduction as the target features corresponding to the face sample image.

[0030] Based on the second aspect, in some possible implementations, the second feature mapping model includes a unique module and a shared module;

[0031] Correspondingly, the mapping model training module is also used to input the face sample image and multiple first face sample features into the unique module to obtain the output third face sample features, where the multiple first face sample features are extracted from the face sample image through different multiple first feature extraction models; input the third face sample features into the shared module to obtain standard features corresponding to the multiple first face sample features; train the unique module and the shared module according to the face sample image, the multiple first face sample features, the standard features corresponding to the multiple first face sample features, and the target features corresponding to the face sample image to obtain the first feature mapping model.

[0032] Based on the second aspect, in some possible implementations, the second feature mapping model includes an image branch module, a feature branch module, and a comprehensive module;

[0033] Correspondingly, the mapping model training module is also used to input the face sample image into the image branch module to obtain the fourth face sample feature after output; input the first face sample feature into the feature branch module to obtain the fifth face sample feature after output, the first face sample feature is extracted from the face sample image through the first feature extraction model; input the fourth face sample feature and the fifth face sample feature into the comprehensive module together to obtain the standard feature corresponding to the first face sample feature; according to the face sample image, the first face sample feature, the standard feature corresponding to the first face sample feature and the target feature corresponding to the face sample image, the image branch module, the feature branch module and the comprehensive module are trained to obtain the first feature mapping model.

[0034] Based on the second aspect, in some possible implementations, the face retrieval module is specifically used to: determine the similarity between the standard features and the standard features of the first face sample image, where the first face template image is any face sample image among multiple face sample images; when the similarity is greater than a first threshold, the first face sample image is the target of the face image retrieval.

[0035] The interface module mentioned in the second aspect above may be a receiving interface, a receiving circuit or a receiver, etc.; the feature extraction module, the feature mapping module, the face retrieval module, the mapping model training module and the target feature acquisition module may be one or more processors.

[0036] In a third aspect, the present application provides a face retrieval device, which may include: a processor and a communication interface. The processor may be used to support the face retrieval device to implement the functions involved in the above-mentioned first aspect or any possible implementation scheme of the first aspect. For example, the processor may obtain the face image to be retrieved through the communication interface.

[0037] Based on the third aspect, in some possible implementations, the face retrieval device may further include a memory, the memory being used to store computer-executable instructions and data necessary for the face retrieval device. When the face retrieval device is running, the processor executes the computer-executable instructions stored in the memory, so that the face retrieval device performs the face retrieval method as described in the first aspect or any possible implementation of the first aspect.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the instructions are run on a computer, they are used to execute any face retrieval method in the above-mentioned first aspect.

[0039] In a fifth aspect, the present application provides a computer program or a computer program product. When the computer program or the computer program product is executed on a computer, the computer implements any face retrieval method in the above-mentioned first aspect.

[0040] It should be understood that the second to fifth aspects of the present application are consistent with the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.

[0042] Figure 1 is a flowchart of a face retrieval method;

[0043] Figure 2 It is a flowchart of another face retrieval method;

[0044] Figure 3 is a flowchart of another face retrieval method;

[0045] Figure 4 A schematic diagram of facial features in an embodiment of the present application;

[0046] Figure 5 Schematic diagram of the process of face retrieval method in the embodiment of the present application;

[0047] Figure 6 This is a flow chart of a method for training a first feature mapping model in an embodiment of the present application;

[0048] Figure 7 This is a flow chart of a method for training a feature mapping model in an embodiment of the present application;

[0049] Figure 8 A schematic diagram of a method flow for obtaining target features corresponding to a sample face image in an embodiment of the present application;

[0050] Fig. 9 A flowchart of a method for training a unique module and a shared module in an embodiment of the present application;

[0051] Fig.10 A schematic flow chart of a method for performing feature mapping using a first feature mapping model in an embodiment of the present application;

[0052] Fig.11 A schematic diagram of a method flow chart of a training image branch module, a feature branch module and a comprehensive module in an embodiment of the present application;

[0053] Fig.12 This is a schematic diagram of the structure of a face retrieval device in an embodiment of the present application;

[0054] Fig.13 Schematic diagram of the structure of the face retrieval device in the embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application. In the following description, reference is made to the drawings that form a part of the present application and show specific aspects of the embodiments of the present application or specific aspects of the embodiments of the present application in an illustrative manner. It should be understood that the embodiments of the present application can be used in other aspects and may include structural or logical changes not depicted in the drawings. For example, it should be understood that the disclosure of the described method in conjunction with the disclosed content can also be applied to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units such as functional units to perform the one or more method steps described (for example, one unit performs one or more steps, or multiple units, each of which performs one or more of the multiple steps), even if such one or more units are not explicitly described or described in the drawings. On the other hand, for example, if a specific device is described based on one or more units such as functional units, the corresponding method may include a step to perform the functionality of one or more units (for example, one step performs the functionality of one or more units, or multiple steps, each of which performs the functionality of one or more units in multiple units), even if such one or more steps are not explicitly described or described in the drawings. Further, it should be understood that, unless explicitly stated otherwise, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0056] The embodiment of the present application provides a face retrieval method, which can be widely used in related scenarios such as identity recognition and identity authentication. The face retrieval system does not directly calculate the similarity between the face image to be retrieved and the face images in the face library, but represents all images as features, and uses these features to calculate the similarity with each other, and then finds the most similar face image or multiple face images. Usually, there are multiple feature extraction models in the face retrieval system, and all face images need to use all feature extraction models to extract features, and the features extracted from the same face image are spliced ​​or the dimensions are compressed using dimensionality reduction methods to obtain the final output features. Figure 1 is a flowchart of a face retrieval method, see Figure 1 As shown, there are different feature extraction models A, B and C in the face retrieval system. Face image 1 and face image 2 are respectively input into these three feature extraction models to obtain features A1, B1 and C1 of face image 1 and features A2, B2 and C2 of face image 2. Then, the face retrieval system splices the three features of the same face image to obtain the final output features, namely, the features of face image 1 and the features of face image 2. Finally, the face retrieval system compares the features of face image 1 and the features of face image 2, thereby completing face retrieval. However, since each face image needs to pass through all feature extraction models, the amount of calculation of the system increases exponentially with the number of feature extraction models. Optionally, feature extraction models A, B and C can be feature extraction models under different feature dimensions, or multiple feature extraction models with the same function under the same feature dimension.

[0057] In an embodiment of the present application, a face retrieval method is also provided, in which a feature mapping model is adopted, and the face retrieval system maps the features extracted by a feature extraction model (i.e., source domain features) to the feature space corresponding to another feature extraction model (i.e., target domain feature space), and performs feature comparison in the feature space, thereby realizing mutual search between features and completing face retrieval. For example, Figure 2 For a flow chart of another face retrieval method, see Figure 2As shown, feature A (i.e., feature of face image 1) is obtained by feature extraction model A for face image 1, feature B is obtained by feature extraction model B for face image 2, and feature C is obtained by feature extraction model C for face image 2. Feature B and feature C are respectively mapped to the feature space corresponding to model A by the feature mapping model to obtain feature 1 of face image 2 and feature 2 of face image 2, and compared with the feature of face image 1 (i.e., feature A) in the feature space corresponding to model A. Feature A forms two groups of feature pairs with feature 1 of face image 2 and feature 2 of face image 2, respectively. Each group of feature pairs is compared in the feature space corresponding to model A, thereby realizing mutual search between features and completing face retrieval. In the embodiment of the present application, since all features are mapped to the feature space corresponding to feature extraction model A, feature A does not need to be mapped and can be directly used as the feature of face image 1 to be compared with the feature of face image 2. In order to realize the comparison between features, a feature mapping model needs to be trained between each pair of features. When the number of feature extraction models is n (n≥2), the number of feature mapping models is n. It can be expressed as the following formula (1):

[0058]

[0059] It can be seen that a huge number of feature mapping models need to be trained in the face retrieval system. Furthermore, since the feature mapping model only takes the source domain features as input, the effect of feature mapping cannot be guaranteed when the source domain features have poor expression ability.

[0060] Furthermore, in an embodiment of the present application, a face retrieval method is provided. In this method, since different feature extraction models can extract different face features, but different face features cannot be directly compared, in order to enable direct comparison of different face features, the face retrieval system needs to map all extracted face features to the same feature space and perform feature comparison in the feature space, thereby realizing mutual search between features and completing face retrieval. For example, Figure 3 FIG. 1 is another flow chart of the face retrieval method in the embodiment of the present application, see Figure 3As shown, the face retrieval system inputs face image 1 into feature extraction model A to obtain feature A, the face retrieval system inputs face image 2 into feature extraction model B to obtain feature B, and the face retrieval system inputs face image 3 into feature extraction model C to obtain feature C. Then, the face retrieval system maps feature B and feature C to the same feature space (assuming it is the feature space corresponding to feature extraction model A) to obtain face image 2 features and face image 3 features. In this embodiment of the application, since all features are mapped to the feature space corresponding to feature extraction model A, feature A does not need to be mapped and can be directly used as a face image 1 feature for comparison with face image 2 features and face image 3 features. In the feature space corresponding to feature extraction model A, face image 2 features and face image 3 features can also be directly compared, thereby achieving mutual search between features and completing face retrieval. However, for feature extraction models A, B and C, the performance capabilities of the feature extraction models are different in different scenarios. When the advantages of a single feature extraction model in each scenario are not prominent, it is a problem to choose the feature space corresponding to which model as the final mapping feature space; further, since all features are ultimately mapped to the feature space corresponding to a single feature extraction model, the comprehensive effect of multiple feature extraction models is not brought into play.

[0061] In order to solve the above problems, an embodiment of the present application provides a face retrieval method, which can be applied to the above face retrieval system. In an embodiment of the present application, the face retrieval system can be set on equipment such as security monitoring and access control gates.

[0062] It should be noted that in the embodiments of the present application, Figure 4 This is a schematic diagram of facial features in the embodiment of the present application, see Figure 4 As shown, the facial features in the face image can be divided into structured features and unstructured features, wherein the structured features may include features for characterizing facial attributes, which may refer to some specific physical meanings of the face image, such as age, gender, angle, etc., which are extracted from the face image through a structured feature extraction model; and the unstructured features may include vectors for representing facial features, which may refer to features without specific physical meanings in the face image, and are composed of a string of numbers, and may also be called feature vectors, which are extracted from the face image through an unstructured feature extraction model, and the similarity between feature vectors can be used to represent the similarity between the face image to be retrieved and the face template image.

[0063] The above-mentioned structured feature extraction model and unstructured feature extraction model are both machine learning models (such as convolutional neural networks (CNN). CNN is essentially an input-to-output mapping, which can learn a large number of mapping relationships between inputs and outputs without requiring any precise mathematical expressions between inputs and outputs. After collecting training samples, CNN is trained, and CNN has the mapping capability between input and output pairs. Of course, the structured feature extraction model and the unstructured feature extraction model can also be other machine learning models, which are not specifically limited in the embodiments of the present application.

[0064] In some possible implementations, the "features" described in the following embodiments may be unstructured features or spliced ​​features obtained by splicing structured features and unstructured features. The "feature extraction model" may be an unstructured feature extraction model or a model combination consisting of a structured feature extraction model and an unstructured feature extraction model. The embodiments of the present application do not make any specific limitations on this.

[0065] Figure 5 FIG. 1 is a flow chart of the face retrieval method in the embodiment of the present application, see Figure 5 As shown, the method may include:

[0066] S501: Acquire a face image to be retrieved;

[0067] In an embodiment of the present application, the facial image acquired by the face retrieval system may be an image directly captured by the retrieval system, such as an image taken by a camera of the face retrieval system; it may also be an image manually input into the face retrieval system by a user, such as when a user needs to retrieve a target person and directly inputs the image of the target person into the face retrieval system; it may also be an image of a certain person in the face retrieval system gallery.

[0068] In an embodiment of the present application, the face retrieval system receives an input face image to be retrieved. Optionally, the face retrieval system may also receive an input base image (i.e., a face template image). The face template image may be used to compare with the face image to realize mutual search of face image features and complete face retrieval of the face image.

[0069] S502: Inputting the face image into a first feature extraction model to obtain a first face feature;

[0070] In an embodiment of the present application, the face retrieval device can use a large number of face sample images to train the first feature extraction model, so that the face retrieval device can obtain the first face feature of the face image after inputting the face image into the first feature extraction model. Optionally, the first feature extraction model can be an unstructured feature extraction model, or a model combination of a structured feature extraction model and an unstructured feature extraction model.

[0071] S503: Inputting the facial image and the first facial feature into a first feature mapping model for feature mapping, and obtaining a standard feature corresponding to the first facial feature as output;

[0072] In an embodiment of the present application, the face retrieval device can pre-train a first feature mapping model using a face sample image as a first feature extraction model, and the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model. Then, the face retrieval device can input the face image to be retrieved and its corresponding first face feature into the first feature mapping model for feature mapping to obtain the output standard feature.

[0073] It should be noted that the first feature mapping model is trained based on the face sample image and the target features corresponding to the face sample image. Optionally, the target features corresponding to the face sample image are obtained by splicing multiple face sample features, and the face sample features are obtained by extracting features from the face sample image using multiple second feature extraction models. The multiple second feature extraction models include the first feature extraction model; the multiple second feature extraction models have at least different training samples, model structures, training strategies or feature dimensions.

[0074] As a possible implementation, the first feature extraction model and the second feature extraction model may be the same feature extraction model or different feature extraction models. The second feature extraction model may be the second feature extraction model of the manufacturer or the second feature extraction model from other different manufacturers; the second feature extraction model may include the first feature extraction model, that is, the second feature extraction model may also include other feature extraction models on the basis of including the first feature extraction model. The training samples, model structures, training strategies or feature dimensions of the second feature extraction models from different manufacturers may also be different. For example, the feature dimension of the second feature extraction model of manufacturer A is 256, and the feature dimension of the second feature extraction model of manufacturer B is 512.

[0075] S504: Perform face retrieval on the face image according to standard features.

[0076] In an embodiment of the present application, after the face retrieval device obtains the standard features of the face image to be retrieved through S503, it can directly compare the standard features with the standard features of multiple face template images to find the most similar features, and then obtain one or more face template images to realize feature mutual search and complete face retrieval.

[0077] It should be noted that the above-mentioned multiple face template images can be input into the face retrieval device together with the face image to be retrieved, and S501 to S503 are executed in sequence to complete the extraction of face features, and then compare them with the standard features of the face image to be retrieved; or, the face template image is pre-input into the face retrieval device to complete the extraction of face features, obtain the standard features corresponding to each face template image, and then store the standard features corresponding to the face template image for subsequent acquisition of the standard features of the face image to be retrieved, read the standard features corresponding to each face template image and compare them, and then complete the face retrieval. Of course, the face image to be retrieved and the face template image can also be subjected to feature extraction and feature comparison in other ways, as long as the face retrieval can be completed, and the embodiments of the present application do not make specific limitations.

[0078] The following is an explanation of the face retrieval process for the face image according to the standard features in the above S504.

[0079] When the face recognition device searches for a face image based on the standard features, it needs to compare the standard features with the standard features of multiple face sample images, which can be provided by the manufacturer or other manufacturers. The standard features of the face sample image currently being compared with the standard features are defined as the standard features of the first face sample image, and the similarity between the standard features and the standard features of the first face sample image is determined. When the similarity is greater than a first threshold, the first face sample image is the target of face image retrieval.

[0080] This method can realize mutual feature search between different models of the same manufacturer, as well as mutual feature search between different manufacturers.

[0081] The training process of the first feature mapping model in S503 above is described below.

[0082] Figure 6 This is a flow chart of the method for training the first feature mapping model in the embodiment of the present application, see Figure 6 As shown, the method may include:

[0083] S601: Acquire a sample face image;

[0084] In an embodiment of the present application, a face retrieval device obtains an input face sample image, which may be one or more images among a large number of face sample images.

[0085] S602: Inputting a face sample image into a first feature extraction model to obtain output first face sample features;

[0086] In an embodiment of the present application, the face retrieval device inputs the face sample image into the first feature extraction model to obtain the first face sample feature of the face sample image. The first face sample feature can be an unstructured feature of the face sample image, or a spliced ​​feature formed by splicing the structured feature and the unstructured feature of the face sample image.

[0087] S603: Training the second feature mapping model according to the face sample image, the first face sample feature, and the target feature corresponding to the face sample image to obtain the first feature mapping model.

[0088] Among them, the second feature mapping model corresponds to the first feature extraction model.

[0089] In an embodiment of the present application, the face retrieval device can input a face sample image into a plurality of different feature extraction models to obtain a plurality of face sample features as output, and then multiply each face sample feature by a corresponding preset coefficient, and splice the multiplied face sample features, and finally, obtain the target feature corresponding to the face sample image based on the spliced ​​face sample features, which can also be called the target standard feature. Next, the face retrieval device inputs the face sample image and its corresponding first face sample feature into the second feature mapping model to obtain the output standard sample feature, and maximizes the similarity between the standard sample feature and the target feature by adjusting the parameters of the second feature mapping model. At this time, the optimized objective function can be to make the cosine similarity between the standard sample feature and the target feature as large as possible. When the objective function converges, the second feature mapping model is trained, and the trained second feature model is the first feature mapping model.

[0090] In the embodiment of the present application, the second feature mapping model can be a feature mapping model obtained by the manufacturer through sample training; or it can be a feature mapping model obtained by the manufacturer and other manufacturers through collaborative training. Specifically, only the face retrieval device of the manufacturer or the manufacturer and other manufacturers inputs the face sample image into multiple different feature extraction models to obtain multiple face sample features as output, and then multiplies each face sample feature with the corresponding preset coefficient, and splices the multiplied face sample features. Finally, the target feature corresponding to the face sample image is obtained according to the spliced ​​face sample features. Next, the face retrieval device of the manufacturer inputs the face sample image and its corresponding first face sample feature into the second feature mapping model to obtain the output standard sample feature, and by adjusting the parameters of the second feature mapping model, the similarity between the standard sample feature and the target feature is maximized (the target feature is obtained only by the manufacturer or the manufacturer and other manufacturers in collaboration). At this time, the optimized objective function can be the cosine similarity between the standard sample feature and the target feature as large as possible. When the objective function converges, the second feature mapping model is trained, and the trained second feature model is the first feature mapping model.

[0091] The solution of the present invention can be used for feature mutual search between different versions of models of the same manufacturer. For example, manufacturer A has deployed a set of face retrieval systems, and currently needs to upgrade the old feature extraction model to a new feature extraction model. The feature mutual search between the new and old models can be completed by the following operations: training new and old feature mapping models that match the new and old feature extraction models; inputting the base library image and the features extracted by the old feature extraction model into the old feature mapping model to obtain the standard features of the base library image; inputting the image to be retrieved into the new feature extraction model to extract features, and inputting the features extracted by the new feature extraction model and the image to be retrieved into the new feature mapping model to obtain the standard features of the image to be retrieved; searching and comparing the standard features of the image to be retrieved with the standard features of the base library image, and the images with higher similarity in the base library are ranked higher.

[0092] The scheme of the present invention can also be used for feature mutual search between models of the same manufacturer running on different devices. For example, if the model of manufacturer A running on the central server is a large feature extraction model (with a more complex model structure), and the model running on the camera is a small feature extraction model (with a more compact model structure), the feature mutual search between the large and small models can be completed by the following operations: training large and small feature mapping models that match the large and small feature extraction models; extracting features from the image stored in the central server using the large feature extraction model, and inputting the features and the image into the large feature mapping model to obtain the standard features of the central server image; extracting features from the image stored in the camera using the small feature extraction model, and inputting the features and the image into the small feature mapping model to obtain the standard features of the camera image; comparing the standard features of the central server image with the standard features of the camera image to calculate the similarity, and use them for retrieval and sorting.

[0093] The scheme of the present invention can also be used for feature mutual search between models of different manufacturers. For example, if the model of manufacturer A is feature extraction model A, and the model of manufacturer B is feature extraction model B, the feature mutual search between models A and B can be completed by the following operations: training feature mapping models A and B that match feature extraction models A and B; extracting features from images assigned to manufacturer A using feature extraction model A, and inputting the features and images into feature mapping model A to obtain standard features of images of manufacturer A; extracting features from images assigned to manufacturer B using feature extraction model B, and inputting the features and images into feature mapping model B to obtain standard features of images of manufacturer B; comparing the standard features of images of manufacturer A with the standard features of images of manufacturer B to obtain their similarity, and use it for retrieval and sorting.

[0094] It should be noted that the above-mentioned multiple feature extraction models can be unstructured feature extraction models, or a model combination consisting of a structured feature extraction model and an unstructured feature extraction model, and the embodiments of the present application do not make specific limitations.

[0095] For example, Figure 7 This is a flow chart of the method for training the feature mapping model in the embodiment of the present application, see Figure 7 As shown, in the training stage, the face retrieval device first obtains the target feature A corresponding to the face sample image, and inputs the face sample image into the feature extraction model, and obtains the face sample features from the feature extraction model. Then, the face retrieval device inputs the face sample image and the face sample features into the feature mapping model corresponding to the feature extraction model to obtain the output standard sample feature B. Then, the face retrieval device calculates the similarity between the target feature A and the standard sample feature B, and adjusts the various parameters in the feature mapping model according to the optimization objective function, that is, the cosine similarity between features A and B is as large as possible, until the objective function converges, thereby completing the training of the feature mapping model.

[0096] In some possible implementations, the cosine similarity of two features can be calculated using formula (2):

[0097]

[0098] Among them, the above A i and B i Represent the components of features A and B respectively, k is the number of components of A and B, and k is a positive integer.

[0099] Of course, optionally, in addition to the cosine similarity as the objective function, other functions that can measure the similarity between features (such as Euclidean similarity, Euclidean distance, etc.) can be used as the objective function, and the embodiments of the present application do not make specific limitations.

[0100] In some possible implementations, the face retrieval device can obtain the target features corresponding to the face sample image by the following method. First, the face retrieval device obtains the face sample image; optionally, the face retrieval device obtains the input face sample image, which can be one or more images from a large number of face sample images; then, the face retrieval device inputs the face sample image into N second feature extraction models to obtain output N second face sample features; wherein N is a positive integer greater than or equal to 2; optionally, the face retrieval device inputs the face sample image into N different second feature extraction models, and these second feature extraction models can be unstructured feature extraction models, or can be a model combination consisting of a structured feature extraction model and an unstructured feature extraction model, which is not specifically limited in the embodiments of the present application. Through the N second feature extraction models, the face retrieval device can obtain the output N second face sample features, and one second feature extraction model outputs one second face sample feature. Next, the face retrieval device multiplies the N second face sample features with the N preset coefficients one by one; in this way, for the second feature extraction models with different capabilities, different coefficients are assigned to the features extracted by different feature extraction models, which can effectively play the role of matching each feature extraction model with its capabilities; the face retrieval device splices the multiplied N second face sample features to obtain the spliced ​​face sample features; the face retrieval device obtains the target features corresponding to the face sample image based on the spliced ​​face sample features, and the dimension of the target features is less than or equal to the sum of the dimensions of the N second feature extraction models. It can be seen that in the process of calculating the target features, the features corresponding to multiple feature extraction models are spliced, and the spliced ​​features are used as the basis for constructing the target features, which can maximize the available information.

[0101] In some optional implementations, after preparing a batch of data, the face retrieval device can obtain a dimensionality reduction matrix using algorithms such as principal component analysis (PCA), linear discriminant analysis (LDA), and autoencoder (AE). Then, for any spliced ​​face sample feature, the reduced-dimensional face sample feature can be obtained by multiplying it with the dimensionality reduction matrix, and the reduced-dimensional face sample feature is determined as the target feature corresponding to the face sample image. In the embodiment of the present application, the spliced ​​features are reduced in dimensionality, which can reduce the feature comparison time and thus improve the retrieval efficiency, and remove redundant information and improve the robustness of the standard features.

[0102] It should be noted that AE is a neural network that uses a back propagation algorithm to make the output value equal to the input value. It includes an encoder and a decoder. For the embodiment of the present application, the encoder first compresses the input splicing features into a latent space representation, and the dimension of the latent space representation is lower than the input splicing features. Then the decoder reconstructs the output (the dimension is the same as the input splicing features) through the latent space representation, and the output should be as close as possible to the input splicing features. In order to achieve this goal, the objective function can be set to the cosine similarity of the input splicing features and the encoder output target features. After determining the form of the objective function, input different face sample features, and calculate the gradient of the objective function with respect to the parameters in the encoder and decoder. Based on the gradient, the parameters in the encoder and decoder can be adjusted until the change in the objective function after the update training is less than the set value (that is, the objective function has converged), so that the parameters of the encoder and decoder are determined. At this time, the encoder can use the parameters obtained by training to complete the dimensionality reduction function of the input splicing features.

[0103] For example, Figure 8 FIG. 1 is a flow chart of a method for obtaining target features corresponding to a face sample image in an embodiment of the present application, see Figure 8 As shown, there are three second feature extraction models A, B and C in the face retrieval device, and the preset coefficients corresponding to each feature extraction model are: w a =0.3, w b =0.5, w c =0.2; the face sample features obtained by the second feature extraction models A, B and C of the input face sample image are: F A =[0.04,…,0.08], F B = [0.06, ..., 0.03] and F C =[0.05,…,0.05]; the face retrieval device can use w a 、wb and w c For each face sample feature F A 、F B and F C Correspondingly, the face sample features are weighted by using the preset coefficients. Specifically, F A Bitwise AND w a WF obtained by multiplication A =[0.012,…,0.024],WF B Bitwise AND w b WF obtained by multiplication B =[0.03,…,0.015],F C Bitwise AND w c WF obtained by multiplication C =[0.01, ..., 0.01]; Next, the face retrieval device is used to retrieve the face after WF A 、WF B and WF C After splicing, the spliced ​​face sample feature CWF = [0.012, ..., 0.024, 0.03, ..., 0.015, 0.01, ..., 0.01] is obtained; finally, CWF is multiplied by the dimensionality reduction matrix to obtain the target feature SF = [0.03, ..., 0.07].

[0104] In the embodiment of the present application, the above preset coefficients can be obtained in the following manner without limitation.

[0105] In the first way, the preset coefficients can be used as learnable parameters, and the preset coefficients are determined by training the face recognition model. Specifically, first, the face retrieval device can obtain a face sample image, at which time the face sample image has corresponding identity information; then, the face retrieval device inputs the face sample image into N second feature extraction models to obtain output N second face sample features; next, the face retrieval device performs face recognition on the face sample image based on the N second face sample features and identity information to obtain N preset coefficients. Optionally, in the process of obtaining the preset coefficients, a face recognition model can be used, and its input data can be the second face sample features extracted by each second feature extraction model and the corresponding identity information; the optimization objective function can be that the face sample features of the same identity information are as close as possible, and the face sample features of different identity information are as far as possible. After determining the form of the objective function, the face retrieval device inputs N second face sample features and corresponding identity information into the face recognition model, calculates the gradient of the objective function with respect to the parameters in the face recognition model and the preset coefficients that need to be determined, and adjusts the parameters of the face recognition model and the preset coefficients that need to be determined based on the gradient until the change of the updated objective function is less than the set value (that is, the objective function converges), and takes the numerical value of the preset coefficient that needs to be determined at this time as the final value. In this way, the above-mentioned preset coefficients are obtained.

[0106] In some possible implementations, the objective function when obtaining the preset coefficients may be a triple loss function as shown in formula (3):

[0107]

[0108] Among them, M is the number of training samples, and is a sample face image and its features, and is a face sample image and its features with the same identity information as the face sample image, and is the face sample image and its features that are different from the identity information of the face sample; α is the difference between the expected distance between the positive sample pairs and the distance between the negative sample pairs. When the distance between the negative sample pairs is greater than the distance between the positive sample pairs by α, the loss function value of the triplet is 0, otherwise it is greater than 0.

[0109] In the embodiment of the present application, the purpose of making the features of the same identity as close as possible and the features of different identities as far as possible can be achieved by minimizing the objective function. It should be noted that the embodiment of the present application has no restrictions on the form of the objective function, and the objective function that can be used to train a single face recognition model can be used for the technical solution described in the embodiment of the present application.

[0110] In the second manner, the preset coefficients may be pre-configured for the second feature extraction model. In this case, N preset coefficients with equal values ​​are configured for the N second feature extraction models.

[0111] In a third manner, the preset coefficients may be configured for N second feature extraction models according to preset evaluation criteria. For example, taking the retrieval accuracy as the evaluation criterion, assuming that there are three second feature extraction models A, B and C in the face retrieval device, and the retrieval accuracy of these second feature extraction models are 0.98, 0.95 and 0.9 respectively, the face retrieval device may determine the preset coefficients of the second feature extraction models A, B and C as 0.98, 0.95 and 0.9; for another example, taking the similarity between face sample images with the same identity information as the evaluation criterion, assuming that the feature extraction model is for a batch of identity information, each identity information corresponds to multiple different face sample images, and for each identity information, the face retrieval device calculates the average similarity S between the features extracted by the second feature extraction model for each pair of face sample images a , after obtaining the S corresponding to all identity information a Then, calculate these S a The average value of AS a , such as the AS of the second feature extraction model A, B and C a If they are 0.8, 0.6 and 0.5 respectively, the preset coefficients of the second feature extraction models A, B and C are determined to be 0.8, 0.6 and 0.5.

[0112] In the fourth method, the preset coefficients can be obtained based on hyperparameter search, and all methods used for hyperparameter search can be used for coefficient search. Specifically, the face retrieval device can obtain N second feature extraction model corresponding coefficient combinations within the preset coefficient range, and then multiply the N second face sample features with the coefficient combinations, and then splice the multiplied N second face sample features to obtain spliced ​​face sample features; finally, the face retrieval device performs face retrieval on the face sample image based on the spliced ​​face sample features to obtain the preset coefficients in the coefficient combination that meet the preset conditions. For example, in a grid search, it is assumed that there are three second feature extraction models A, B and C in a face retrieval device, and their coefficient ranges are all from 0 to 1. The coefficient ranges of A, B and C are equally divided into 10 parts, that is, the coefficients of A, B and C are 0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9 and 1, for a total of 11 coefficients. Therefore, there are 11×11×11=1331 coefficient combinations for A, B and C. These 1331 coefficient combinations are used to splice the vital features of N face samples, and the spliced ​​face sample features are used for face retrieval, and the set of hyperparameters with the highest retrieval accuracy (that is, the coefficient combination of A, B and C) is determined as the final preset coefficients.

[0113] Of course, the above-mentioned methods are only examples for determining the preset coefficients. The face retrieval device can also determine the preset coefficients in other ways, and the embodiments of the present application do not make specific limitations.

[0114] In some possible implementations, the face retrieval device can also jointly train the mapping of multiple face sample features. The above-mentioned second feature mapping model includes a unique module and a shared module. Assume that a neural network model contains 7 layers, of which the first 4 layers can be unique modules and the last 3 layers can be shared modules. The shared modules and the unique modules are actually neural network layers. The difference between them is that the parameters of the unique modules can be changed more flexibly to adapt to the characteristics of each second face sample feature itself, while the parameters of the shared modules need to process the input of multiple unique modules and comprehensively utilize all second face sample features. Its parameters will have stronger restrictions during the training process. It can be seen from this that the unique module can learn the characteristics of the feature itself, and the shared module can learn the common attributes of the features of each model.

[0115] Correspondingly, the above-mentioned S603 may include: inputting the face sample image and multiple first face sample features into a unique module for feature mapping to obtain an output third face sample feature, wherein the multiple first face sample features are extracted from the face sample image through different multiple first feature extraction models; inputting the third face sample feature into a shared module to obtain standard features corresponding to the multiple first face sample features; training the unique module and the shared module according to the face sample image, the multiple first face sample features, the standard features corresponding to the multiple first face sample features, and the target features corresponding to the face sample image to obtain a first feature mapping model.

[0116] For example, a face sample image is input into multiple first feature extraction models to obtain multiple first face sample features as output. The face retrieval device takes each first face sample feature and the face sample image as input at the same time, and the optimization goal is to fit the target feature corresponding to the face sample image. Fig. 9 This is a flow chart of the method for training unique modules and shared modules in the embodiment of the present application, see Fig. 9 As shown, there are two first feature extraction models A and B in the face retrieval device. After the face sample image is processed by the two first feature extraction models A and B to obtain the first face sample features A and B, the third face sample features are output through the unique module A of the second feature mapping model A and the unique module B of the second mapping model B. At the same time, the original face sample image is used as the input of the unique modules of the second feature mapping models A and B, and finally the standard features F are output through the shared modules of the mapping models A and B. A and F B , and the training optimization goal is to be as close as possible to the target features of the face sample image, that is, FA should be as similar as possible to the target feature F, B It should also be as similar as possible to F. When the objective function converges, the second feature mapping model is trained, and the trained second feature model is the first feature mapping model.

[0117] Furthermore, after jointly training each first feature mapping model, Fig.10 This is a flow chart of a method for performing feature mapping on the first feature mapping model in an embodiment of the present application, see Fig.10 As shown, each first feature mapping model can be used independently to map different facial features to standard features and be used for face retrieval.

[0118] In some possible implementations, the second feature mapping model includes an image branch module, a feature branch module and a comprehensive module, wherein the image branch module may be a convolutional neural network, the feature branch module and the comprehensive module may be a fully connected neural network, and the fully connected neural network and the convolutional neural network have similar functions, except that the connection method of neurons in the network is different.

[0119] Correspondingly, the above S603 may include: inputting the face sample image into the image branch module to obtain the output fourth face sample feature; inputting the first face sample feature into the feature branch module to obtain the output fifth face sample feature, the first face sample feature is extracted from the face sample image through the first feature extraction model; inputting the fourth and fifth face sample features together into the comprehensive module to obtain the standard feature corresponding to the first face sample feature; training the image branch module, the feature branch module and the comprehensive module according to the face sample image, the first face sample feature, the standard feature corresponding to the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model.

[0120] For example, a face sample image is input into a first feature extraction model to obtain an output first face sample feature. The face retrieval device uses the face sample image and the first face sample feature as inputs to a first feature mapping model at the same time. The optimization goal is to fit the target feature corresponding to the face sample image. Fig.11 This is a flow chart of the method for training the image branch module, feature branch module and comprehensive module in the embodiment of the present application, see Fig.11As shown, the face sample image is passed through the first feature extraction model to obtain the first face sample feature, the first face sample feature is passed through the feature branch module to obtain the fifth face sample feature after output, the face sample image is passed through the image branch module to obtain the fourth face sample feature after output, the fourth face sample feature and the fifth face sample feature are input into the comprehensive module together to obtain the output standard feature, and the training optimization goal is to be as close as possible to the target feature of the face sample image. When the objective function converges, the second feature mapping model is trained, and the trained second feature model is the first feature mapping model. Furthermore, after each first feature mapping model is trained, each first feature mapping model can be used to map the face features obtained by the corresponding first feature extraction model to the standard feature, and used for face retrieval.

[0121] It should be understood that in this embodiment, the target features used in training the second feature mapping model may be obtained only by the manufacturer, or may be obtained by the manufacturer in collaboration with other manufacturers.

[0122] As can be seen from the above, in the embodiment of the present application, by taking the facial features and the facial image as the input of the feature mapping model, when it is difficult to obtain the appropriate standard features using only the facial features, more appropriate standard features are obtained through the additional information provided by the facial image, thereby improving the accuracy of face retrieval. In addition, by splicing the features extracted by multiple feature extraction models and using the spliced ​​features as the basis for constructing the standard features, the face retrieval device can utilize the combined effect of multiple feature extraction models to select the appropriate feature space and improve the accuracy of face retrieval. Furthermore, since each facial image only needs to pass through one feature extraction model and one feature mapping model to obtain the standard features, the amount of calculation of the system will not increase exponentially with the number of models, thereby reducing the amount of calculation of the system. Furthermore, since the feature mapping model and the feature extraction model are one-to-one corresponding, the number of feature mapping models is consistent with the number of feature extraction models, thereby eliminating the need for the face retrieval device to train a huge number of feature mapping models, thereby reducing the amount of calculation of the system.

[0123] Based on the same inventive concept as the above method, an embodiment of the present application provides a face retrieval device, which can be a face retrieval device in the face retrieval device described in the above embodiment or a chip or system on chip in the face retrieval device, or a functional module in the face retrieval device for implementing the methods described in the above embodiments. The face retrieval device can implement the functions performed by the face retrieval device in the above embodiments, and the functions can be implemented by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, in one possible implementation, Fig.12 FIG. 1 is a schematic diagram of the structure of the face retrieval device in the embodiment of the present application, see Fig.12 As shown, the face retrieval device 1200 includes: an interface module 1201, used to obtain a face image to be retrieved; a feature extraction module 1202, used to input the face image into a first feature extraction model to obtain face features; a feature mapping module 1203, used to input the face image and face features into a first feature mapping model to obtain standard features corresponding to the output face features, the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model, and the first feature mapping model is trained according to the target features corresponding to the face image; a face retrieval module 1204, used to perform face retrieval on the face image according to the standard features.

[0124] In some possible embodiments, the above-mentioned device also includes: a mapping model training module, used to obtain a face sample image; input the face sample image into the first feature extraction model to obtain an output first face sample feature; train the second feature mapping model according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain a first feature mapping model, and the second feature mapping model corresponds one-to-one to the first feature extraction model.

[0125] In some possible embodiments, the above-mentioned device also includes: a target feature acquisition module, which is used to obtain a face sample image before the mapping model training module obtains the first feature mapping model that satisfies the objective function; inputs the face sample image into N second feature extraction models to obtain N second face sample features as output, where N is a positive integer greater than or equal to 2; multiplies the N second face sample features with N preset coefficients one by one; splices the multiplied N second face sample features to obtain spliced ​​face sample features; and obtains the target features corresponding to the face sample image based on the spliced ​​face sample features.

[0126] In some possible implementations, the target feature acquisition module is also used to acquire a face sample image, which has identity information; the face sample image is input into N second feature extraction models to obtain N second face sample features as output; and face recognition is performed on the face sample image based on the N second face sample features and the identity information to obtain N preset coefficients.

[0127] In some possible implementations, the target feature acquisition module is specifically used to configure N preset coefficients for N second feature extraction models, and the N preset coefficients are equal; or, according to preset evaluation criteria, configure N preset coefficients for N second feature extraction models.

[0128] In some possible implementations, the target feature acquisition module is specifically used to obtain N second feature extraction model corresponding coefficient combinations within a preset coefficient range; multiply the N second face sample features by the coefficient combinations accordingly; splice the multiplied N second face sample features to obtain spliced ​​face sample features; perform face retrieval on the face sample image based on the spliced ​​face sample features to obtain preset coefficients in the coefficient combination that meet preset conditions.

[0129] In some possible implementations, the target feature acquisition module is further used to reduce the dimension of the spliced ​​face sample features; and determine the face sample features after dimensionality reduction as the target features corresponding to the face sample image.

[0130] In some possible implementations, the second feature mapping model includes a unique module and a shared module;

[0131] Correspondingly, the mapping model training module is also used to input the face sample image and multiple first face sample features into the unique module to obtain the output third face sample features, where the multiple first face sample features are extracted from the face sample image through different multiple first feature extraction models; input the third face sample features into the shared module to obtain standard features corresponding to the multiple first face sample features; train the unique module and the shared module according to the face sample image, the multiple first face sample features, the standard features corresponding to the multiple first face sample features, and the target features corresponding to the face sample image to obtain the first feature mapping model.

[0132] It should also be noted that the specific implementation process of the interface module 1201, the feature extraction module 1202, the feature mapping module 1203, the face retrieval module 1204, the mapping model training module and the target feature acquisition module can be referred to. Figures 4 to 11 The detailed description of the embodiment is not repeated here for the sake of brevity. In the embodiment of the present application, the interface module 1201 can be used to execute S501 in the above embodiment, the feature extraction module 1202 can be used to execute S502 in the above embodiment, the feature mapping module 1203 can be used to execute S503 in the above embodiment, and the face retrieval module 1203 can be used to execute S504 in the above embodiment.

[0133] The interface module mentioned in the embodiments of the present application may be a receiving interface, a receiving circuit or a receiver, etc.; the feature extraction module, the feature mapping module, the face retrieval module, the mapping model training module and the target feature acquisition module may be one or more processors.

[0134] Based on the same inventive concept as the above method, the embodiment of the present application provides a face retrieval device, Fig.13This is a schematic diagram of the structure of the face retrieval device in the embodiment of the present application, see Fig.13 As shown by the solid line in the middle, the face retrieval device 1300 may include: a processor 1301 and a communication interface 1302. The processor 1301 may be used to support the face retrieval device 1300 to implement the functions involved in the above-mentioned embodiments. For example, the processor 1301 may obtain the face image to be retrieved through the communication interface 1302.

[0135] In some possible implementations, see Fig.13 As shown by the dotted line, the face retrieval device 1300 may also include a memory 1303, which is used to store computer-executable instructions and data necessary for the face retrieval device 1300. When the face retrieval device 1300 is running, the processor 1301 executes the computer-executable instructions stored in the memory 1303, so that the face retrieval device 1300 performs the face retrieval method described in the above embodiments.

[0136] Based on the same inventive concept as the above method, an embodiment of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are run on a computer, they are used to execute the face retrieval methods described in the above embodiments.

[0137] Based on the same inventive concept as the above method, an embodiment of the present application provides a computer program or a computer program product. When the computer program or the computer program product is executed on a computer, the computer implements the face retrieval method described in the above embodiments.

[0138] Those skilled in the art will appreciate that the functions described in conjunction with the various illustrative logic blocks, modules, and algorithm steps disclosed herein can be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions described in the various illustrative logic blocks, modules, and steps can be stored or transmitted as one or more instructions or codes on a computer-readable medium and executed by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to tangible media, such as data storage media, or includes any media that facilitates the transfer of computer programs from one place to another (e.g., according to a communication protocol). In this manner, computer-readable media may generally correspond to (1) non-temporary tangible computer-readable storage media, or (2) communication media, such as signals or carrier waves. Data storage media can be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and / or data structures for implementing the techniques described in this application. A computer program product may include computer-readable media.

[0139] As an example and not limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage, flash memory, or any other medium that can be used to store the desired program code in the form of instructions or data structures and can be accessed by a computer. Also, any connection is properly referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio and microwave, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of media. However, it should be understood that the computer-readable storage media and data storage media do not include connections, carrier waves, signals or other temporary media, but are actually directed to non-temporary tangible storage media. As used herein, disks and optical disks include compact disks (CDs), laser optical disks, optical optical disks, digital versatile disks (DVDs), and Blu-ray disks, where disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0140] Instructions may be executed by one or more processors, such as one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Thus, the term "processor" as used herein may refer to any of the aforementioned structures or any other structures suitable for implementing the techniques described herein. Additionally, in some aspects, the functions described by the various illustrative logic blocks, modules, and steps described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Moreover, the techniques may be fully implemented in one or more circuits or logic elements.

[0141] The techniques of the present application may be implemented in a variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC), or a set of ICs (e.g., a chipset). Various components, modules, or units are described in this application to emphasize functional aspects of devices for performing the disclosed techniques, but do not necessarily need to be implemented by different hardware units. In fact, as described above, the various units may be combined in a codec hardware unit in conjunction with appropriate software and / or firmware, or provided by interoperating hardware units (including one or more processors as described above).

[0142] In the above embodiments, the description of each embodiment has different emphases. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0143] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A face retrieval method, characterized in that: include: Obtain the face image to be retrieved; Extracting features from the facial image using a first feature extraction model to obtain a first facial feature; Inputting the face image and the first face feature into a first feature mapping model to obtain a standard feature corresponding to the first face feature as output, wherein the first feature mapping model is trained based on the target feature corresponding to the face sample image; the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model; Perform face retrieval on the face image according to the standard features.

2. The method according to claim 1, characterized in that The target feature corresponding to the face sample image is obtained by splicing multiple face sample features, and the multiple face sample features are obtained by extracting features from the face sample image by multiple second feature extraction models; the multiple second feature extraction models include the first feature extraction model; the multiple second feature extraction models have at least different training samples, model structures, training strategies or feature dimensions.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Obtain a sample face image; Inputting the face sample image into the first feature extraction model to obtain output first face sample features; The second feature mapping model is trained according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model, and the second feature mapping model corresponds to the first feature extraction model.

4. The method according to claim 3, characterized in that Before training the second feature mapping model according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model that satisfies the target function, the method further includes: Obtain a sample face image; Inputting the face sample image into N second feature extraction models to obtain N second face sample features as output, where N is a positive integer greater than or equal to 2; Multiplying the N second face sample features by N preset coefficients one by one; splicing the multiplied N second face sample features to obtain spliced ​​face sample features; According to the spliced ​​face sample features, a target feature corresponding to the face sample image is obtained, wherein the dimension of the target feature is less than or equal to the sum of the dimensions of the N second feature extraction models.

5. The method according to claim 4, characterized in that The method further comprises: Acquire a sample face image, wherein the sample face image has identity information; Inputting the face sample image into the N second feature extraction models to obtain the N second face sample features as output; According to the N second face sample features and the identity information, face recognition is performed on the face sample image to obtain the N preset coefficients.

6. The method according to claim 4, characterized in that The method further comprises: configuring the N preset coefficients for the N second feature extraction models, wherein the N preset coefficients are equal; or, According to a preset evaluation criterion, the N preset coefficients are configured for the N second feature extraction models.

7. The method according to claim 4, characterized in that The method further comprises: Within a preset coefficient range, obtaining coefficient combinations corresponding to the N second feature extraction models; Multiplying the N second face sample features by the coefficient combination correspondingly; splicing the multiplied N second face sample features to obtain spliced ​​face sample features; According to the features of the spliced ​​face samples, face retrieval is performed on the face sample images to obtain the preset coefficients in the coefficient combination that meet the preset conditions.

8. The method according to any one of claims 4 to 7, characterized in that: The step of obtaining target features corresponding to the face sample images according to the spliced ​​face sample features includes: Performing dimensionality reduction on the features of the spliced ​​face samples; The face sample features after dimensionality reduction are determined as target features corresponding to the face sample images.

9. The method according to any one of claims 4 to 7, characterized in that: The second feature mapping model includes a unique module and a shared module; The training of the second feature mapping model according to the face sample image, the first face sample feature, and the target feature corresponding to the face sample image to obtain the first feature mapping model includes: Inputting the face sample image and a plurality of first face sample features into the unique module to obtain a third face sample feature after output, wherein the plurality of first face sample features are extracted from the face sample image through a plurality of different first feature extraction models; Inputting the third face sample feature into the sharing module to obtain standard features corresponding to the plurality of first face sample features; The unique module and the shared module are trained according to the face sample image, the multiple first face sample features, the standard features corresponding to the multiple first face sample features, and the target features corresponding to the face sample image to obtain the first feature mapping model.

10. The method according to any one of claims 4 to 7, characterized in that: The second feature mapping model includes an image branch module, a feature branch module and a comprehensive module; The training of the second feature mapping model according to the face sample image, the first face sample feature, and the target feature corresponding to the face sample image to obtain the first feature mapping model includes: Inputting the face sample image into the image branch module to obtain a fourth face sample feature after output; Inputting the first face sample feature into the feature branch module to obtain a fifth face sample feature after output, wherein the first face sample feature is extracted from the face sample image through a first feature extraction model; Inputting the fourth face sample feature and the fifth face sample feature into the integration module to obtain a standard feature corresponding to the first face sample feature; The image branch module, the feature branch module and the comprehensive module are trained according to the face sample image, the first face sample feature, the standard feature corresponding to the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model.

11. The method according to any one of claims 1 to 2 and 4 to 7, characterized in that: The step of performing face retrieval on the face image according to the standard features includes: Determining a similarity between the standard feature and a standard feature of a first face sample image, wherein the first face sample image is any face sample image among a plurality of face sample images; When the similarity is greater than a first threshold, the first face sample image is a target for face image retrieval.

12. A face retrieval device, characterized in that: include: An interface module, used for acquiring a face image to be retrieved; A feature extraction module, used to extract features from the face image using a first feature extraction model to obtain a first face feature; a feature mapping module, configured to input the face image and the first face feature into a first feature mapping model, and obtain a standard feature corresponding to the first face feature as output, wherein the first feature mapping model is trained based on the target feature corresponding to the face sample image; and the feature output dimension of the first feature extraction model is the same as the feature input dimension of the first feature mapping model; The face retrieval module is used to perform face retrieval on the face image according to the standard features.

13. The device according to claim 12, characterized in that The target feature corresponding to the face sample image is obtained by splicing multiple face sample features, and the multiple face sample features are obtained by extracting features from the face sample image by multiple second feature extraction models; the multiple second feature extraction models include the first feature extraction model; the multiple second feature extraction models have at least different training samples, model structures, training strategies or feature dimensions.

14. The device according to claim 12 or 13, characterized in that The device also includes: a mapping model training module, which is used to obtain a face sample image; input the face sample image into the first feature extraction model to obtain an output first face sample feature; train a second feature mapping model according to the face sample image, the first face sample feature and the target feature corresponding to the face sample image to obtain the first feature mapping model, and the second feature mapping model corresponds one-to-one to the first feature extraction model.

15. The device according to claim 14, characterized in that The device also includes: a target feature acquisition module, which is used to obtain a face sample image before the mapping model training module obtains the first feature mapping model that satisfies the target function; inputs the face sample image into N second feature extraction models to obtain output N second face sample features, where N is a positive integer greater than or equal to 2; multiplies the N second face sample features with N preset coefficients one by one; splices the multiplied N second face sample features to obtain spliced ​​face sample features; and obtains a target feature corresponding to the face sample image based on the spliced ​​face sample features, wherein the dimension of the target feature is less than or equal to the sum of the dimensions of the N second feature extraction models.

16. The device according to claim 15, characterized in that The target feature acquisition module is also used to obtain a face sample image, the face sample image having identity information; input the face sample image into N second feature extraction models to obtain the N second face sample features as output; perform face recognition on the face sample image based on the N second face sample features and the identity information to obtain the N preset coefficients.

17. The device according to claim 15, characterized in that The target feature acquisition module is specifically used to configure the N preset coefficients for the N second feature extraction models, and the N preset coefficients are equal; or, according to a preset evaluation criterion, configure the N preset coefficients for the N second feature extraction models.

18. The device according to claim 15, characterized in that The target feature acquisition module is specifically used to obtain N second feature extraction model corresponding coefficient combinations within a preset coefficient range; multiply the N second face sample features by the coefficient combinations accordingly; splice the multiplied N second face sample features to obtain spliced ​​face sample features; perform face retrieval on the face sample image based on the spliced ​​face sample features to obtain the preset coefficients in the coefficient combination that meet the preset conditions.

19. The device according to any one of claims 15 to 18, characterized in that The target feature acquisition module is also used to reduce the dimension of the spliced ​​face sample features; and determine the face sample features after dimension reduction as the target features corresponding to the face sample image.

20. The device according to any one of claims 15 to 18, characterized in that The second feature mapping model includes a unique module and a shared module; The mapping model training module is also used to input the face sample image and multiple first face sample features into the unique module to obtain the output third face sample features, where the multiple first face sample features are extracted from the face sample image through different multiple first feature extraction models; input the third face sample features into the shared module to obtain standard features corresponding to the multiple first face sample features; train the unique module and the shared module according to the face sample image, the multiple first face sample features, the standard features corresponding to the multiple first face sample features, and the target features corresponding to the face sample image to obtain the first feature mapping model.

21. The device according to any one of claims 15 to 18, characterized in that The second feature mapping model includes an image branch module, a feature branch module and a comprehensive module; The mapping model training module is also used to input the face sample image into the image branch module to obtain the fourth face sample feature after output; input the first face sample feature into the feature branch module to obtain the fifth face sample feature after output, and the first face sample feature is extracted from the face sample image through the first feature extraction model; input the fourth face sample feature and the fifth face sample feature into the comprehensive module to obtain the standard feature corresponding to the first face sample feature; according to the face sample image, the first face sample feature, the standard feature corresponding to the first face sample feature and the target feature corresponding to the face sample image, the image branch module, the feature branch module and the comprehensive module are trained to obtain the first feature mapping model.

22. The device according to any one of claims 12 to 13, 15 to 18, characterized in that The face retrieval module is specifically used for: Determining a similarity between the standard feature and a standard feature of a first face sample image, wherein the first face sample image is any face sample image among a plurality of face sample images; When the similarity is greater than a first threshold, the first face sample image is a target for face image retrieval.

23. A face retrieval device, characterized in that: include: Processor and communication interface; The communication interface is coupled to the processor, and the processor obtains the face image to be retrieved through the communication interface; The processor is used to support the face retrieval device to implement the face retrieval method described in any one of claims 1 to 11.

24. The face retrieval device according to claim 23, characterized in that: The face retrieval device also includes: a memory for storing computer execution instructions and data necessary for the face retrieval device; when the face retrieval device is running, the processor executes the computer execution instructions stored in the memory so that the face retrieval device performs the face retrieval method as described in any one of claims 1 to 11.

Images