A face comparison and recognition method, device and medium based on deep feature combination

By adopting the deep feature joint method in facial recognition technology, a three-dimensional spatial coordinate system is established and a CNN network is used to compare face data, the problem of insufficient face recognition accuracy at different shooting angles is solved, and higher recognition accuracy is achieved degree and robustness.

CN118762391BActive Publication Date: 2024-11-26CHENGDU YUNZHITIANXIA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411242625.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-05
Publication Date
2024-11-26
Estimated Expiration
2044-09-05

AI Technical Summary

Technical Problem

When existing facial recognition technology processes images from different shooting angles, it is difficult to obtain complete facial features data, resulting in insufficient recognition accuracy and robustness. The existing methods require finding face images frame by frame, which is time-consuming and labor-intensive and difficult to meet actual needs. .

Method used

Using a face comparison and recognition method based on deep feature combination, a three-dimensional spatial coordinate system is established by obtaining local features of preset face image data, a three-dimensional spatial coordinate system is calculated, the relative center coordinates of the face data to be identified, the positioning image data is obtained, and the CNN is used. The network performs comparison and outputs similar values ​​as recognition results.

Benefits of technology

It improves the recognition accuracy and robustness of facial data at different shooting angles, simplifies the recognition process, and meets the application needs of facial recognition technology under multi-angle data.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118762391B_ABST
    Figure CN118762391B_ABST
Patent Text Reader

Abstract

The present invention discloses a face comparison and recognition method, device and medium based on deep feature combination, and relates to the technical field of face comparison and recognition. The present invention obtains local features of preset face image data and its spatial composition in a three-dimensional space coordinate system, substitutes face data to be recognized into the three-dimensional space coordinate system, and obtains the relative center coordinates of the face data to be recognized, further obtains the shooting angle, that is, the imaging space position, through the relative center coordinates, and then returns to the positioning image data to obtain a two-dimensional image of the preset face image data at the same imaging space position, and compares it with the face data to be recognized, so that face data at different shooting angles can be easily recognized, the accuracy and robustness of recognition can be improved, and improvements are made to the application and development of face recognition technology.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of face comparison and recognition, and in particular to a face comparison and recognition method, device and medium based on deep feature combination. Background Art

[0002] Face recognition is a machine vision recognition technology that extracts facial features based on face photos and then identifies identity information. In recent years, with the rapid development of deep neural networks and their application in the field of face recognition, the effect of face recognition has been greatly improved. In addition, face recognition technology has been widely used in many fields such as finance, education, public security, and medical care. Existing face recognition usually only obtains the similarity value of each facial feature data in a two-dimensional plane as the recognition result, which is more suitable for frontal recognition. However, in the images obtained by actual videos, due to the change of shooting position, usually only face data at a specific angle can be obtained. The face data at this specific angle may have some missing facial features, and the distribution position of the facial features is different from the plane data. Generally, before using the existing two-dimensional plane face recognition method, it is necessary to search for image data containing complete frontal facial features frame by frame. This method is time-consuming and labor-intensive and difficult to meet the use requirements. Summary of the invention

[0003] In order to overcome the above-mentioned defects, the present invention provides a face comparison and recognition method, device and medium based on deep feature combination. The present invention can easily recognize face data under different shooting angles, improve the accuracy and robustness of recognition, and make improvements to the application and development of face recognition technology.

[0004] A face comparison and recognition method based on deep feature combination includes the following steps:

[0005] Acquire preset face image data, the preset face image data including preset front face data and preset side face data, collect local features of the preset face image data, and establish a three-dimensional space coordinate system;

[0006] Acquire the imaging spatial position of the face data to be identified, where the imaging spatial position is the vector angle between the assumed camera device and the relative center coordinates of the face data to be identified in the three-dimensional space coordinate system, and obtain the positioning image data of the preset face image data at the same imaging spatial position;

[0007] The CNN network structure is used to compare the positioning image data and the face data to be identified, and a similarity value between the face data to be identified and the positioning image data is output as a recognition result.

[0008] Preferably, when establishing the three-dimensional space coordinate system, it is obtained by the following steps:

[0009] Collecting local features of the preset face image data, the local features including: left eyeball, right eyeball, nose tip, mouth corner, chin, left earlobe and right earlobe;

[0010] A two-dimensional rectangular coordinate system is established with the front face data of the preset face image data, wherein the two-dimensional rectangular coordinate system takes the right earlobe of the preset face image data as the origin, establishes a Y axis in the direction from the right earlobe to the top, and establishes an X axis in the direction from the right earlobe to the left earlobe;

[0011] A Z axis is established in the horizontal direction from the right earlobe to the nose tip in the side face data of the preset facial image data, and a three-dimensional space coordinate system is established by combining the two-dimensional rectangular coordinate system and the Z axis. The unit length of the X axis, the Y axis and the Z axis is the distance value between two adjacent pixel points.

[0012] Preferably, when obtaining the relative center coordinates of the face data to be recognized, the following steps are specifically included:

[0013] Substitute the local feature points of the face data to be identified into the three-dimensional space coordinate system, and calculate the relative center coordinates of the face data of the face data to be identified according to the coordinates of the local feature points in the three-dimensional space coordinate system.

[0014] Preferably, when calculating the relative center coordinates of the face data of the face data to be identified, the following formula is specifically used for calculation:

[0015]

[0016] is the coordinate of the relative center coordinate in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the x-axis in the three-dimensional space coordinate system, is the distance between the ith local feature point and the origin on the y-axis in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the z-axis in the three-dimensional space coordinate system, is the number of local feature points of the face data to be recognized, It is a dynamic convolution kernel operation.

[0017] Preferably, the step of obtaining the image spatial position of the face data to be identified specifically includes the following steps:

[0018] Obtaining the feature point coordinates and relative center coordinates of the nose tip in the local feature points of the face data to be identified;

[0019] The vector angle between the nose tip feature point coordinates and the relative center coordinates is calculated to be the imaging spatial position of the face data to be identified.

[0020] Preferably, when obtaining the positioning image data of the preset face image data at the same imaging space position, the following steps are specifically included:

[0021] Make the vector angle between the coordinates of the nose tip feature point of the preset face image data and the relative center coordinates consistent with the imaging space position;

[0022] The two-dimensional image data formed by the preset face image data with the imaging space position as the observation point is obtained as the positioning image data.

[0023] Preferably, the use of a CNN network structure to compare the positioning image data with the face data to be identified, and outputting a similarity value between the face data to be identified and the positioning image data as a recognition result, specifically comprises the following steps:

[0024] Using a multi-scale rectangular frame to intercept multiple parts of the 3D face image data, integrating the multiple parts of the image into a training set, and using the training set to train a CNN network to obtain a first training model;

[0025] The two first trained models with the same parameters are combined to form a bilinear joint CNN network. The fully connected layers of the two CNN models are replaced by a joint three-layer fully connected layer. The input of this three-layer fully connected layer is obtained by multiplying the output matrices of the last convolution-pooling layer of the two initial CNNs. The softmax multi-class classifier structure of the last layer is replaced by a binary classifier for judging whether it is the same face. Then the parameters of the three fully connected layers are initialized to zero-mean, variance σ Gaussian distribution random values;

[0026] The face images in the training set are paired in pairs and input into the two ends of the bilinear joint CNN network respectively. After multiple iterations of fine-tuning, a bilinear CNN model for face data comparison is obtained;

[0027] The positioning image data and the face data to be identified are respectively input into the bilinear CNN model, and the reference face image and the detected face image captured in a multi-scale, multi-channel and multi-region manner are subjected to feature binary classification using a three-layer deep autoencoder network, and the feature binary classification result is obtained as a similarity value.

[0028] Preferably, the fine-tuning is: first fixing the parameters of one side structure of the bilinear CNN model, and then adjusting the parameters of the other side CNN model using the gradient descent method.

[0029] According to a second aspect, an electronic device is provided, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and wherein the processor implements the steps of the face comparison and recognition method based on deep feature combination when executing the computer program.

[0030] According to a third aspect, a computer-readable medium having a non-volatile program code executable by a processor is provided, wherein the program code enables the processor to execute the face comparison and recognition method based on deep feature combination.

[0031] The beneficial effects of the present invention are embodied in:

[0032] In the present invention, by obtaining local features of preset facial image data and its spatial composition in a three-dimensional space coordinate system, the facial data to be identified is substituted into the three-dimensional space coordinate system, and the relative center coordinates of the facial data to be identified are obtained. The shooting angle, that is, the imaging space position, is further obtained through the relative center coordinates. Then, the positioning image data is returned to obtain a two-dimensional image of the preset facial image data at the same imaging space position, and the two-dimensional image is compared with the facial data to be identified. In this way, facial data at different shooting angles can be easily identified, the accuracy and robustness of identification can be improved, and improvements are made to the application and development of facial recognition technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation of the present invention or the technical solution in the prior art, the following briefly introduces the drawings required for use in the specific implementation or the description of the prior art.

[0034] Figure 1 A flowchart of a face comparison and recognition method based on deep feature combination provided by the present invention. DETAILED DESCRIPTION

[0035] The following embodiments of the technical solution of the present invention are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and cannot be used to limit the protection scope of the present invention.

[0036] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the present invention belongs.

[0037] like Figure 1 As shown, a face comparison and recognition method based on deep feature combination includes the following steps:

[0038] Acquire preset face image data, the preset face image data including preset front face data and preset side face data, collect local features of the preset face image data, and establish a three-dimensional space coordinate system;

[0039] Acquire the imaging spatial position of the face data to be identified, where the imaging spatial position is the vector angle between the assumed camera device and the relative center coordinates of the face data to be identified in the three-dimensional space coordinate system, and obtain the positioning image data of the preset face image data at the same imaging spatial position;

[0040] The CNN network structure is used to compare the positioning image data and the face data to be identified, and a similarity value between the face data to be identified and the positioning image data is output as a recognition result.

[0041] In this scheme, by obtaining the local features of the preset facial image data and its spatial composition in the three-dimensional space coordinate system, the facial data to be identified is substituted into the three-dimensional space coordinate system, and the relative center coordinates of the facial data to be identified are obtained. The shooting angle, that is, the imaging space position, is further obtained through the relative center coordinates. Then, the positioning image data is returned to obtain a two-dimensional image of the preset facial image data at the same imaging space position, and compared with the facial data to be identified. This makes it possible to easily identify facial data at different shooting angles, improve the accuracy and robustness of recognition, and make improvements to the application and development of facial recognition technology.

[0042] More specifically, when establishing the three-dimensional space coordinate system, it is obtained by the following steps:

[0043] Collecting local features of the preset face image data, the local features including: left eyeball, right eyeball, nose tip, mouth corner, chin, left earlobe and right earlobe;

[0044] A two-dimensional rectangular coordinate system is established with the front face data of the preset face image data, wherein the two-dimensional rectangular coordinate system takes the right earlobe of the preset face image data as the origin, establishes a Y axis in the direction from the right earlobe to the top, and establishes an X axis in the direction from the right earlobe to the left earlobe;

[0045] A Z axis is established in the horizontal direction from the right earlobe to the nose tip in the side face data of the preset facial image data, and a three-dimensional space coordinate system is established by combining the two-dimensional rectangular coordinate system and the Z axis. The unit length of the X axis, the Y axis and the Z axis is the distance value between two adjacent pixel points.

[0046] When establishing a three-dimensional space coordinate system, the feature points of the face model are marked, the feature points include: eyes, nose, mouth, eyebrows, and local features are constructed for each feature point, for example, the outline of the eyes, the outline of the mouth, the outline of the nose, the shape of the eyebrows, the shape of the right earlobe, and the shape of the left earlobe. When the right earlobe of the preset face image data is used as the origin, the point with the lowest vertical height of the right earlobe is used as the origin. When the Z axis is established in the horizontal direction from the right earlobe to the nose tip in the side face data of the preset face image data, the point at which the nose tip is farthest from the right earlobe is used to establish the Z axis;

[0047] The outline of the eye needs to express the center position of the circle formed by the eye.

[0048] More specifically, when obtaining the relative center coordinates of the face data to be recognized, the following steps are specifically included:

[0049] Substitute the local feature points of the face data to be identified into the three-dimensional space coordinate system, and calculate the relative center coordinates of the face data of the face data to be identified according to the coordinates of the local feature points in the three-dimensional space coordinate system.

[0050] When the local feature points of the face data to be identified are substituted into the three-dimensional space coordinate system, the facial features that can be obtained from the face data to be identified are overlapped with the image of the preset face image data in the three-dimensional space coordinate system. This method is to facilitate the subsequent acquisition of positioning image data at the same imaging position.

[0051] More specifically, when calculating the relative center coordinates of the face data to be identified, the following formula is specifically used:

[0052]

[0053] is the coordinate of the relative center coordinate in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the x-axis in the three-dimensional space coordinate system, is the distance between the ith local feature point and the origin on the y-axis in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the z-axis in the three-dimensional space coordinate system, is the number of local feature points of the face data to be recognized, It is a dynamic convolution kernel operation.

[0054] More specifically, the acquisition of the image capturing space position of the face data to be recognized includes the following steps:

[0055] Obtaining the coordinates of each feature point and the relative center coordinates of the local feature points of the face data to be identified;

[0056] The vector angle between the coordinates of each feature point and the relative center coordinate is calculated to be the imaging spatial position of the face data to be identified.

[0057] More specifically, when obtaining the positioning image data of the preset face image data at the same imaging space position, the following steps are specifically included:

[0058] Make the vector angle between the coordinates of the nose tip feature point of the preset face image data and the relative center coordinates consistent with the imaging space position;

[0059] The two-dimensional image data formed by the preset face image data with the imaging space position as the observation point is obtained as the positioning image data.

[0060] More specifically, the use of the CNN network structure to compare the positioning image data with the face data to be identified, and outputting a similarity value between the face data to be identified and the positioning image data as a recognition result, specifically includes the following steps:

[0061] Using a multi-scale rectangular frame to intercept multiple parts of the 3D face image data, integrating the multiple parts of the image into a training set, and using the training set to train a CNN network to obtain a first training model;

[0062] The two first trained models with the same parameters are combined to form a bilinear joint CNN network. The fully connected layers of the two CNN models are replaced by a joint three-layer fully connected layer. The input of this three-layer fully connected layer is obtained by multiplying the output matrices of the last convolution-pooling layer of the two initial CNNs. The softmax multi-class classifier structure of the last layer is replaced by a binary classifier for judging whether it is the same face. Then the parameters of the three fully connected layers are initialized to zero-mean, variance σ Gaussian distribution random values;

[0063] The face images in the training set are paired in pairs and input into the two ends of the bilinear joint CNN network respectively. After multiple iterations of fine-tuning, a bilinear CNN model for face data comparison is obtained;

[0064] The positioning image data and the face data to be identified are respectively input into the bilinear CNN model, and the reference face image and the detected face image captured by a multi-scale, multi-channel and multi-region method are subjected to feature binary classification using a three-layer deep autoencoder network, and the feature binary classification result is obtained as a similarity value.

[0065] A three-layer deep autoencoder network is used to perform data dimensionality reduction and binary classification, and finally the face verification result, that is, the binary classification result, is obtained. If the neuron value of the deep autoencoder network outputting "belongs to the same person" is greater than the neuron value of "does not belong to the same person", it is considered that the input image is the same person. If used for face classification tasks, the reference face corresponding to the highest neuron value of the input face to be classified and the reference face outputting the same person can be used as the classification result.

[0066] More specifically, the fine-tuning is: first fix the parameters of one side of the bilinear CNN model structure, and then adjust the parameters of the other side of the CNN model using the gradient descent method.

[0067] An electronic device includes a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and is characterized in that when the processor executes the computer program, the steps of the face comparison and recognition method based on deep feature combination are implemented.

[0068] Among them, the memory may include a large capacity memory for data or instructions. By way of example and not limitation, the memory may include a hard disk drive, a floppy disk drive, a solid-state drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape or a universal serial bus drive or a combination of two or more of these. Where appropriate, the memory may include a removable or non-removable (or fixed) medium. Where appropriate, the memory may be inside or outside the data processing device. In a specific embodiment, the memory is a non-volatile memory. In a specific embodiment, the memory includes a read-only memory (ROM) and a random access memory. Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM, an electrically erasable PROM, an electrically rewritable ROM or a flash memory (FLASH) or a combination of two or more of these. Where appropriate, the RAM may be a static random access memory) or a dynamic random access memory (DRAM), wherein the DRAM may be a fast page mode dynamic random access memory, an extended data output dynamic random access memory, a synchronous dynamic random access memory, etc.

[0069] The memory may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor.

[0070] The processor implements any one of the face comparison and recognition methods based on deep feature combination in the above embodiments by reading and executing computer program instructions stored in the memory.

[0071] A computer-readable medium having a non-volatile program code executable by a processor, characterized in that the program code enables the processor to execute the face comparison and recognition method based on deep feature combination.

[0072] The readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory, a magnetic memory, a disk, an optical disk, etc. In some embodiments, the readable storage medium may be an internal storage unit of an electronic device, such as a mobile hard disk of the electronic device. In other embodiments, the readable storage medium may also be an external storage device of an electronic device, such as a plug-in mobile hard disk, a smart memory card, a secure digital card, a flash memory card, etc. equipped on the electronic device. The readable storage medium may also include both an internal storage unit and an external storage device of the electronic device. The readable storage medium may be used not only to store application software and various types of data installed in the electronic device, but also to temporarily store data that has been output or is to be output.

[0073] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A face comparison and recognition method based on deep feature combination, characterized in that: The following steps are involved: Acquire preset face image data, the preset face image data including preset front face data and preset side face data, collect local features of the preset face image data, and establish a three-dimensional space coordinate system; Acquire the imaging spatial position of the face data to be identified, where the imaging spatial position is the vector angle between the assumed camera device and the relative center coordinates of the face data to be identified in the three-dimensional space coordinate system, and obtain the positioning image data of the preset face image data at the same imaging spatial position; Using a CNN network structure to compare the positioning image data with the face data to be identified, and outputting a similarity value between the face data to be identified and the positioning image data as a recognition result; When establishing the three-dimensional space coordinate system, it is specifically obtained by the following steps: Collecting local features of the preset face image data, the local features including: left eyeball, right eyeball, nose tip, mouth corner, chin, left earlobe and right earlobe; A two-dimensional rectangular coordinate system is established with the front face data of the preset face image data, wherein the two-dimensional rectangular coordinate system takes the right earlobe of the preset face image data as the origin, establishes a Y axis in the direction from the right earlobe to the top, and establishes an X axis in the direction from the right earlobe to the left earlobe; A Z axis is established in the horizontal direction from the right earlobe to the nose tip in the side face data of the preset face image data, and a three-dimensional space coordinate system is established by combining the two-dimensional rectangular coordinate system and the Z axis, wherein the unit lengths of the X axis, the Y axis and the Z axis are the distance values ​​of two adjacent pixel points; When obtaining the relative center coordinates of the face data to be recognized, the following steps are specifically included: Substituting the local feature points of the face data to be identified into the three-dimensional space coordinate system, and calculating the relative center coordinates of the face data of the face data to be identified according to the coordinates of the local feature points in the three-dimensional space coordinate system; When calculating the relative center coordinates of the face data of the face data to be identified, the following formula is specifically used for calculation: is the coordinate of the relative center coordinate in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the x-axis in the three-dimensional space coordinate system, is the distance between the ith local feature point and the origin on the y-axis in the three-dimensional space coordinate system, is the distance between the i-th local feature point and the origin on the z-axis in the three-dimensional space coordinate system, is the number of local feature points of the face data to be recognized, It is a dynamic convolution kernel operation.

2. The face comparison and recognition method based on deep feature combination according to claim 1 is characterized in that: The acquisition of the image spatial position of the face data to be identified specifically includes the following steps: Obtaining the feature point coordinates and relative center coordinates of the nose tip in the local feature points of the face data to be identified; The vector angle between the nose tip feature point coordinates and the relative center coordinates is calculated to be the imaging spatial position of the face data to be identified.

3. The face comparison and recognition method based on deep feature combination according to claim 2 is characterized in that: When obtaining the positioning image data of the preset face image data at the same imaging space position, the following steps are specifically included: Make the vector angle between the coordinates of the nose tip feature point of the preset face image data and the relative center coordinates consistent with the imaging space position; The two-dimensional image data formed by the preset face image data with the imaging space position as the observation point is obtained as the positioning image data.

4. The face comparison and recognition method based on deep feature combination according to claim 3 is characterized in that: The use of the CNN network structure to compare the positioning image data with the face data to be identified and outputting a similarity value between the face data to be identified and the positioning image data as a recognition result specifically includes the following steps: Using a multi-scale rectangular frame to intercept multiple parts of the 3D face image data, integrating the multiple parts of the image into a training set, and using the training set to train a CNN network to obtain a first training model; The two first trained models with the same parameters are combined to form a bilinear joint CNN network. The fully connected layers of the two CNN models are replaced by a joint three-layer fully connected layer. The input of this three-layer fully connected layer is obtained by multiplying the output matrices of the last convolution-pooling layer of the two initial CNNs. The softmax multi-class classifier structure of the last layer is replaced by a binary classifier for judging whether it is the same face. Then the parameters of the three fully connected layers are initialized to zero-mean, variance σ Gaussian distribution random values; The face images in the training set are paired in pairs and input into the two ends of the bilinear joint CNN network respectively. After multiple iterations of fine-tuning, a bilinear CNN model for face data comparison is obtained; The positioning image data and the face data to be identified are respectively input into the bilinear CNN model, and the reference face image and the detected face image captured in a multi-scale, multi-channel and multi-region manner are subjected to feature binary classification using a three-layer deep autoencoder network, and the feature binary classification result is obtained as a similarity value.

5. The face comparison and recognition method based on deep feature combination according to claim 4 is characterized in that: The fine-tuning is as follows: firstly, the parameters of one side structure of the bilinear CNN model are fixed, and then the parameters of the other side CNN model are adjusted using the gradient descent method.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the computer program, the steps of the face comparison and recognition method based on deep feature combination described in any one of claims 1 to 5 are implemented.

7. A computer-readable medium having a non-volatile program code executable by a processor, characterized in that: The program code enables the processor to execute the face comparison and recognition method based on deep feature combination according to any one of claims 1 to 5.