A cross-spectral face recognition method and monitoring device based on image conversion

Abstract

The present invention belongs to the technical fields of pattern recognition, digital image processing and machine learning, and discloses a cross-spectral face recognition method and monitoring equipment based on image conversion, wherein the cross-spectral face recognition method based on image conversion includes: obtaining a visible light face image set and an infrared face image set, and preprocessing the image set; constructing a cross-spectral bidirectional image conversion network model, and performing model training; inputting a face preprocessing test image into the cross-spectral bidirectional image conversion network model for image bidirectional conversion; performing face matching according to the generated image, and obtaining a visible light face matching score and an infrared face matching score; using a decision fusion algorithm to fuse the matching scores, and using the fused matching scores to perform face recognition. The present invention can solve the problems that the application scope of traditional face recognition technology is narrow and the recognition performance of existing cross-spectral face recognition technology cannot meet the demand well.

Description

Technical Field

[0001] The present invention belongs to the technical fields of pattern recognition, digital image processing and machine learning, and in particular relates to a cross-spectrum face recognition method and monitoring equipment based on image conversion. Background Art

[0002] At present, face recognition is a biometric technology that automatically identifies people based on their facial features. Compared with other biometric features, face has the advantages of universality, low cost of collection, and non-invasiveness, making it one of the most widely used biometric recognition technologies. Although the research results of face recognition technology are remarkable and have been widely used in the industry, most face recognition technologies use the visible light band. This method has insurmountable defects. For example, when the ambient light is unstable or under complex lighting conditions such as at night, the performance of the face recognition algorithm will drop significantly. The infrared band has the advantages of low background light requirements, strong penetration, and imaging in rainy and foggy weather, which can well solve the two problems of light changes and night imaging. Therefore, the use of cross-spectral face recognition technology combining visible light and infrared is particularly urgent in practice. From the current research, cross-spectral face recognition methods can be divided into two categories: methods based on traditional machine learning and methods based on deep learning. Due to the problems that traditional machine learning methods are generally cumbersome and complex in design, good performance for specific situations but poor robustness, they are gradually replaced by deep learning methods.

[0003] With the rapid development of deep learning, some cross-spectral face recognition methods based on deep learning use image generation methods, such as Pix2pix, CycleGAN, etc. However, most of these image generation technologies are designed for image style transfer, not for face images, so the technology itself does not involve the preservation of identity features. The identity preservation problem is particularly critical in face recognition, and the few networks that are used for face image generation are also limited to one-way image generation and cannot achieve the purpose of two-way image conversion. Therefore, directly using these methods cannot well meet the current needs of cross-spectral face recognition for image conversion.

[0004] Through the above analysis, the problems and defects of the existing technology are: the application scope of traditional face recognition technology is narrow and the recognition performance of existing cross-spectral face recognition technology cannot meet the needs well.

[0005] The difficulty of solving the above problems and defects is:

[0006] (1) For the problem of cross-spectral face recognition, the existing deep learning method based on style transfer image generation technology cannot effectively maintain the facial identity feature information before and after image conversion. Therefore, how to effectively maintain the facial identity features during the image conversion process is one of the difficulties in the problem of cross-spectral face recognition.

[0007] (2) In order to improve the accuracy of cross-spectral face recognition, the existing face image generation networks are mostly limited to unidirectional image generation and cannot play the role of averaging errors for bidirectional heterogeneous images. Therefore, it is necessary to design a bidirectional image conversion network and construct a composite loss function for cross-spectral face image generation.

[0008] (3) In order to further improve the accuracy of cross-spectral face recognition, it is necessary to design a suitable decision fusion strategy to fuse the matching scores. Therefore, how to design a decision fusion algorithm to achieve matching score fusion is a very challenging task. Currently, there is little research on this issue in the industry.

[0009] The significance of solving the above problems and defects is:

[0010] (1) Using deep learning theory and image generation technology, a new cross-spectral face recognition method based on image conversion is proposed. This method ensures the preservation of facial identity features during the cross-spectral face image conversion process, and designs a bidirectional image conversion network, which can effectively overcome the image modality differences caused by different spectral bands. Compared with other traditional methods and deep methods, it has higher recognition performance and stronger robustness.

[0011] (2) By designing a decision fusion algorithm, the fused decision can be made into the global optimal decision, which can further improve the accuracy of cross-spectral face recognition. Summary of the invention

[0012] In view of the problems existing in the prior art, the present invention provides a cross-spectrum face recognition method and monitoring equipment based on image conversion.

[0013] The present invention is implemented as follows: a cross-spectral face recognition method based on image conversion, the cross-spectral face recognition method based on image conversion comprising:

[0014] Step 1: Obtain a visible light face image set and an infrared face image set, and divide the visible light face image set and the infrared face image set into a visible light face training and test image set, and an infrared face training and test image set, respectively, and pre-process the visible light face training and test image set, and the infrared face training and test image set, respectively, so as to enhance the quality of the face images in each image set;

[0015] Step 2: construct a cross-spectral bidirectional image conversion network model between visible light images and infrared images for feature learning of face images of each spectrum. The cross-spectral bidirectional image conversion network model is trained by using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model;

[0016] Step 3, inputting the visible light and infrared face preprocessing test images into the trained cross-spectral bidirectional image conversion network model for image bidirectional conversion to obtain a generated visible light image and a generated infrared image;

[0017] Step 4, performing face matching on the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, respectively, to obtain a visible light face matching score and an infrared face matching score;

[0018] Step 5: Use the decision fusion algorithm to fuse the visible light face matching score and the infrared face matching score to achieve the global optimal decision. Use the fused matching score to perform face recognition to achieve cross-spectrum face recognition.

[0019] Furthermore, in the step 1, the visible light face training and test image sets, and the infrared face training and test image sets are preprocessed to improve the subsequent image conversion effect, and a visible light face preprocessing training image set, a visible light face preprocessing test image set, an infrared face preprocessing training image set, and an infrared face preprocessing test image set are obtained respectively;

[0020] The visible light face training image set, the visible light face test image set, the infrared face training image set and the infrared face test image set are preprocessed to obtain a visible light face preprocessing training image set, a visible light face preprocessing test image set, an infrared face preprocessing training image set and an infrared face preprocessing test image set, respectively, including:

[0021] Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0022] The infrared face training image set and the infrared face test image set are respectively subjected to image enhancement and normalization processing to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set.

[0023] Further, the specific process of step 1 is as follows:

[0024] Before performing the bidirectional image conversion, normalizing and contrast adjusting the face images in the visible light face training image set, the visible light face test image set, the infrared face training image set, and the infrared face test image set;

[0025] Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0026] The face images in the visible light face image set are first converted into grayscale images. The grayscale conversion formula is designed as follows:

[0027] I gray =0.2989×R+0.5870×G+0.1140×B (1)

[0028] Among them, R, G, B are the RGB values ​​corresponding to the visible light image, I gray A grayscale image output after grayscale conversion of a visible light image;

[0029] Then, the grayscale image I gray Perform maximum and minimum normalization processing, normalize to [0, 255], and the normalization formula design is as follows:

[0030]

[0031] Among them, I n is the grayscale image I gray After normalization, the image output is max and I min They are grayscale images I gray The maximum and minimum grayscale values ​​in ;

[0032] Each visible light face training image and each visible light face test image in the visible light face training image set and the visible light face test image set are processed by the above formula (1) and formula (2) respectively, thereby obtaining a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0033] Performing image enhancement and normalization processing on the infrared face training image set and the infrared face test image set respectively to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set;

[0034] The face images in the infrared face training image set and the infrared face test image set are first enhanced using the log operator. The image enhancement formula is designed as follows:

[0035] I e =log c+1 (1+cX) (3)

[0036] Among them, I eis the image after image enhancement, X is the image before image enhancement, c is the parameter for adjusting contrast, the default value is c=10, the larger the c is, the more obvious the image contrast enhancement is;

[0037] Then, the enhanced image I e Perform normalization processing, and the normalization formula for normalization to [0, 255] is as shown in the above formula (2);

[0038] Each infrared face training image and infrared face test image in the infrared face training image set and the infrared face test image set is processed by the above formula (3) and formula (2) respectively, thereby obtaining the infrared face preprocessing training image set and the infrared face preprocessing test image set.

[0039] Furthermore, in the step 2, a cross-spectral bidirectional image conversion network model between visible light images and infrared images is constructed, including a Vis-to-IR generator module, an IR-to-Vis generator module, a visible light discriminator module, an infrared discriminator module and an identity preservation module;

[0040] The visible light face training image is used as the input of the Vis-to-IR generator module, and the output of the Vis-to-IR generator module is connected to the input of the IR-to-Vis generator module; the infrared face training image is used as the input of the IR-to-Vis generator module, and the output of the IR-to-Vis generator module is connected to the input of the Vis-to-IR generator module; the input of the visible light discriminator module is the visible light face training image and the output of the IR-to-Vis generator module, and the output of the visible light discriminator module is the matching result of the two input face images, and is reversely transmitted as an adversarial loss function to update the network parameters and optimize the network; the input of the infrared discriminator module is the infrared face training image and the Vis-to-IR generator module The output of the infrared discriminator module is the matching result of the two input face images, which is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; the identity retaining module (IRM) includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence; the input of the identity retaining module is the output of the visible light or infrared face training image and the Vis-to-IR or IR-to-Vis generator module; the output of the identity retaining module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity retaining loss function, which is conducted back to update the network parameters and optimize the network;

[0041] The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model. The specific process is as follows:

[0042] Construct a composite loss function based on adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss;

[0043] The cross-spectral bidirectional image translation network model is trained using visible light and infrared face preprocessing training image sets and a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss to obtain a trained cross-spectral bidirectional image translation network model.

[0044] Further, in step 2, the specific process is:

[0045] Construct a cross-spectral bidirectional image conversion network model, which is a highly robust network model specifically designed for cross-spectral bidirectional image conversion; construct a cross-spectral bidirectional image conversion network model between visible light images and infrared images, including Vis-to-IR generator module G, IR-to-Vis generator module F, and visible light discriminator module D VIS 、Infrared discriminator module D IR and an identity maintenance module IRM, wherein the input of the Vis-to-IR generator module G is connected to the visible light face training image, and the output of the Vis-to-IR generator module G is connected to the input of the IR-to-Vis generator module F; the input of the IR-to-Vis generator module F is connected to the infrared face training image, and the output of the IR-to-Vis generator module F is connected to the input of the Vis-to-IR generator module G; a visible light discriminator module D VIS The input is the visible light face training image and the output of the IR-to-Vis generator module F, and the visible light discriminator module D VIS The output of is the matching result of the two input face images, and is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; infrared discriminator module D IR The input is the infrared face training image and the output of the Vis-to-IR generator module F, the infrared discriminator module D IRThe output of is the matching result of the two input face images, which is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; the identity preservation module IRM includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence; the input of the identity preservation module is the output of the visible light or infrared face training image and the Vis-to-IR or IR-to-Vis generator module; the output of the identity preservation module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity preservation loss function, which is conducted back to update the network parameters and optimize the network;

[0046] The generator module uses an autoencoder network, which consists of 1 convolution layer with a 7*7 convolution kernel, 2 convolution layers with a 3*3 convolution kernel, 9 residual blocks, 2 deconvolution layers with a 3*3 convolution kernel and 1 convolution layer with a 7*7 convolution kernel. The output of each convolution layer will undergo instance regularization, of which the first half is the encoding process and the second half is the decoding process. The network in the discriminator module consists of 9 convolution layers, 5 maximum pooling layers, 2 fully connected layers and 1 binary classification layer. The identity preservation module network includes the ResNet feature extraction module, the feature normalization module and the feature space mapping module in sequence. After feature extraction and normalization of the two input face images, the Euclidean distance is calculated in the Euclidean space and used as the identity preservation loss function. The network parameters are updated and the network is optimized through reverse conduction.

[0047] The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model; during the model training process, a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is constructed, and the sum of adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is specifically used as the composite loss function; the specific expression of the composite loss function is as follows:

[0048] Loss = L GAN +λ cyc L cyc +λ syn L syn +λ pc L pc +λ IRM L IRM (4)

[0049] Among them, L GAN , L cyc , L syn , L pc , L IRMThey represent adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss respectively; λ cyc For L cyc Corresponding coefficient, λ syn For L syn Corresponding coefficient, λ pc For L pc Corresponding coefficient, λ IRM For L IRM Corresponding coefficients; the above coefficients are used to adjust the proportion of each loss in the composite loss function; for the specific experimental data set, their values ​​are set to λ cyc =10,λ syn =30,λ pc =10,λ IRM =10;

[0050] Specific L GAN , L cyc , L syn , L pc and L IRM The designs are shown below:

[0051]

[0052] L cyc (G,F)=E v～Pdata(V) [||F(G(v))-v||1]+E i～Pdata(I) [||G(F(i))-i||1]… (6)

[0053] L syn (G,F)=E v～P(V),i～P(I) [||F(i)-F(G(v))||1]+E v～P(V),i～P(I) [||G(v)-G(F(i))||1]…… (7)

[0054] L pc (G,F)=E v～P(V),i～P(I) [||vF(i)||1]+E v～P(V),i～P(I) [||iG(v)||1]… (8)

[0055]

[0056] Where V and I represent the visible light input image set and infrared input image set corresponding to the cross-spectral bidirectional image conversion network model, respectively, and v and i represent single samples of visible light and infrared input images, respectively; L GAN (G,D IR ,V,I) is composed of D IRThe binary cross entropy loss value is calculated by calculating the discrimination result of i and the discrimination result of v after G, L GAN (F,D VIS ,I,V) by D VIS The binary cross entropy loss value is calculated by calculating the discrimination result of v and the discrimination result of i after F generation;

[0057] Where L cyc It is used to ensure that the cyclically synthesized image is as close to the original image as possible, so as to train a high-precision generator; it is composed of the L1 loss value between v and the image generated by G and then cyclically generated by F, and the L1 loss value between i and the image generated by F and then cyclically generated by G;

[0058] Where L syn It is used to ensure that the generated image is as close as possible to the cyclically synthesized image. It is composed of the L1 loss value between the image generated by i after F and the image generated by v after G and then cyclically generated by F, and the L1 loss value between the image generated by v after G and the image generated by i after F and then cyclically generated by G;

[0059] Where L pc It is used to ensure the accuracy of the low-frequency components of the image in the cross-spectral bidirectional image conversion network. It is composed of the L1 loss value of the image generated by v and i through F, and the L1 loss value of the image generated by i and v through G;

[0060] Where L IRM To ensure the accuracy and consistency of identity information during the bidirectional image conversion process, the original image and the generated cross-spectral image are simultaneously input into the face feature extraction network to obtain the corresponding feature vectors with a dimension of 128. and Then calculate the Euclidean distance between these two features as their loss value;

[0061] The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model; the composite loss function of the cross-spectral bidirectional image conversion network model constructed by the above formula (4) is based on the composite loss function, and the face training images in the visible light face preprocessing training image set and the infrared face preprocessing training image set are input into the cross-spectral bidirectional image conversion network model for training. During the training process, the back propagation algorithm is used to train the cross-spectral bidirectional image conversion network model, thereby obtaining the final trained cross-spectral bidirectional image conversion network model, which is used to subsequently obtain the generated visible light image and the generated infrared image.

[0062] Furthermore, in step 4, face matching is performed on the visible light face preprocessing test image and the generated visible light image, and on the infrared face preprocessing test image and the generated infrared image, respectively, to obtain a visible light face matching score and an infrared face matching score, respectively, including:

[0063] Using a standard single-spectrum face feature extraction model, feature extraction and normalization are performed on the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, to obtain a visible light face preprocessing test image feature vector and a generated visible light image feature vector, as well as an infrared face preprocessing test image feature vector and a generated infrared image feature vector;

[0064] By calculating the Euclidean distance of the feature vector, the feature vector of the visible light face preprocessing test image and the generated visible light image feature vector, as well as the feature vector of the infrared face preprocessing test image and the generated infrared image feature vector are calculated to obtain a visible light face matching score matrix and an infrared face matching score matrix respectively;

[0065] According to the visible light face matching score matrix and the infrared face matching score matrix, the false acceptance rate, true acceptance rate, error rate and normalized spacing of the visible light face matching score matrix and the infrared face matching score matrix are calculated respectively.

[0066] Furthermore, in step 5, the visible light face matching score and the infrared face matching score are fused using a decision fusion algorithm, and the fused matching score is used for face recognition to achieve cross-spectrum face recognition. The specific process is as follows:

[0067] Construct a decision fusion algorithm based on GAR and d-prime, and perform decision fusion on the visible light face matching score matrix and the infrared face matching score matrix according to the decision fusion algorithm based on GAR and d-prime;

[0068] The fused matching score is used for face recognition. The FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to perform face recognition to achieve cross-spectral face recognition.

[0069] Furthermore, in step 5, the specific process is: normalizing the visible light face matching score and the infrared face matching score, fusing the matching scores using a decision fusion algorithm, and performing face recognition using the fused matching scores to achieve cross-spectrum face recognition;

[0070] A decision fusion algorithm based on GAR and d-prime is constructed to fuse the matching scores and calculate the fused matching score MatchScore. The specific formula is designed as follows:

[0071] When d-prime V >d-prime I hour:

[0072]

[0073] When d-prime V <d-prime I hour:

[0074]

[0075] When d-prime V =d-prime I hour:

[0076] The final fusion score is:

[0077] MatchScore=w1×MatchScore V +w2×MatchScore I (13)

[0078] Among them, G V and d-prime V are the GAR value and d-prime value of the visible light face matching score, G I and d-prime I are the GAR value and d-prime value of infrared face matching score, MatchScore V and MatchScore I are the visible light matching score and the infrared matching score respectively, w1 and w2 are the weights calculated by the fusion formula in the fusion process; the above fusion strategy is adopted to adaptively calculate the fusion weight according to the GAR value of the two matching scores, and it can ensure that the fusion weight of the matching matrix with a small d-prime value is greater than 1 / 2, so as to achieve the purpose of improving the fusion effect and matching degree;

[0079] The decision fusion algorithm based on GAR and d-prime fuses the visible light face matching score matrix and the infrared face matching score matrix according to formula (14); the fused matching score MatchScore is used for face recognition, and the FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to achieve cross-spectral face recognition.

[0080] Another object of the present invention is to provide a cross-spectral face recognition monitoring device based on image conversion, the monitoring device includes a visible light camera and an infrared camera, a video recorder, a read-only memory, a display, a router, an external hard disk and a power supply. The read-only memory stores a cross-spectral face recognition computer program, the video recorder is used to execute the computer program and control the information interaction of the entire monitoring device, and when the computer program is executed by the video recorder, the video recorder executes the steps of the cross-spectral face recognition method based on image conversion.

[0081] Combining all the above technical solutions, the advantages and positive effects of the present invention are as follows: the present invention proposes an image conversion model including an identity preservation module, and a bidirectional image conversion idea between visible light and infrared images, in order to generate an image distribution that is close to the real image while effectively maintaining the identity information, thereby achieving the effect of significantly improving the current cross-spectral face recognition performance; and proposes a decision fusion strategy for fusing the two matching scores generated after the bidirectional conversion, and comparing them with the traditional fusion algorithm, so as to further improve the accuracy of cross-spectral face image recognition, and provide basic support for the promotion of cross-spectral face recognition technology.

[0082] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0083] Figure 1 It is a flow chart of a cross-spectral face recognition method based on image conversion provided by an embodiment of the present invention.

[0084] Figure 2 It is a schematic diagram of the structure of a cross-spectral bidirectional image conversion network model in the cross-spectral face recognition method based on image conversion provided by an embodiment of the present invention.

[0085] Figure 3 It is a structural schematic diagram of a generator module and a discriminator module in a cross-spectral face recognition method based on image conversion provided by an embodiment of the present invention;

[0086] Figure 3 Middle: Figure a, generator module; Figure b, discriminator module.

[0087] Figure 4 It is a structural schematic diagram of an identity preservation module in a cross-spectral face recognition method based on image conversion provided by an embodiment of the present invention.

[0088] Figure 5 It is a structural schematic diagram of a cross-spectrum face recognition monitoring device based on image conversion provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0089] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0090] Embodiment 1

[0091] Most current face recognition technologies use visible light imaging, but they usually perform poorly in special imaging environments such as insufficient light and bad weather. Infrared imaging technology has the advantages of low background light requirements and imaging in rainy and foggy weather. Therefore, cross-spectral face recognition technology using visible light and infrared images is particularly urgent in practical applications. However, current cross-spectral face recognition technologies do not consider factors such as bidirectional image conversion and score fusion, resulting in their generally low recognition rates. Therefore, directly using these methods cannot well meet the current face recognition needs in special imaging environments. Based on the above-mentioned problems, an embodiment of the present invention provides a cross-spectral face recognition method based on image conversion, and ordinary technicians in the industry can also use other steps to implement it. Figure 1 The cross-spectral face recognition method based on image conversion provided by the present invention is only a specific embodiment.

[0092] like Figure 1 As shown, the cross-spectral face recognition method based on image conversion provided by the embodiment of the present invention includes:

[0093] S101: Obtain a visible light face image set and an infrared face image set, and divide the visible light face image set and the infrared face image set into a visible light face training and test image set and an infrared face training and test image set, respectively, and preprocess the visible light face training and test image set and the infrared face training and test image set respectively;

[0094] S102: constructing a cross-spectral bidirectional image conversion network model between visible light images and infrared images, and training the cross-spectral bidirectional image conversion network model using a visible light and infrared face preprocessing training image set to obtain a trained cross-spectral bidirectional image conversion network model;

[0095] S103: inputting the visible light and infrared face preprocessing test images into the trained cross-spectral bidirectional image conversion network model for image bidirectional conversion to obtain a generated visible light image and a generated infrared image;

[0096] S104: performing face matching on the visible light face preprocessing test image and the generated visible light image, and on the infrared face preprocessing test image and the generated infrared image, respectively, to obtain visible light face matching scores and infrared face matching scores;

[0097] S105: The visible light face matching score and the infrared face matching score are fused using a decision fusion algorithm, and the fused matching score is used for face recognition to achieve cross-spectrum face recognition.

[0098] In S101 provided in the embodiment of the present invention, the visible light face training and test image set and the infrared face training and test image set are preprocessed to improve the subsequent image conversion effect, and a visible light face preprocessing training image set, a visible light face preprocessing test image set, an infrared face preprocessing training image set and an infrared face preprocessing test image set are obtained respectively;

[0099] The visible light face training image set, the visible light face test image set, the infrared face training image set and the infrared face test image set are preprocessed to obtain a visible light face preprocessing training image set, a visible light face preprocessing test image set, an infrared face preprocessing training image set and an infrared face preprocessing test image set, respectively, including:

[0100] Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0101] The infrared face training image set and the infrared face test image set are respectively subjected to image enhancement and normalization processing to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set.

[0102] The specific process of S101 provided in the embodiment of the present invention is as follows:

[0103] Before performing the bidirectional image conversion, normalizing and contrast adjusting the face images in the visible light face training image set, the visible light face test image set, the infrared face training image set, and the infrared face test image set;

[0104] Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0105] The face images in the visible light face image set are first converted into grayscale images. The grayscale conversion formula is designed as follows:

[0106] I gray =0.2989×R+0.5870×G+0.1140×B (1)

[0107] Among them, R, G, B are the RGB values ​​corresponding to the visible light image, I grayA grayscale image output after grayscale conversion of a visible light image;

[0108] Then, the grayscale image I gray Perform maximum and minimum normalization processing, normalize to [0, 255], and the normalization formula design is as follows:

[0109]

[0110] Among them, I n is the grayscale image I gray After normalization, the image output is max and I min They are grayscale images I gray The maximum and minimum grayscale values ​​in ;

[0111] Each visible light face training image and each visible light face test image in the visible light face training image set and the visible light face test image set are processed by the above formula (1) and formula (2) respectively, thereby obtaining a visible light face preprocessing training image set and a visible light face preprocessing test image set;

[0112] Performing image enhancement and normalization processing on the infrared face training image set and the infrared face test image set respectively to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set;

[0113] The face images in the infrared face training image set and the infrared face test image set are first enhanced using the log operator. The image enhancement formula is designed as follows:

[0114] I e =log c+1 (1+cX) (3)

[0115] Among them, I e is the image after image enhancement, X is the image before image enhancement, c is the parameter for adjusting contrast, the default value is c=10, the larger the c is, the more obvious the image contrast enhancement is;

[0116] Then, the enhanced image I e Perform normalization processing, and the normalization formula for normalization to [0, 255] is as shown in the above formula (2);

[0117] Each infrared face training image and infrared face test image in the infrared face training image set and the infrared face test image set is processed by the above formula (3) and formula (2) respectively, thereby obtaining the infrared face preprocessing training image set and the infrared face preprocessing test image set.

[0118] In S102 provided in the embodiment of the present invention, a cross-spectral bidirectional image conversion network model between a visible light image and an infrared image is constructed, including a Vis-to-IR generator module, an IR-to-Vis generator module, a visible light discriminator module, an infrared discriminator module, and an identity preservation module;

[0119] Among them, the visible light face training image is used as the input of the Vis-to-IR generator module, and the output of the Vis-to-IR generator module is connected to the input of the IR-to-Vis generator module; the infrared face training image is used as the input of the IR-to-Vis generator module, and the output of the IR-to-Vis generator module is connected to the input of the Vis-to-IR generator module; the input of the visible light discriminator module is the visible light face training image and the output of the IR-to-Vis generator module, the output of the visible light discriminator module is the matching result of the two input face images, and is used as an adversarial loss function to back-conduct to update the network parameters and optimize the network; the input of the infrared discriminator module is the infrared face training image and the output of the Vis-to-IR generator module, the output of the infrared discriminator module is the matching result of the two input face images, and is used as an adversarial loss function to back-conduct to update the network parameters and optimize the network; the identity retaining module (IRM) includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence. The input of the identity preservation module is the visible light or infrared face training image and the output of the Vis-to-IR or IR-to-Vis generator module; the output of the identity preservation module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity preservation loss function, which is reversely transmitted to update the network parameters and optimize the network.

[0120] In S102 provided in the embodiment of the present invention, a cross-spectral bidirectional image conversion network model is trained using a visible light and infrared face preprocessing training image set to obtain a trained cross-spectral bidirectional image conversion network model. The specific process is as follows:

[0121] Construct a composite loss function based on adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss;

[0122] The cross-spectral bidirectional image translation network model is trained using visible light and infrared face preprocessing training image sets and a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss to obtain a trained cross-spectral bidirectional image translation network model.

[0123] The specific process of S102 provided in the embodiment of the present invention is as follows:

[0124] Construct a cross-spectral bidirectional image conversion network model, which is a highly robust network model specifically designed for cross-spectral bidirectional image conversion;

[0125] Specifically: a cross-spectral bidirectional image conversion network model between visible light images and infrared images is constructed, including the Vis-to-IR generator module G, the IR-to-Vis generator module F, and the visible light discriminator module D VIS 、Infrared discriminator module D IR and an identity maintenance module IRM, wherein the input of the Vis-to-IR generator module G is connected to the visible light face training image, and the output of the Vis-to-IR generator module G is connected to the input of the IR-to-Vis generator module F; the input of the IR-to-Vis generator module F is connected to the infrared face training image, and the output of the IR-to-Vis generator module F is connected to the input of the Vis-to-IR generator module G; a visible light discriminator module D VIS The input is the visible light face training image and the output of the IR-to-Vis generator module F, and the visible light discriminator module D VIS The output of is the matching result of the two input face images, and is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; infrared discriminator module D IR The input is the infrared face training image and the output of the Vis-to-IR generator module F, the infrared discriminator module D IR The output of is the matching result of the two input face images, which is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; the identity preservation module IRM includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence. The input of the identity preservation module is the output of the visible light or infrared face training image and the Vis-to-IR or IR-to-Vis generator module; the output of the identity preservation module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity preservation loss function, which is conducted back to update the network parameters and optimize the network.

[0126] The generator module uses an autoencoder network, which consists of 1 convolution layer with a 7*7 convolution kernel, 2 convolution layers with a 3*3 convolution kernel, 9 residual blocks, 2 deconvolution layers with a 3*3 convolution kernel and 1 convolution layer with a 7*7 convolution kernel. The output of each convolution layer will undergo instance regularization, of which the first half is the encoding process and the second half is the decoding process. The network in the discriminator module consists of 9 convolution layers, 5 maximum pooling layers, 2 fully connected layers and 1 binary classification layer. The identity preservation module network includes the ResNet feature extraction module, the feature normalization module and the feature space mapping module in sequence. After feature extraction and normalization of the two input face images, the Euclidean distance is calculated in the Euclidean space and used as the identity preservation loss function. The network parameters are updated and the network is optimized through reverse conduction.

[0127] The cross-spectral bidirectional image conversion network model is trained using visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model.

[0128] During the model training process, a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is constructed. Specifically, the sum of adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is used as the composite loss function; the specific expression of the composite loss function is as follows:

[0129] Loss = L GAN +λ cyc L cyc +λ syn L syn +λ pc L pc +λ IRM L IRM (4)

[0130] Among them, L GAN , L cyc , L syn , L pc , L IRM They represent adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss respectively; λ cyc For L cyc Corresponding coefficient, λ syn For L syn Corresponding coefficient, λ pc For L pc Corresponding coefficient, λ IRM For L IRM Corresponding coefficients; the above coefficients are used to adjust the proportion of each loss in the composite loss function; for the specific experimental data set, their values ​​are set to λcyc =10,λ syn =30,λ pc =10,λ IRM =10;

[0131] Specific L GAN , L cyc , L syn , L pc and L IRM The designs are shown below:

[0132]

[0133] L cyc (G,F)=E v～Pdata(V) [||F(G(v))-v||1]+E i～Pdata(I) [||G(F(i))-i||1]…… (6)

[0134] L syn (G,F)=E v～P(V),i～P(I) [||F(i)-F(G(v))||1]+E v～P(V),i～P(I) [||G(v)-G(F(i))||1]…… (7)

[0135] L pc (G,F)=E v～P(V),i～P(I) [||vF(i)||1]+E v～P(V),i～P(I) [||iG(v)||1]… (8)

[0136]

[0137] Among them, V and I represent the visible light input image set and infrared input image set corresponding to the cross-spectral bidirectional image conversion network model, respectively, and v and i represent single samples of visible light and infrared input images, respectively. GAN (G,D IR ,V,I) is composed of D IR The binary cross entropy loss value is calculated by calculating the discrimination result of i and the discrimination result of v after G, L GAN (F,D VIS ,I,V) by D VIS The binary cross entropy loss value is calculated by calculating the discrimination result of v and the discrimination result of i after F generation;

[0138] Where L cyc It is used to ensure that the cyclically synthesized image is as close to the original image as possible, so as to train a high-precision generator. It is composed of the L1 loss value between the image generated by v after G and then the F cycle and v, and the L1 loss value between the image generated by i after F and then the G cycle and i;

[0139] Where L syn It is used to ensure that the generated image is as close as possible to the cyclically synthesized image. It is composed of the L1 loss value between the image generated by i after F and the image generated by v after G and then cyclically generated by F, and the L1 loss value between the image generated by v after G and the image generated by i after F and then cyclically generated by G;

[0140] Where L pc It is used to ensure the accuracy of the low-frequency components of the image in the cross-spectral bidirectional image conversion network. It is composed of the L1 loss value of the image generated by v and i through F, and the L1 loss value of the image generated by i and v through G;

[0141] Where L IRM To ensure the accuracy and consistency of identity information during the bidirectional image conversion process, the original image and the generated cross-spectral image are simultaneously input into the face feature extraction network to obtain the corresponding feature vectors with a dimension of 128. and Then calculate the Euclidean distance between these two features as their loss value;

[0142] Furthermore, the cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image set to obtain a trained cross-spectral bidirectional image conversion network model.

[0143] The composite loss function of the cross-spectral bidirectional image conversion network model constructed by the above formula (4) is based on the composite loss function, and the face training images in the visible light face preprocessing training image set and the infrared face preprocessing training image set are input into the cross-spectral bidirectional image conversion network model for training. During the training process, the back propagation algorithm can be used to train the cross-spectral bidirectional image conversion network model, so as to obtain the cross-spectral bidirectional image conversion network model finally trained in this embodiment, which is used to subsequently obtain the generated visible light image and the generated infrared image.

[0144] In S104 provided in the embodiment of the present invention, face matching is performed on the visible light face preprocessing test image and the generated visible light image, and on the infrared face preprocessing test image and the generated infrared image, respectively, to obtain a visible light face matching score and an infrared face matching score, respectively, including:

[0145] Use a standard single-spectrum face feature extraction model such as FaceNet to extract and normalize the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, to obtain a visible light face preprocessing test image feature vector and a generated visible light image feature vector, as well as an infrared face preprocessing test image feature vector and a generated infrared image feature vector;

[0146] By calculating the Euclidean distance of the feature vector, the feature vector of the visible light face preprocessing test image and the generated visible light image feature vector, as well as the feature vector of the infrared face preprocessing test image and the generated infrared image feature vector are calculated to obtain a visible light face matching score matrix and an infrared face matching score matrix respectively;

[0147] According to the visible light face matching score matrix and the infrared face matching score matrix, the false acceptance rate (FAR), true acceptance rate (GAR), equal error rate (EER) and normalized distance (d-prime) of the visible light face matching score matrix and the infrared face matching score matrix are calculated respectively.

[0148] In S105 provided in the embodiment of the present invention, the visible light face matching score and the infrared face matching score are fused using a decision fusion algorithm, and the fused matching score is used for face recognition to achieve cross-spectrum face recognition. The specific process is as follows:

[0149] Construct a decision fusion algorithm based on GAR and d-prime, and perform decision fusion on the visible light face matching score matrix and the infrared face matching score matrix according to the decision fusion algorithm based on GAR and d-prime;

[0150] The fused matching score is used for face recognition. The FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to perform face recognition to achieve cross-spectral face recognition.

[0151] The specific process is: normalizing the visible light face matching score and the infrared face matching score, fusing the matching scores using a decision fusion algorithm, and performing face recognition using the fused matching scores to achieve cross-spectrum face recognition;

[0152] A decision fusion algorithm based on GAR and d-prime is constructed to fuse the matching scores and calculate the fused matching score MatchScore. The specific formula is designed as follows:

[0153] When d-primeV >d-prime I hour:

[0154]

[0155] When d-prime V <d-prime I hour:

[0156]

[0157] When d-prime V =d-prime I hour:

[0158] The final fusion score is:

[0159] MatchScore=w1×MatchScore V +w2×MatchScore I (13)

[0160] Among them, G V and d-prime V are the GAR value and d-prime value of the visible light face matching score, G I and d-prime I are the GAR value and d-prime value of infrared face matching score, MatchScore V and MatchScore I are the visible light matching score and the infrared matching score respectively, and w1 and w2 are the weights calculated by the fusion formula during the fusion process. The advantage of adopting the above fusion strategy is that the fusion weight can be adaptively calculated according to the GAR value of the two matching scores, and the fusion weight of the matching matrix with a small d-prime value can be guaranteed to be greater than 1 / 2, so as to achieve the purpose of improving the fusion effect and matching degree.

[0161] Furthermore, the decision fusion algorithm based on GAR and d-prime fuses the visible light face matching score matrix with the infrared face matching score matrix according to formula (14); the fused matching score MatchScore is used for face recognition, and the FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to achieve cross-spectral face recognition.

[0162] In order to verify the superiority of the cross-spectral face recognition method based on image conversion proposed in the present invention, the present invention provides samples of image conversion results on multiple public face image sets.

[0163] Table 1 Specific network parameters of the generator in the cross-spectral image bidirectional conversion network model of this embodiment

[0164]

[0165]

[0166] The parameter design of each layer in the generator of the cross-spectral bidirectional image conversion network model in the verification process of this embodiment is specifically referred to Table 1. In the convolution process, in order to reduce the artifacts of the input image and the generated image, the reflection padding method (reflection padding) is used to fill 3 in layers 1 and 15, and the padding method in other layers is 1. Among them, the parameters of each layer in the generator of the cross-spectral bidirectional image conversion network model are designed according to the actual situation. This embodiment uses the specific parameter design in Table 1 for identification verification.

[0167] For the parameter design of each layer of the discriminator in the cross-spectral image bidirectional conversion network model during the verification process of this embodiment, please refer to Table 2.

[0168] Table 2 Specific network parameters of the discriminator in the cross-spectral image bidirectional conversion network model of this embodiment

[0169]

[0170]

[0171] The schematic diagram of the identity preservation module structure in the cross-spectral image bidirectional conversion network model in the verification process of this embodiment is specifically referred to as Figure 4 The identity preservation module IRM includes a ResNet feature extraction module, a feature normalization module L2, and a feature space mapping module ED in sequence. The input image Batch of the identity preservation module IRM is first cropped to a fixed size; the ResNet feature extraction module, the feature normalization module L2, and the feature space mapping module ED can be implemented by existing common methods, and the specific implementation method is not limited.

[0172] In order to illustrate that the cross-spectral bidirectional image conversion network model of the present invention has better cross-spectral conversion effect than other mainstream image generation methods, it is compared with other existing public methods such as the image generation method based on CF shared subspace, the image generation method based on Pix2pix and the image generation method based on CycleGAN, and the true acceptance rate (GAR), equal error rate (EER), normalized spacing (d-prime) and Fréchet Inception distance (FID) of the bidirectional generated image and the corresponding band original image are calculated respectively. The larger the true acceptance rate (GAR) and the normalized spacing (d-prime), the better the image generation quality, and the smaller the equal error rate (EER), the better the image generation quality. The Fréchet Inception distance (FID) represents the distance between the feature vector of the generated image and the feature vector of the real image. The closer the distance is, the better the effect of the generation model is, that is, the image has high clarity and rich diversity.

[0173] Table 3 Comparison results of matching scores of different image conversion methods provided in this embodiment (CASIA dataset)

[0174]

[0175]

[0176] The comparison results in Table 3 show that the cross-spectral bidirectional image conversion network model of the present invention achieves the maximum true acceptance rate (GAR) and normalized spacing (d-prime), minimum equal error rate (EER) and Fréchet Inception distance in the visible light band of the CASIA dataset; similarly, the cross-spectral bidirectional image conversion network model of the present invention also has the maximum true acceptance rate (GAR) and normalized spacing (d-prime), minimum equal error rate (EER) and suboptimal Fréchet Inception distance (FID) in the near-infrared band of the CASIA dataset.

[0177] Table 4 Comparison results of matching scores of different image conversion methods provided in this embodiment (TINDERS dataset)

[0178]

[0179] The results in Table 4 show that the cross-spectral bidirectional image conversion network model of the present invention achieves the maximum true acceptance rate (GAR) and the second-best normalized spacing (d-prime), as well as the minimum Fréchet Inception distance (FID) and the second-best equal error rate (EER) in the visible light band of the TINDERS dataset; and the maximum true acceptance rate (GAR) and normalized spacing (d-prime), the minimum Fréchet Inception distance (FID) and the second-best equal error rate (EER) are obtained in the infrared band.

[0180] Judging from the experimental results of the overall image bidirectional conversion, most of the indicators of the bidirectional face generation performance of the cross-spectral bidirectional image conversion network of the present invention are optimal results, and a few are suboptimal results, which shows that the performance of the method of the present invention is better than other mainstream image generation methods as a whole.

[0181] Design decision fusion algorithms to compare matching scores. See Table 5, which shows the matching score fusion using the maximum fusion algorithm, arithmetic mean fusion algorithm, and decision fusion algorithms based on GAR and d-prime, and obtain the corresponding true acceptance rate (GAR), equal error rate (EER), and normalized spacing (d-prime) comparison results.

[0182] From the comparison results of Table 5 and Table 6, it can be seen that after the matching score fusion is performed using the decision fusion algorithm based on GAR and d-prime of the present invention, its performance evaluation indicators such as GAR, EER and d-prime all significantly exceed the results before fusion (see Table 3 and Table 4); at the same time, the performance evaluation indicators after the decision fusion of the present invention are also higher than traditional fusion methods such as the maximum fusion algorithm, the geometric mean fusion algorithm and the arithmetic fusion algorithm. The decision fusion algorithm based on GAR and d-prime takes into account the true acceptance rate (GAR) and the normalized spacing (d-prime) of the matching score under the converted single spectrum, so that the fused matching score not only plays the role of average error, but also ensures the distance between the intra-class and inter-class features, so that the final decision result is the global optimal decision.

[0183] Table 5 Comparison of matching scores of different decision fusion algorithms (CASIA dataset)

[0184]

[0185] Table 6 Comparison of matching scores of different decision fusion algorithms (TINDERS dataset)

[0186]

[0187] It can be seen that this embodiment introduces the infrared band for bidirectional image conversion to address the limitation of traditional face recognition methods that only use visible light, effectively overcoming the influence of uncertain changes in ambient light, making the technology applicable to a variety of environments such as daytime, nighttime, sunny days, rain and snow; this embodiment designs a cross-spectral bidirectional image conversion network model for the problem of cross-spectral face recognition, and performs bidirectional image conversion on visible light and infrared face images to obtain the corresponding generated face images; this embodiment specially designs a composite loss function for the problem of cross-spectral face recognition, which is specifically composed of adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss. The loss function is composed of five parts. The selection of this composite loss function not only ensures the authenticity of the generated image quality, but also retains the identity information of the face, so as to transform the cross-spectrum problem into a single-spectrum problem, which is beneficial to the improvement of face recognition accuracy. This embodiment designs a decision fusion algorithm based on GAR and d-prime for the visible light matching scores and infrared matching scores generated under a single spectrum. The matching scores under a single spectrum are fused using the decision fusion algorithm. The fused matching scores not only play the role of averaging errors, but also ensure the feature distances within and between classes, so that the final decision result is the global optimal decision, thereby improving the final face recognition performance.

[0188] This embodiment proposes a complete set of cross-spectral face recognition technology based on image conversion, which can solve many defects of traditional face recognition technology, such as narrow application scope, low recognition performance, and poor robustness of feature extraction; this embodiment provides new theory and new algorithm support for the practical application of face recognition, making face recognition technology more practical, reliable and popular; this embodiment can be widely used in attendance, civil monitoring, public security law enforcement, access control, community entrance and other application scenarios outdoors, at night, in rain and snow and other complex environments.

[0189] Embodiment 2

[0190] Based on the above embodiment 1, this embodiment provides a cross-spectrum face recognition monitoring device based on image conversion. Figure 5 The schematic diagram of the structure of the monitoring device is shown in FIG. The monitoring device includes a visible light camera and an infrared camera, a video recorder, a display, a router, a read-only memory, an external hard disk and a power supply;

[0191] The video recorder is used to control the information interaction of the entire set of monitoring equipment, the visible light camera and the infrared camera are used to collect visible light and infrared monitoring images respectively, the read-only memory (ROM) is used to store the computer program corresponding to the cross-spectral face recognition method of the present invention, the display is used to display the collected monitoring images in real time, the router is used to connect the monitoring equipment with other external equipment, the external hard disk is used to store the monitoring image data collected by the camera, and the power supply is used to power the entire monitoring equipment.

[0192] When the cross-spectrum face recognition monitoring device is running, the computer program code in the read-only memory is read by the video recorder processor (CPU) and the program is executed. When the computer program is executed, the following steps are implemented:

[0193] Step 1: Control the visible light and infrared cameras to collect face images, and obtain a visible light face image set and an infrared face image set.

[0194] Step 2: preprocess the visible light face image set and the infrared face image set to obtain a visible light face preprocessed image set and an infrared face preprocessed image set.

[0195] Specifically, step 2 of the embodiment of the present invention performs grayscale conversion and normalization processing on the visible light face training image set and the visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set; and performs image enhancement and normalization processing on the infrared face training image set and the infrared face test image set to obtain an infrared face preprocessing training image set and an infrared face preprocessing test image set.

[0196] Step 3: Build a cross-spectral bidirectional image conversion network model between visible light images and infrared images, train it, and save the training model parameters.

[0197] Specifically, step 3 of this embodiment includes:

[0198] The constructed cross-spectral bidirectional image conversion network model includes the Vis-to-IR generator module G, the IR-to-Vis generator module F, and the visible light discriminator module D VIS 、Infrared discriminator module D IR, and the identity preservation module IRM. The generator module consists of 1 7*7 convolution, 2 3*3 convolutions, 9 residual blocks, 2 3*3 deconvolutions and 1 7*7 convolution. The output of each convolution layer will undergo instance regularization, of which the first half is the encoding process and the second half is the decoding process; the discriminator module consists of 9 convolution layers, 5 maximum pooling layers, 2 fully connected layers and 1 binary classification layer; the identity preservation module IRM includes the ResNet feature extraction module, feature normalization module, and feature space mapping module in sequence;

[0199] A composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss was constructed as the loss function; the visible light and infrared face preprocessing training image sets were used and the cross-spectral bidirectional image conversion network model was trained using a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss to obtain a trained cross-spectral bidirectional image conversion network model, and the corresponding parameters in each case were saved. There are two network models in total, corresponding to the bidirectional image conversion network parameters of visible light and near-infrared, and the bidirectional image conversion network parameters of visible light and short-wave infrared.

[0200] Step 4: Use the trained network model to generate images.

[0201] Specifically, step 4 of this embodiment inputs the visible light and infrared face preprocessing test images into the trained cross-spectral bidirectional image conversion network model for image bidirectional conversion to obtain a generated visible light image and a generated infrared image.

[0202] Step 5: Perform face matching on the visible light face preprocessing test image and the generated visible light image, as well as on the infrared face preprocessing test image and the generated infrared image, to obtain visible light face matching scores and infrared face matching scores, respectively.

[0203] Specifically, step 5 of this embodiment includes:

[0204] Use a single-spectrum face feature extraction model such as FaceNet to extract and normalize the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, to obtain a visible light face preprocessing test image feature vector and a generated visible light image feature vector, as well as an infrared face preprocessing test image feature vector and a generated infrared image feature vector;

[0205] By calculating the Euclidean distance of the feature vector, the feature vector of the visible light face preprocessing test image and the generated visible light image feature vector, as well as the feature vector of the infrared face preprocessing test image and the generated infrared image feature vector are calculated to obtain a visible light face matching score matrix and an infrared face matching score matrix respectively;

[0206] According to the visible light face matching score matrix and the infrared face matching score matrix, the FAR value, GAR value, EER value and d-prime value of the visible light face matching score matrix and the infrared face matching score matrix are calculated respectively.

[0207] Step 6: Perform decision fusion and calculate matching results to achieve cross-spectral face recognition, and finally output the cross-spectral face recognition results to the display.

[0208] Specifically, step 6 of this embodiment includes:

[0209] Construct a decision fusion algorithm based on GAR and d-prime to perform matching score fusion;

[0210] The visible light face matching score matrix and the infrared face matching score matrix are fused according to the decision fusion algorithm;

[0211] The fused matching score is used for face recognition, and the FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to perform face recognition and obtain a final matching result.

[0212] The cross-spectral face recognition monitoring device based on image conversion provided in this embodiment can execute the above-mentioned cross-spectral face recognition method embodiment. Its implementation principle and technical effects are similar and will not be repeated here.

[0213] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When the use is implemented in whole or in part in the form of a computer program product, the computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL) or wireless (e.g., infrared, wireless, microwave, etc.) mode) to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0214] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A cross-spectral face recognition method based on image conversion, characterized in that: The cross-spectral face recognition method based on image conversion includes: Step 1: Obtain a visible light face image set and an infrared face image set, and divide the visible light face image set and the infrared face image set into a visible light face training and test image set, and an infrared face training and test image set, respectively, and preprocess the visible light face training and test image set, and the infrared face training and test image set, respectively; Step 2: construct a cross-spectral bidirectional image conversion network model between visible light images and infrared images, and use visible light and infrared face preprocessing training image sets to train the cross-spectral bidirectional image conversion network model to obtain a trained cross-spectral bidirectional image conversion network model; Step 3, inputting the visible light and infrared face preprocessing test images into the trained cross-spectral bidirectional image conversion network model for image bidirectional conversion to obtain a generated visible light image and a generated infrared image; Step 4, performing face matching on the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, respectively, to obtain a visible light face matching score and an infrared face matching score; Step 5: The visible light face matching score and the infrared face matching score are fused using a decision fusion algorithm, and the fused matching score is used for face recognition to achieve cross-spectrum face recognition; In the step 2, a cross-spectral bidirectional image conversion network model between visible light images and infrared images is constructed, including a Vis-to-IR generator module, an IR-to-Vis generator module, a visible light discriminator module, an infrared discriminator module and an identity maintenance module IRM; The visible light face training image is used as the input of the Vis-to-IR generator module, and the output of the Vis-to-IR generator module is connected to the input of the IR-to-Vis generator module; the infrared face training image is used as the input of the IR-to-Vis generator module, and the output of the IR-to-Vis generator module is connected to the input of the Vis-to-IR generator module; the input of the visible light discriminator module is the visible light face training image and the output of the IR-to-Vis generator module, and the output of the visible light discriminator module is the matching result of the two input face images, which is used as the adversarial loss function to back-conduct to update the network parameters and optimize the network; the input of the infrared discriminator module is the infrared face training image and the V The output of the is-to-IR generator module and the output of the infrared discriminator module are the matching results of the two input face images, which are used as the adversarial loss function to conduct back to update the network parameters and optimize the network; the identity preservation module includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence; the input of the identity preservation module IRM is the visible light or infrared face training image and the output of the Vis-to-IR or IR-to-Vis generator module; the output of the identity preservation module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity preservation loss function, and is conducted back to update the network parameters and optimize the network; The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model. The specific process is as follows: Construct a composite loss function based on adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss; Use visible light and infrared face preprocessing training image sets and use a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss to train the cross-spectral bidirectional image conversion network model to obtain a trained cross-spectral bidirectional image conversion network model; The specific process of constructing the cross-spectral bidirectional image conversion network model in step 2 is as follows: The cross-spectral bidirectional image conversion network model is a highly robust network model specifically designed for cross-spectral image conversion. It constructs a cross-spectral bidirectional image conversion network model between visible light images and infrared images, including Vis-to-IR generator module G, IR-to-Vis generator module F, and visible light discriminator module D. VIS 、Infrared discriminator module D IR and an identity maintenance module IRM, wherein the input of the Vis-to-IR generator module G is connected to the visible light face training image, and the output of the Vis-to-IR generator module G is connected to the input of the IR-to-Vis generator module F; the input of the IR-to-Vis generator module F is connected to the infrared face training image, and the output of the IR-to-Vis generator module F is connected to the input of the Vis-to-IR generator module G; a visible light discriminator module D VIS The input is the visible light face training image and the output of the IR-to-Vis generator module F, and the visible light discriminator module D VIS The output of is the matching result of the two input face images, and is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; infrared discriminator module D IR The input is the infrared face training image and the output of the Vis-to-IR generator module F, the infrared discriminator module D IR The output of is the matching result of the two input face images, which is used as the adversarial loss function to conduct back to update the network parameters and optimize the network; the identity preservation module IRM includes a ResNet feature extraction module, a feature normalization module, and a feature space mapping module connected in sequence; the input of the identity preservation module IRM is the output of the visible light or infrared face training image and the Vis-to-IR or IR-to-Vis generator module; the output of the identity preservation module IRM is the Euclidean distance calculated in the Euclidean space after feature extraction and normalization of the two input face images, and it is used as the identity preservation loss function, which is conducted back to update the network parameters and optimize the network; The generator module adopts an autoencoder network, which consists of 1 convolution layer with a 7*7 convolution kernel, 2 convolution layers with a 3*3 convolution kernel, 9 residual blocks, 2 deconvolution layers with a 3*3 convolution kernel and 1 convolution layer with a 7*7 convolution kernel. The output of each convolution layer will undergo a regularization operation, of which the first half is the encoding process and the second half is the decoding process; the network in the discriminator module consists of 9 convolution layers, 5 maximum pooling layers, 2 fully connected layers and 1 binary classification layer; the identity preservation module network includes the ResNet feature extraction module, the feature normalization module, and the feature space mapping module in sequence. After feature extraction and normalization of the two input face images, the Euclidean distance is calculated in the Euclidean space and used as the identity preservation loss function. It is reversely transmitted to update the network parameters and optimize the network.

2. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: In the step 1, the visible light face training and test image set and the infrared face training and test image set are preprocessed to improve the subsequent image conversion effect, and a visible light face preprocessed training image set, a visible light face preprocessed test image set, an infrared face preprocessed training image set and an infrared face preprocessed test image set are obtained respectively; The visible light face training image set, the visible light face test image set, the infrared face training image set and the infrared face test image set are preprocessed to obtain the visible light face preprocessing training image set, the visible light face preprocessing test image set, the infrared face preprocessing training image set and the infrared face preprocessing test image set, respectively. Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set; The infrared face training image set and the infrared face test image set are respectively subjected to image enhancement and normalization processing to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set.

3. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: The specific process of step 1 is: Before performing the bidirectional image conversion, normalizing and contrast adjusting the face images in the visible light face training image set, the visible light face test image set, the infrared face training image set, and the infrared face test image set; Performing grayscale conversion and normalization processing on a visible light face training image set and a visible light face test image set to obtain a visible light face preprocessing training image set and a visible light face preprocessing test image set; The face images in the visible light face image set are first converted into grayscale images. The grayscale conversion formula is designed as follows: I gray =0.2989×R+0.5870×G+0.1140×B (1) Among them, R, G, B are the RGB values ​​corresponding to the visible light image, I gray The grayscale image is output after the visible light image is grayscale converted; Then, the grayscale image I gray Perform maximum and minimum normalization processing, normalize to [0, 255], and the normalization formula design is as follows: Among them, I n is the grayscale image I gray After normalization, the image output is max and I min They are grayscale images I gray The maximum and minimum grayscale values ​​in ; Each visible light face training image and each visible light face test image in the visible light face training image set and the visible light face test image set are processed by the above formula (1) and formula (2) respectively, thereby obtaining a visible light face preprocessing training image set and a visible light face preprocessing test image set; Performing image enhancement and normalization processing on the infrared face training image set and the infrared face test image set respectively to obtain the infrared face preprocessing training image set and the infrared face preprocessing test image set; The face images in the infrared face training image set and the infrared face test image set are first enhanced using the log operator. The image enhancement formula is designed as follows: I e =log c+1 (1+cX) (3) Among them, I e is the image after image enhancement, X is the image before image enhancement, c is the parameter for adjusting contrast, the default value is c=10, the larger the c is, the more obvious the image contrast enhancement is; Then, the enhanced image I e Perform normalization processing, and the normalization formula for normalization to [0, 255] is as shown in the above formula (2); Each infrared face training image and infrared face test image in the infrared face training image set and the infrared face test image set is processed by the above formula (3) and formula (2) respectively, thereby obtaining the infrared face preprocessing training image set and the infrared face preprocessing test image set.

4. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: The specific training process of the cross-spectral bidirectional image conversion network model in step 2 is: The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model; during the model training process, a composite loss function based on adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is constructed, and the sum of adversarial loss, cycle consistency loss, synthetic consistency loss, pixel consistency loss and identity preservation loss is specifically used as the composite loss function; the specific expression of the composite loss function is as follows: Loss=L GAN +λ cyc L cyc +λ syn L syn +λ pc L pc +λ IRM L IRM (4) Among them, L GAN , L cyc , L syn , L pc , L IRM They represent adversarial loss, cycle consistency loss, composition consistency loss, pixel consistency loss and identity preservation loss respectively; λ cyc For L cyc Corresponding coefficient, λ syn For L syn Corresponding coefficient, λ pc For L pc Corresponding coefficient, λ IRM For L IRM Corresponding coefficients; the above coefficients are used to adjust the proportion of each loss in the composite loss function; Specific L GAN , L cyc , L syn , L pc and L IRM The designs are shown below: L cyc (G,F)=E v～Pdata(V) [||F(G(v))-v||1]+E i～Pdata(I) [||G(F(i))-i||1]…(6) L syn (G,F)=E v～P(V),i～P(I) [||F(i)-F(G(v))||1]+E v～P(V),i～P(I) [||G(v)-G(F(i))||1]……(7) L pc (G,F)=E v～P(V),i～P(I) [||v-F(i)||1]+E v～P(V),i～P(I) [||i-G(v)||1]…(8) Where V and I represent the visible light input image set and infrared input image set corresponding to the cross-spectral bidirectional image conversion network model, respectively, and v and i represent single samples of visible light and infrared input images, respectively; L GAN (G,D IR ,V,I) is composed of D IR The binary cross entropy loss value is calculated by calculating the discrimination result of i and the discrimination result of v after G, L GAN (F,D VIS ,I,V) by D VIS The binary cross entropy loss value is calculated by calculating the discrimination result of v and the discrimination result of i after F generation; Where L cyc It is used to ensure that the cyclically synthesized image is as close to the original image as possible, so as to train a high-precision generator; it is composed of the L1 loss value between v and the image generated by G and then cyclically generated by F, and the L1 loss value between i and the image generated by F and then cyclically generated by G; Where L syn It is used to ensure that the generated image is as close as possible to the cyclically synthesized image. It is composed of the L1 loss value between the image generated by i after F and the image generated by v after G and then cyclically generated by F, and the L1 loss value between the image generated by v after G and the image generated by i after F and then cyclically generated by G; Where L pc It is used to ensure the accuracy of the low-frequency components of the image in the cross-spectral bidirectional image conversion network. It is composed of the L1 loss value of the image generated by v and i through F, and the L1 loss value of the image generated by i and v through G; Where L IRM To ensure the accuracy and consistency of identity information during the bidirectional image conversion process, the original image and the generated cross-spectral image are simultaneously input into the face feature extraction network to obtain the corresponding feature vectors with a dimension of 128. and Then calculate the Euclidean distance between these two features as their loss value; The cross-spectral bidirectional image conversion network model is trained using the visible light and infrared face preprocessing training image sets to obtain a trained cross-spectral bidirectional image conversion network model; the composite loss function of the cross-spectral bidirectional image conversion network model constructed by the above formula (4) is based on the composite loss function, and the face training images in the visible light face preprocessing training image set and the infrared face preprocessing training image set are input into the cross-spectral bidirectional image conversion network model for training. During the training process, the back propagation algorithm is used to train the cross-spectral bidirectional image conversion network model, thereby obtaining the final trained cross-spectral bidirectional image conversion network model, which is used to subsequently obtain the generated visible light image and the generated infrared image.

5. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: In the step 4, face matching is performed on the visible light face preprocessing test image and the generated visible light image, and on the infrared face preprocessing test image and the generated infrared image, respectively, and the visible light face matching score and the infrared face matching score are obtained respectively, including: Using a standard single-spectrum face feature extraction model, feature extraction and normalization are performed on the visible light face preprocessing test image and the generated visible light image, as well as the infrared face preprocessing test image and the generated infrared image, to obtain a visible light face preprocessing test image feature vector and a generated visible light image feature vector, as well as an infrared face preprocessing test image feature vector and a generated infrared image feature vector; By calculating the Euclidean distance of the feature vector, the feature vector of the visible light face preprocessing test image and the generated visible light image feature vector, as well as the feature vector of the infrared face preprocessing test image and the generated infrared image feature vector are calculated to obtain a visible light face matching score matrix and an infrared face matching score matrix respectively; According to the visible light face matching score matrix and the infrared face matching score matrix, the false acceptance rate (FAR), true acceptance rate (GAR), equal error rate (EER) and normalized distance (d-prime) of the visible light face matching score matrix and the infrared face matching score matrix are calculated respectively.

6. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: In step 5, the visible light face matching score and the infrared face matching score are fused using a decision fusion algorithm, and the fused matching score is used for face recognition to achieve cross-spectrum face recognition. The specific process is as follows: Construct a decision fusion algorithm based on GAR and d-prime, and perform decision fusion on the visible light face matching score matrix and the infrared face matching score matrix according to the decision fusion algorithm based on GAR and d-prime; The fused matching score is used for face recognition. The FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to perform face recognition to achieve cross-spectral face recognition.

7. The cross-spectral face recognition method based on image conversion as claimed in claim 1, characterized in that: The specific process of step 5 is: normalizing the visible light face matching score and the infrared face matching score, fusing the matching scores using a decision fusion algorithm, and performing face recognition using the fused matching scores to achieve cross-spectrum face recognition; Construct a decision fusion algorithm based on GAR and d-prime to fuse the matching scores and calculate the fused matching score MatchScore. The specific formula is designed as follows: When d-prime V >d-prime I hour: When d-prime V <d-prime I hour: When d-prime V =d-prime I hour: The final fusion score is: MatchScore=w1×MatchScore V +w2×MatchScore I (13) Among them, G V and d-prime V are the GAR value and d-prime value of the visible light face matching score, G I and d-prime I are the GAR value and d-prime value of infrared face matching score, MatchScore V and MatchScore I are the visible light matching score and the infrared matching score respectively, w1 and w2 are the weights calculated by the fusion formula in the fusion process; the above fusion strategy is adopted to adaptively calculate the fusion weight according to the GAR value of the two matching scores, and ensure that the fusion weight of the matching matrix with a small d-prime value is greater than 1 / 2, so as to achieve the purpose of improving the fusion effect and matching degree; The decision fusion algorithm based on GAR and d-prime fuses the visible light face matching score matrix and the infrared face matching score matrix according to formula (13); the fused matching score MatchScore is used for face recognition, and the FAR value, GAR value, EER value and d-prime value of the fused matching score matrix are calculated to achieve cross-spectral face recognition.

8. A monitoring device for cross-spectral face recognition based on image conversion, characterized in that: The monitoring device includes a visible light camera and an infrared camera, a video recorder, a read-only memory, a display, a router, an external hard disk and a power supply. The read-only memory stores a cross-spectral face recognition computer program. The video recorder is used to execute the computer program and control the information interaction of the entire set of monitoring equipment. When the computer program is executed by the video recorder, the monitoring device executes the steps of the cross-spectral face recognition method based on image conversion as described in any one of claims 1 to 6.

Images