Face Recognition System, Method, Electronic Device and Storage Medium for Multispectral Image Fusion

Abstract

The present invention discloses a face recognition system, method, electronic device and storage medium for multi-spectral image fusion, which relates to the technical field of face recognition. A multi-spectral face image set and an environmental condition set are collected in advance. The clustering algorithm is used for the environmental condition set to divide K types of environmental types. For each environmental type, a single-spectral feature extraction model is constructed and trained based on the multi-spectral face image set. Based on the recognition performance of the single-spectral feature extraction model, a fusion weight calculation strategy is designed for each environmental type. In the application environment, the single-spectral feature extraction model is fine-tuned. When face recognition is performed on the person to be verified, the actual environmental condition set is collected, and the fine-tuned single-spectral feature extraction model is used to perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy; the accuracy and stability of the face recognition system in various complex environments are significantly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of face recognition, and specifically to a face recognition system, method, electronic device and storage medium for multi-spectral image fusion. Background Art

[0002] In the field of face recognition, environmental factors such as light intensity, light angle, weather conditions, and indoor and outdoor scene changes have a crucial impact on the recognition accuracy. Under visible light conditions, strong sunlight may cause overexposure, while low light conditions such as at night or in shadow areas will reduce the clarity of the image, thereby affecting the accuracy of recognition. However, the near-infrared spectrum can still provide good recognition ability under low light, so it is often used to supplement the deficiency of the visible spectrum under night conditions. The mid-infrared and far-infrared spectra can provide better recognition ability in the case of smoke or other occlusions.

[0003] Precisely due to the performance differences of different spectra in different environments, it becomes particularly important to fuse the feature information of multiple spectra. By combining the advantages of each spectrum, the overall recognition performance can be improved under complex environmental conditions. For example, in a low light environment, by enhancing the weight of the infrared spectrum, the deficiency of visible spectrum information can be compensated; under strong light, giving a higher weight to the recognition model using the visible spectrum can better capture the detailed information. Such a multi-spectral fusion strategy can dynamically adjust the contributions of each spectrum, thereby overcoming the recognition bottleneck under a single spectrum condition and achieving higher robustness and accuracy.

[0004] However, existing face recognition technologies usually rely on images of a single spectrum and are prone to failure or poor performance under specific lighting or environmental conditions, which limits their application effects in variable environments.

[0005] Chinese Patent with the authorization announcement number CN102831400B discloses a multi-spectral face recognition method and its system. The multi-spectral imaging system outputs the captured face image data to the face recognition module, and the face recognition module performs recognition according to the information in the standard face database in the data storage module, and then outputs the recognition result; however, this method fails to solve the problem of large differences in recognition accuracy under different environmental conditions.

[0006] Therefore, the present invention proposes a face recognition system, method, electronic device and storage medium for multi-spectral image fusion. Summary of the Invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art. For this reason, the present invention proposes a face recognition system, method, electronic device and storage medium for multi-spectral image fusion, which significantly improves the accuracy and stability of the face recognition system in various complex environments.

[0008] It should be noted that all the multispectral face image data involved in this application are legally collected and used after being authorized by the user's device and with the user's consent.

[0009] To achieve the above object, a face recognition method based on multispectral image fusion is proposed, including the following steps:

[0010] Step 1: Collect a multispectral face image set and an environmental condition set in advance;

[0011] Step 2: Use a clustering algorithm for the environmental condition set to divide it into K environmental clustering clusters, and each environmental clustering cluster corresponds to an environmental type; K is the preset number of environmental clustering clusters;

[0012] Step 3: For each environmental type, construct and train a single-spectral feature extraction model based on the multispectral face image set;

[0013] Step 4: Design a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectral feature extraction model;

[0014] Step 5: In the application environment, collect a reference image of the person to be verified in advance;

[0015] Step 6: Fine-tune the single-spectral feature extraction model based on the reference image;

[0016] Step 7: When the person to be verified undergoes face recognition, collect the actual environmental condition set, use the fine-tuned single-spectral feature extraction model, and perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy.

[0017] The method for collecting the multispectral face image set and the environmental condition set in advance is as follows:

[0018] In the test environment, collect several groups of multispectral images for each experimental personnel. When collecting multispectral images each time, use a multispectral camera to collect single-spectral images of the experimental personnel at each wavelength to form a multispectral image, and collect the corresponding environmental parameter set at that time through the environmental sensor network. All the multispectral images form a multispectral face image set, and all the environmental parameter sets corresponding to the multispectral images form an environmental condition set.

[0019] The steps for using a clustering algorithm for the environmental condition set to divide it into K environmental types include the following:

[0020] Step 21: For the environmental parameter set corresponding to each group of multispectral images, extract all environmental parameters to form an environmental feature vector;

[0021] Step 22: Take each environmental feature vector as an M-dimensional discrete point; where M is the number of environmental parameters, and the coordinate of the M-dimensional discrete point in each dimension corresponds to an environmental parameter in the environmental feature vector.

[0022] Step 23: Randomly select K M-dimensional discrete points as the initial clustering centers, and the remaining M-dimensional discrete points as non-initial clustering centers.

[0023] Step 24: For each non-initial clustering center, calculate the Euclidean distance to each initial clustering center, and divide each M-dimensional discrete point into the clustering cluster where the nearest initial clustering center is located.

[0024] Step 25: Calculate the average value of the environmental parameters corresponding to each dimension of the M-dimensional discrete points in each clustering cluster, form a new M-dimensional discrete point with the average values of the environmental parameters corresponding to each dimension, and use this new M-dimensional discrete point as the new clustering center of this clustering cluster.

[0025] Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new clustering center, and re-divide each M-dimensional discrete point into the clustering cluster where the nearest new clustering center is located.

[0026] Step 27: Repeat Step 25 - Step 26 until the M-dimensional discrete points in all the divided clustering clusters no longer change, mark the number of each clustering cluster as k, k = 1, 2, 3,..., K; obtain all the M-dimensional discrete points included in each clustering cluster, and form a discrete point set with all the M-dimensional discrete points; the k-th environmental clustering cluster includes the M-dimensional space coordinates of the new clustering center of the k-th clustering cluster and the environmental feature vectors corresponding to each M-dimensional discrete point in the k-th clustering cluster.

[0027] The method for constructing and training a single-spectrum feature extraction model based on the multi-spectrum face image set for each environmental type is as follows:

[0028] For each environmental type:

[0029] Screen out the multi-spectrum images of each experimental personnel in all such environmental types as the control image set.

[0030] Set a unique personnel number for each experimental personnel.

[0031] Set a sample label for each single-spectrum image in the control image set; the sample label is the personnel number of the experimental personnel corresponding to the single-spectrum image.

[0032] For each spectral condition of each wavelength:

[0033] Divide the single-spectrum images corresponding to the spectral condition in the control image set into a training set, a validation set, and a test set.

[0034] Select a convolutional neural network as the single-spectral feature extraction model;

[0035] Construct a single-spectral feature extraction model, whose input layer matches the size and number of channels of the single-spectral image.

[0036] The single-spectral feature extraction model further includes, in sequence, a convolutional layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully-connected layer for classification tasks that connects the output of the convolutional layer to the output layer, and an output layer;

[0037] The output layer uses the Softmax activation function to map the output to the probability distribution of each personnel number;

[0038] Set the loss function of the single-spectral feature extraction model to multi-class cross-entropy;

[0039] Select the Adam optimizer as the optimizer of the single-spectral feature extraction model;

[0040] Iteratively train the single-spectral feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision metrics.

[0041] Based on the recognition performance of the single-spectral feature extraction model, the way to design the fusion weight calculation strategy for each environmental type is as follows:

[0042] For each environmental type:

[0043] Filter out all the multi-spectral images of each experimental personnel in this environmental type as the control image set;

[0044] Set a unique personnel number for each experimental personnel;

[0045] Set sample labels for each single-spectral image in the control image set; the sample label is the personnel number of the experimental personnel corresponding to the single-spectral image;

[0046] For the spectral conditions of each wavelength:

[0047] Divide the single-spectral images corresponding to the spectral conditions in the control image set into a training set, a validation set, and a test set;

[0048] Select a convolutional neural network as the single-spectral feature extraction model;

[0049] Construct a single-spectral feature extraction model, whose input layer matches the size and number of channels of the single-spectral image;

[0050] The single-spectrum feature extraction model further sequentially includes a convolutional layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully-connected layer for classification tasks that connects the output of the convolutional layer to the output layer, and an output layer;

[0051] The output layer uses the Softmax activation function to map the output to the probability distribution of each personnel number;

[0052] Set the loss function of the single-spectrum feature extraction model to multi-class cross-entropy;

[0053] Select the Adam optimizer as the optimizer for the single-spectrum feature extraction model;

[0054] Iteratively train the single-spectrum feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision metrics;

[0055] The method for designing the fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectrum feature extraction model is as follows:

[0056] For each environmental type:

[0057] Read all the single-spectrum images in the test sets of all experimental personnel under the spectral conditions of each wavelength;

[0058] Mark the number of each experimental personnel as i;

[0059] Mark the number of the spectral condition of each wavelength as j;

[0060] For the spectral condition of the j-th wavelength, input the single-spectrum image of the i-th experimental personnel in the corresponding test set into the single-spectrum feature extraction model of this spectral condition to obtain the probability distribution of the personnel numbers output by the single-spectrum feature extraction model; and read the probability that the personnel number is i in the output probability distribution as the accuracy of correct recognition;

[0061] For each environmental type, construct an accuracy fitting function F, and use the accuracy of correct recognition of each single-spectrum image in the spectral conditions of each wavelength to perform weight fitting on the accuracy fitting function F; The accuracy fitting function F is used as the fusion weight calculation strategy.

[0062] The method for pre-collecting the reference images of the personnel to be verified in the application environment is as follows:

[0063] Pre-load all the single-spectrum feature extraction models into the face recognition devices in the application environment;

[0064] Before face recognition of the person to be verified, one or several single-spectrum face images of the person to be verified under various spectral conditions are pre-collected as reference images, and a label with a new person number is set for each reference image.

[0065] The method for fine-tuning the single-spectrum feature extraction model based on the reference image is as follows:

[0066] Perform data augmentation on the reference image of the person to be verified, and merge the data-augmented reference image with the original multi-spectrum face image set to form a new multi-spectrum face image set;

[0067] Use the new multi-spectrum face image set to retrain each single-spectrum feature extraction model.

[0068] The method for performing face recognition using the fine-tuned single-spectrum feature extraction model according to the actual environmental condition set and the fusion weight calculation strategy is as follows:

[0069] Convert the actual environmental condition set into a new environmental feature vector to obtain a new M-dimensional discrete point;

[0070] Calculate the Euclidean distance between the new M-dimensional discrete point and the center points of the K environmental clustering clusters, and use the environmental type corresponding to the environmental clustering cluster with the closest Euclidean distance as the actual environmental type when the person to be verified undergoes face recognition;

[0071] Read all the single-spectrum feature extraction models corresponding to the actual environmental type, and collect the target single-spectrum images at each wavelength collected by the face recognition device during face recognition;

[0072] Input each target single-spectrum image into the single-spectrum feature extraction model of the spectral condition corresponding to the wavelength to obtain the actual probability distribution output by each single-spectrum feature extraction model;

[0073] Obtain the recognition accuracy of the new person number from the actual probability distribution;

[0074] Input the recognition accuracies of each single-spectrum feature extraction model into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F;

[0075] If the comprehensive accuracy is greater than the preset accuracy coefficient, it is determined as successful recognition; otherwise, it is determined as failed recognition.

[0076] A face recognition system based on multi-spectrum image fusion is proposed, including a sample collection module, an environmental clustering module, an identification model training module, a model fusion module, and a face recognition module; among them, each module is connected electrically.

[0077] The sample collection module pre-collects a multi-spectral face image set and an environmental condition set, and sends the multi-spectral face image set to the recognition model training module and the environmental condition set to the environmental clustering module.

[0078] The environmental clustering module uses a clustering algorithm on the environmental condition set to divide K environmental clustering clusters, where each environmental clustering cluster corresponds to an environmental type; K is the preset number of environmental clustering clusters, and sends all the environmental clustering clusters to the face recognition module.

[0079] For each environmental type, the recognition model training module constructs and trains a single-spectral feature extraction model based on the multi-spectral face image set, and sends all the single-spectral feature extraction models to the model fusion module.

[0080] Based on the recognition performance of the single-spectral feature extraction models, the model fusion module designs a fusion weight calculation strategy for each environmental type and sends the fusion weight calculation strategy to the face recognition module.

[0081] In the application environment, the face recognition module pre-collects a reference image of the person to be verified, fine-tunes the single-spectral feature extraction model based on the reference image, collects the actual environmental condition set when the person to be verified undergoes face recognition, and uses the fine-tuned single-spectral feature extraction model to perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy.

[0082] An electronic device is proposed, including: a processor and a memory, wherein a computer program that can be called by the processor is stored in the memory;

[0083] The processor executes the above-mentioned face recognition method for multi-spectral image fusion by calling the computer program stored in the memory.

[0084] A computer-readable storage medium is proposed, on which a rewritable computer program is stored;

[0085] When the computer program runs on a computer device, the computer device is enabled to execute the above-mentioned face recognition method for multi-spectral image fusion.

[0086] Compared with the prior art, the beneficial effects of the present invention are:

[0087] The present invention pre-collects a multi-spectral face image set and an environmental condition set, uses a clustering algorithm for the environmental condition set to divide into K environmental clustering clusters, each environmental clustering cluster corresponding to an environmental type. For each environmental type, based on the multi-spectral face image set, a single-spectral feature extraction model is constructed and trained. Based on the recognition performance of the single-spectral feature extraction model, a fusion weight calculation strategy is designed for each environmental type. In the application environment, a reference image of the person to be verified is pre-collected, and based on the reference image, the single-spectral feature extraction model is fine-tuned. When the person to be verified undergoes face recognition, the actual environmental condition set is collected, and the fine-tuned single-spectral feature extraction model is used to perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy; First, for different environmental conditions, the environment is divided into different types by using a clustering algorithm. Under each environmental type, a single-spectral feature extraction model is trained separately to extract the corresponding spectral features. Then, by designing a weight strategy, the fusion ratio of different spectral features is adjusted according to the real-time environmental conditions, improving the recognition accuracy and the adaptability of the system, and significantly improving the accuracy and stability of the face recognition system in various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 FIG. is a flowchart of the face recognition method for multi-spectral image fusion in Embodiment 1 of the present invention;

[0089] Figure 2 FIG. is a module connection diagram of the face recognition system for multi-spectral image fusion in Embodiment 2 of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0090] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0091] Embodiment 1

[0092] As Figure 1 shown, the face recognition method for multi-spectral image fusion includes the following steps:

[0093] Step 1: Pre-collect a multi-spectral face image set and an environmental condition set;

[0094] Step 2: Use a clustering algorithm for the environmental condition set to divide into K environmental clustering clusters, each environmental clustering cluster corresponding to an environmental type; K is the preset number of environmental clustering clusters;

[0095] Step 3: For each environmental type, construct and train a single-spectrum feature extraction model based on the multi-spectral face image set;

[0096] Step 4: Design a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectrum feature extraction model;

[0097] Step 5: In the application environment, pre-collect the reference images of the person to be verified;

[0098] Step 6: Fine-tune the single-spectrum feature extraction model based on the reference images;

[0099] Step 7: When the person to be verified undergoes face recognition, collect the set of actual environmental conditions, use the fine-tuned single-spectrum feature extraction model, and perform face recognition according to the set of actual environmental conditions and the fusion weight calculation strategy.

[0100] Among them, the method for pre-collecting the multi-spectral face image set and the environmental condition set is as follows:

[0101] In the test environment, collect several groups of multi-spectral images for each experimental person. When collecting multi-spectral images each time, use a multi-spectral camera to collect the single-spectrum images of the experimental person at each wavelength to form a multi-spectral image, and collect the corresponding set of environmental parameters at that time through the environmental sensor network. All the multi-spectral images form a multi-spectral face image set, and the set of environmental parameters corresponding to all the multi-spectral images forms an environmental condition set.

[0102] Preferably, the test environment is:

[0103] In the laboratory, by setting control conditions, simulate various lighting, background, and reflection environments to systematically and controllably collect test sample data;

[0104] Or:

[0105] Collect data under various actual conditions such as public places and office environments, covering different weather, times, and degrees of crowding to fully restore the face recognition requirements in the real scenario.

[0106] Furthermore, the multi-spectral image includes spectral images under spectral conditions of different wavelengths. The spectral conditions include but are not limited to: visible light wavelength, near-infrared wavelength, mid-infrared wavelength, and far-infrared wavelength; among them, the visible light image refers to a standard RGB image, usually in the wavelength range of 400 - 700 nanometers; the wavelength range of the near-infrared image is usually 700 - 900 nanometers, providing good face features under low-light or nighttime conditions; mid-infrared and far-infrared images are used to provide different face features under extreme lighting or temperature conditions.

[0107] The multi-spectral camera is a camera that can capture images of different spectra simultaneously or separately, and can capture images of different spectra at the same time point to achieve better feature alignment.

[0108] Furthermore, the environmental sensor network is composed of sensors corresponding to several environmental parameters deployed in the test environment, and the environmental parameters include but are not limited to lighting conditions, weather conditions and time information. The lighting conditions include but are not limited to light intensity and source direction (such as natural light, indoor lights, etc.), and the weather conditions include but are not limited to sunny days, cloudy days, rainy days, etc.; it can be understood that under different lighting conditions and weather conditions, the face recognition accuracy of different spectra is different. For example, in low-light or no-light environment, infrared spectrum can be used to capture facial information, while under high-light conditions, visible light can provide detailed features; therefore, under different environmental conditions, different spectral recognition models can be used in combination to improve the recognition accuracy.

[0109] Furthermore, the step of using a clustering algorithm to classify the set of environmental conditions into K environmental types includes the following steps:

[0110] Step 21: for each set of environmental parameters corresponding to the multispectral images, extract all environmental parameters to form an environmental feature vector;

[0111] Step 22: Treat each environmental feature vector as an M-dimensional discrete point; wherein M is the number of environmental parameters, and the coordinates of the M-dimensional discrete point in each dimension correspond to an environmental parameter in the environmental feature vector; it can be understood that one M-dimensional discrete point corresponds to one M-dimensional spatial coordinate;

[0112] Step 23: Randomly select K M-dimensional discrete points as initial cluster centers, and the remaining M-dimensional discrete points as non-initial cluster centers;

[0113] Step 24: For each non-initial cluster center, calculate the Euclidean distance to each initial cluster center, and divide each M-dimensional discrete point into the cluster cluster where the nearest initial cluster center is located;

[0114] Step 25: Calculate the average value of the environmental parameter corresponding to each dimension of the M-dimensional discrete points in each cluster, form a new M-dimensional discrete point with the average value of the environmental parameter corresponding to each dimension, and use the new M-dimensional discrete point as the new cluster center of the cluster;

[0115] Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new cluster center, and re-divide each M-dimensional discrete point into the cluster where the nearest new cluster center is located;

[0116] Step 27: Repeat Step 25 - Step 26 until the M - dimensional discrete points in all the divided clustering clusters no longer change. Mark the number of each clustering cluster as k, where k = 1, 2, 3,..., K; obtain all the M - dimensional discrete points included in each clustering cluster, and form a discrete point set with all these M - dimensional discrete points; the k - th environmental clustering cluster includes the M - dimensional spatial coordinates of the new clustering center of the k - th clustering cluster and the environmental feature vectors corresponding to each M - dimensional discrete point in the k - th clustering cluster.

[0117] It can be understood that there are some environmental similarities to a certain extent among each group of environmental condition sets in each environmental clustering cluster, that is, corresponding to one environmental type, and the face photos taken in this environmental type also have a certain degree of similarity in light and shadow. Therefore, the same recognition model can be used for face recognition.

[0118] Furthermore, for each environmental type, the method of constructing and training a single - spectrum feature extraction model based on the multi - spectrum face image set is as follows:

[0119] For each environmental type:

[0120] Screen out the multi - spectrum images of each experimental personnel in all such environmental types as the control image set;

[0121] Set a unique personnel number for each experimental personnel;

[0122] Set a sample label for each single - spectrum image in the control image set; the sample label is the personnel number of the experimental personnel corresponding to the single - spectrum image;

[0123] For each spectral condition of each wavelength:

[0124] Divide the single - spectrum images corresponding to the spectral condition in the control image set into a training set, a validation set, and a test set; generally, the division ratio of the training set, the validation set, and the test set is: 70%, 15%, and 15%;

[0125] Select a convolutional neural network as the single - spectrum feature extraction model; preferably, the convolutional neural network includes but is not limited to ResNet, VGG, MobileNet, etc.;

[0126] Construct a single - spectrum feature extraction model, whose input layer matches the size and number of channels of the single - spectrum image.

[0127] The single - spectrum feature extraction model further includes a convolutional layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully - connected layer for classification tasks, connecting the output of the convolutional layer to the output layer, and an output layer in sequence.

[0128] The output layer uses the Softmax activation function to map the output to the probability distribution of each person's number;

[0129] Set the loss function of the single-spectral feature extraction model to multi-class cross-entropy;

[0130] Select the Adam optimizer as the optimizer of the single-spectral feature extraction model;

[0131] Iteratively train the single-spectral feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision metrics.

[0132] It can be understood that through the above training process of the single-spectral feature extraction model, a face recognition model is trained for each spectral condition of each wavelength in each environment type. For example, for outdoor high-light conditions, face recognition models are trained for visible light wavelength, near-infrared wavelength, mid-infrared wavelength, and far-infrared wavelength respectively. Obviously, the recognition accuracies of each face recognition model are different. Therefore, it is necessary to fuse and calibrate the recognition accuracies.

[0133] Therefore, the method for designing the fusion weight calculation strategy for each environment type based on the recognition performance of the single-spectral feature extraction model is as follows:

[0134] For each environment type:

[0135] Read all single-spectral images in the test sets of all experimental personnel under the spectral conditions of each wavelength;

[0136] Mark the number of each experimental personnel as i;

[0137] Mark the number of the spectral condition of each wavelength as j;

[0138] For the spectral condition of the j-th wavelength, input the single-spectral image of the i-th experimental personnel in the corresponding test set into the single-spectral feature extraction model of this spectral condition to obtain the probability distribution of the personnel numbers output by the single-spectral feature extraction model; and read the probability pij of the personnel number being i in the output probability distribution; it can be understood that the probability pij is the recognition accuracy of the single-spectral feature extraction model correctly recognized in the spectral condition of the j-th wavelength;

[0139] For each environment type, construct an accuracy fitting function F, and use the recognition accuracy of each single-spectral image in the spectral condition of each wavelength to perform weight fitting on the accuracy fitting function F; the accuracy fitting function F is used as the fusion weight calculation strategy.

[0140] Specifically, the accuracy fitting function F can be set as:

[0141] ; where Pj is the accuracy rate of correct recognition by the single-spectrum feature extraction model in the spectral condition of the j-th wavelength, and wj is the accuracy weight of the spectral condition of the j-th wavelength. Thus, the accuracy fitting function F can be understood as the total accuracy rate that fuses the recognition accuracy rates under each wavelength condition.

[0142] Furthermore, the weight fitting of the accuracy fitting function F means using the accuracy rate of correct recognition of each single-spectrum image in the spectral condition of each wavelength to fit and solve each accuracy weight wj in the accuracy fitting function F.

[0143] Specifically, the method for weight fitting of the accuracy fitting function F is as follows:

[0144] Construct an accuracy loss function , the accuracy loss function .

[0145] By using the gradient descent algorithm to iteratively optimize the accuracy loss function , obtain the value of the accuracy weight wj of the spectral condition of each wavelength that minimizes the accuracy loss function .

[0146] It can be understood that by fitting the accuracy fitting function F, the appropriate accuracy weights of the spectral conditions of different wavelengths can be calculated under the corresponding environmental type. Through the weighted calculation of the accuracy weights, the accuracy of face recognition in this environmental type can be improved.

[0147] Furthermore, the method for pre-collecting the reference images of the person to be verified in the application environment is as follows:

[0148] Pre-load all single-spectrum feature extraction models in the face recognition device of the application environment;

[0149] Before the person to be verified undergoes face recognition, pre-collect one or several single-spectrum face images of the person to be verified under each spectral condition as reference images, and set a new person number label for each reference image.

[0150] Furthermore, the method for fine-tuning the single-spectrum feature extraction model based on the reference images is as follows:

[0151] Perform data augmentation on the reference images of the person to be verified, and merge the data-augmented reference images with the original multi-spectrum face image set to form a new multi-spectrum face image set; the data augmentation includes but is not limited to horizontal flipping, slight rotation, scaling, etc. to increase diversity;

[0152] Retrain each single - spectrum feature extraction model using the new multi - spectral face image set.

[0153] Furthermore, when performing face recognition on the person to be verified, the way to collect the set of actual environmental conditions is to collect the set of environmental parameters composed of the actual values of various environmental parameters during face recognition.

[0154] The method of performing face recognition using the fine - tuned single - spectrum feature extraction model according to the set of actual environmental conditions and the fusion weight calculation strategy is as follows:

[0155] Convert the set of actual environmental conditions into a new environmental feature vector to obtain a new set of M - dimensional discrete points;

[0156] Calculate the Euclidean distance between the new set of M - dimensional discrete points and the center points of the K environmental clustering clusters, and use the environmental type corresponding to the environmental clustering cluster with the closest Euclidean distance as the actual environmental type when performing face recognition on the person to be verified;

[0157] Read all the single - spectrum feature extraction models corresponding to the actual environmental type, and collect the target single - spectrum images at each wavelength collected by the face recognition device during face recognition;

[0158] Input each target single - spectrum image into the single - spectrum feature extraction model corresponding to the spectral conditions of the corresponding wavelength to obtain the actual probability distribution output by each single - spectrum feature extraction model;

[0159] Obtain the recognition accuracy of the new person number from the actual probability distribution;

[0160] Input the recognition accuracies of the single - spectrum feature extraction models into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F;

[0161] If the comprehensive accuracy is greater than the preset accuracy coefficient, it is judged as successful recognition, otherwise it is judged as failed recognition.

[0162] Embodiment 2

[0163] As Figure 2 shown, the face recognition system for multi - spectral image fusion includes a sample collection module, an environmental clustering module, a recognition model training module, a model fusion module, and a face recognition module; among them, each module is connected electrically.

[0164] The sample collection module pre - collects a multi - spectral face image set and a set of environmental conditions, and sends the multi - spectral face image set to the recognition model training module, and sends the set of environmental conditions to the environmental clustering module.

[0165] The environmental clustering module uses a clustering algorithm for the set of environmental conditions to divide K environmental clustering clusters, where each environmental clustering cluster corresponds to an environmental type; K is the preset number of environmental clustering clusters, and all environmental clustering clusters are sent to the face recognition module.

[0166] The recognition model training module constructs and trains a single-spectral feature extraction model for each environmental type based on the multi-spectral face image set, and sends all single-spectral feature extraction models to the model fusion module.

[0167] The model fusion module designs a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectral feature extraction model, and sends the fusion weight calculation strategy to the face recognition module.

[0168] The face recognition module pre-collects the reference images of the person to be verified in the application environment, fine-tunes the single-spectral feature extraction model based on the reference images, collects the set of actual environmental conditions when the person to be verified undergoes face recognition, uses the fine-tuned single-spectral feature extraction model, and performs face recognition according to the set of actual environmental conditions and the fusion weight calculation strategy.

[0169] Embodiment 3

[0170] According to another aspect of the present application, an electronic device is also provided. The electronic device may include one or more processors and one or more memories. Among them, computer-readable code is stored in the memory, and when the computer-readable code is run by one or more processors, it can execute the face recognition method for multi-spectral image fusion as described above.

[0171] The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to the network, input / output components, a hard disk, etc. The storage device in the electronic device, such as ROM or the hard disk, can store the face recognition method for multi-spectral image fusion provided by the present application.

[0172] Furthermore, the electronic device may further include a user interface. Of course, this architecture is only exemplary, and when implementing different devices, one or more components in the electronic device can be omitted according to actual needs.

[0173] Embodiment 4

[0174] A computer-readable storage medium according to an embodiment of the present application. Computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are run by a processor, the face recognition method for multi-spectral image fusion according to the embodiment of the present application described with reference to the above drawings can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0175] In addition, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium storing machine-readable instructions that can be run by a processor to execute instructions corresponding to the method steps provided by the present application. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0176] The method, apparatus, and device of the present application can be implemented in many ways. For example, the method, apparatus, and device of the present application can be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present application are not limited to the specific order described above, unless otherwise specifically stated. In addition, in some embodiments, the present application can also be implemented as a program recorded in a recording medium, and these programs include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers a recording medium storing a program for executing the method according to the present application.

[0177] In addition, in the above technical solutions provided by the embodiments of the present application, the parts that are consistent with the corresponding technical solutions in the prior art in terms of implementation principles are not described in detail to avoid excessive elaboration.

[0178] As described above in the specific embodiments, the purpose, technical solutions, and beneficial effects of the present invention are further described in detail. It should be understood that the above are only specific embodiments of the present invention and are not used to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included in the protection scope of the present invention.

[0179] The above preset parameters or preset thresholds are all set by those skilled in the art according to the actual situation or obtained through a large amount of data simulation.

[0180] The above embodiments are only used to illustrate the technical method of the present invention and not to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical method of the present invention can be modified or equivalently replaced without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A face recognition method for multi - spectral image fusion, characterized in that It includes the following steps: Step 1: Pre-collect a multi-spectral face image set and an environmental condition set; Step 2: Use a clustering algorithm for the environmental condition set to divide it into K environmental clustering clusters, and each environmental clustering cluster corresponds to an environmental type; K is the preset number of environmental clustering clusters; Step 3: For each environmental type, construct and train a single-spectral feature extraction model based on the multi-spectral face image set; Step 4: Design a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectral feature extraction model; Step 5: In the application environment, pre-collect the reference images of the person to be verified; Step 6: Fine-tune the single-spectral feature extraction model based on the reference images; Step 7: When the person to be verified undergoes face recognition, collect the actual environmental condition set, use the fine-tuned single-spectral feature extraction model, and perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy; The method of using the fine-tuned single-spectral feature extraction model to perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy is as follows: Convert the actual environmental condition set into a new environmental feature vector, so as to obtain a new M-dimensional discrete point; Calculate the Euclidean distance between the new M-dimensional discrete point and the center points of the K environmental clustering clusters, and use the environmental type corresponding to the environmental clustering cluster with the closest Euclidean distance as the actual environmental type when the person to be verified undergoes face recognition; Read all the single-spectral feature extraction models corresponding to the actual environmental type, and collect the target single-spectral images at each wavelength collected by the face recognition device during face recognition; Input each target single-spectral image into the single-spectral feature extraction model under the spectral conditions corresponding to the wavelength to obtain the actual probability distribution output by each single-spectral feature extraction model; Obtain the recognition accuracy of the new person number from the actual probability distribution; Input the recognition accuracies of the single-spectral feature extraction models into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F; If the comprehensive accuracy is greater than the preset accuracy coefficient, it is judged as a successful recognition, otherwise it is judged as a failed recognition.

2. The face recognition method for multi-spectral image fusion according to claim 1, wherein The method of pre-collecting the multi-spectral face image set and the environmental condition set is as follows: In the test environment, collect several groups of multi-spectral images for each experimental person. When collecting multi-spectral images each time, use a multi-spectral camera to collect the single-spectral images of the experimental person at each wavelength to form a multi-spectral image, and collect the corresponding environmental parameter set at that time through the environmental sensor network. All multi-spectral images form a multi-spectral face image set, and all the environmental parameter sets corresponding to the multi-spectral images form an environmental condition set.

3. The face recognition method for multi-spectral image fusion according to claim 2, characterized in that, The steps of using a clustering algorithm for the environmental condition set to divide it into K environmental types include the following steps: Step 21: For the environmental parameter set corresponding to each group of multi-spectral images, extract all environmental parameters to form an environmental feature vector; Step 22: Take each environmental feature vector as an M-dimensional discrete point; where M is the number of environmental parameters, and the coordinate of the M-dimensional discrete point in each dimension corresponds to an environmental parameter in the environmental feature vector; Step 23: Randomly select K M-dimensional discrete points as the initial clustering centers, and the remaining M-dimensional discrete points as non-initial clustering centers; Step 24: For each non-initial clustering center, calculate the Euclidean distance to each initial clustering center, and divide each M-dimensional discrete point into the clustering cluster where the nearest initial clustering center is located; Step 25: Calculate the average value of the environmental parameters corresponding to each dimension of the M-dimensional discrete points within each clustering cluster, form a new M-dimensional discrete point by combining the average values of the environmental parameters corresponding to each dimension, and use this new M-dimensional discrete point as the new clustering center of this clustering cluster; Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new clustering center, and re-divide each M-dimensional discrete point into the clustering cluster where the nearest new clustering center is located; Step 27: Repeat Step 25 - Step 26 until the M-dimensional discrete points in all the divided clustering clusters no longer change. Mark the number of each clustering cluster as k, where k = 1, 2, 3,..., K; Obtain all the M-dimensional discrete points included in each clustering cluster, and form a discrete point set by combining all the M-dimensional discrete points therein; The k-th environmental clustering cluster includes the M-dimensional spatial coordinates of the new clustering center of the k-th clustering cluster, and the environmental feature vectors corresponding to each M-dimensional discrete point in the k-th clustering cluster.

4. The face recognition method for multi-spectral image fusion according to claim 3, wherein, The method of constructing and training a single-spectral feature extraction model based on a multi-spectral face image set for each environmental type is as follows: For each environmental type: Screen out the multi-spectral images of each experimental personnel in all such environmental types as the control image set; Set a unique personnel number for each experimental personnel; Set a sample label for each single-spectral image in the control image set; The sample label is the personnel number of the experimental personnel corresponding to the single-spectral image; For each spectral condition of each wavelength: Divide the single-spectral images corresponding to the spectral condition in the control image set into a training set, a validation set, and a test set; Select a convolutional neural network as the single-spectral feature extraction model; Construct a single-spectral feature extraction model, the input layer of which matches the size and number of channels of the single-spectral image; The single-spectral feature extraction model further includes a convolutional layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully connected layer for classification tasks, connecting the output of the convolutional layer to the output layer, and an output layer in sequence; The output layer uses the Softmax activation function to map the output to the probability distribution of each personnel number; Set the loss function of the single-spectral feature extraction model as multi-class cross-entropy; Select the Adam optimizer as the optimizer of the single-spectral feature extraction model; Iteratively train the single-spectral feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision metrics.

5. The face recognition method for multi-spectral image fusion according to claim 4, wherein The method of designing a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectral feature extraction model is as follows: For each environmental type: Read all the single-spectral images in the test sets of all experimental personnel under the spectral conditions of each wavelength; Mark the number of each experimental personnel as i; Label the number of the spectral condition for each wavelength as j; For the spectral condition of the j-th wavelength, input the single-spectral image of the i-th experimenter in the corresponding test set into the single-spectral feature extraction model of this spectral condition to obtain the probability distribution of the person numbers output by the single-spectral feature extraction model; and read the probability that the person number is i in the output probability distribution as the recognition accuracy of being correct; For each environmental type, construct an accuracy fitting function F, and use the recognition accuracy of being correct for each single-spectral image in the spectral conditions of each wavelength to perform weight fitting on the accuracy fitting function F; the accuracy fitting function F is used as the fusion weight calculation strategy.

6. The face recognition method for multi-spectral image fusion according to claim 5, characterized in that, The method for pre-collecting the reference images of the person to be verified in the application environment is as follows: Pre-load all single-spectral feature extraction models in the face recognition device in the application environment; Before the person to be verified undergoes face recognition, pre-collect one or several single-spectral face images of the person to be verified under each spectral condition as reference images, and set a label with a new person number for each reference image.

7. The face recognition method for multi-spectral image fusion according to claim 6, characterized in that, The method for fine-tuning the single-spectral feature extraction model based on the reference images is as follows: Perform data augmentation on the reference images of the person to be verified, and merge the data-augmented reference images with the original multi-spectral face image set to form a new multi-spectral face image set; Use the new multi-spectral face image set to retrain each single-spectral feature extraction model.

8. A face recognition system for multi-spectral image fusion, which is used to implement the face recognition method for multi-spectral image fusion described in any one of claims 1-7, characterized in that, It includes a sample collection module, an environment clustering module, an identification model training module, a model fusion module, and a face recognition module; among them, each module is connected electrically; The sample collection module pre-collects a multi-spectral face image set and an environmental condition set, and sends the multi-spectral face image set to the identification model training module and the environmental condition set to the environment clustering module; The environment clustering module uses a clustering algorithm on the environmental condition set to divide it into K environmental clustering clusters, and each environmental clustering cluster corresponds to an environmental type; K is the preset number of environmental clustering clusters, and sends all environmental clustering clusters to the face recognition module; The identification model training module, for each environmental type, constructs and trains a single-spectral feature extraction model based on the multi-spectral face image set, and sends all single-spectral feature extraction models to the model fusion module; The model fusion module designs a fusion weight calculation strategy for each environmental type based on the recognition performance of the single-spectral feature extraction model, and sends the fusion weight calculation strategy to the face recognition module; The face recognition module, in the application environment, pre-collects the reference images of the person to be verified, fine-tunes the single-spectral feature extraction model based on the reference images, collects the actual environmental condition set when the person to be verified undergoes face recognition, and uses the fine-tuned single-spectral feature extraction model to perform face recognition according to the actual environmental condition set and the fusion weight calculation strategy.

9. An electronic device, characterized in that, It includes: A processor and a memory, where: The memory stores a computer program that can be called by the processor; The processor executes the face recognition method for multispectral image fusion according to any one of claims 1-7 in the background by calling the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that, A rewritable computer program is stored thereon; When the computer program runs on the computer device, the computer device executes the face recognition method for multispectral image fusion according to any one of claims 1-7 in the background.

Images