Face recognition system and method based on multispectral image fusion, electronic equipment and storage medium

Abstract

The invention discloses a multi-spectral image fusion face recognition system and method, electronic equipment and a storage medium, and relates to the technical field of face recognition, a multi-spectral face image set and an environmental condition set are collected in advance, a clustering algorithm is used for the environmental condition set, K environmental types are divided, and for each environmental type, K is a positive integer, and K is a positive integer; the method comprises the steps of constructing and training a single-spectrum feature extraction model based on a multi-spectrum face image set, designing a fusion weight calculation strategy for each environment type based on the recognition performance of the single-spectrum feature extraction model, performing fine adjustment on the single-spectrum feature extraction model in an application environment, and when face recognition is performed on a to-be-verified person, performing face recognition on the to-be-verified person. Collecting an actual environment condition set, and performing face recognition by using the fine-tuned single-spectrum feature extraction model according to the actual environment condition set and a fusion weight calculation strategy; and the accuracy and the stability of the face recognition system in various complex environments are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of face recognition technology, and in particular to a face recognition system, method, electronic equipment and storage medium fused with multi-spectral images. Background Art

[0002] In the field of face recognition, environmental factors such as light intensity, light angle, weather conditions, and changes in indoor and outdoor scenes have a crucial impact on recognition accuracy. Under visible light conditions, strong sunlight may cause overexposure, while low light conditions such as night or shadow areas will reduce the clarity of the image, thereby affecting recognition accuracy. However, the near-infrared spectrum can still provide good recognition capabilities under low light conditions, so it is often used to supplement the deficiencies of the visible spectrum under night conditions. The mid-infrared and far-infrared spectra can provide better recognition capabilities in smoke or other obstructions.

[0003] It is precisely because of the differences in the performance of different spectra in different environments that it is particularly important to fuse the characteristic information of multiple spectra. By combining the advantages of each spectrum, the overall recognition performance can be improved under complex environmental conditions. For example, in a low-light environment, by enhancing the weight of the infrared spectrum, the lack of visible spectrum information can be compensated; under strong light, giving a higher weight to the recognition model using the visible spectrum can better capture detailed information. Such a multi-spectral fusion strategy can dynamically adjust the contribution of each spectrum, thereby overcoming the recognition bottleneck under a single spectrum condition and achieving higher robustness and accuracy.

[0004] However, existing face recognition technology usually relies on single-spectrum images and is prone to failure or poor performance under specific lighting or environmental conditions, which limits its application in changing environments.

[0005] A Chinese patent with authorization announcement number CN102831400B discloses a multispectral face recognition method and system thereof. The multispectral imaging system outputs the captured face image data to the face recognition module. The face recognition module performs recognition based on the information of the standard face database in the data storage module, and then outputs the recognition result. However, this method does not solve the problem of large differences in recognition accuracy under different environmental conditions.

[0006] To this end, the present invention proposes a multi-spectral image fusion face recognition system, method, electronic device and storage medium. Summary of the invention

[0007] The present invention aims to solve at least one of the technical problems existing in the prior art. To this end, the present invention proposes a face recognition system, method, electronic device and storage medium based on multispectral image fusion, which significantly improves the accuracy and stability of the face recognition system in various complex environments.

[0008] It should be noted that the multispectral facial image data involved in this application are all authorized by the user device and are legally collected and used with the user's consent.

[0009] To achieve the above purpose, a face recognition method based on multispectral image fusion is proposed, which includes the following steps: Step 1: Collect a set of multispectral face images and a set of environmental conditions in advance; Step 2: Use a clustering algorithm to divide the environmental condition set into K environmental clusters, each of which corresponds to an environmental type; K is the number of preset environmental clusters; Step 3: For each environment type, build and train a single spectrum feature extraction model based on a collection of multispectral face images; Step 4: Based on the recognition performance of the single spectrum feature extraction model, a fusion weight calculation strategy is designed for each environment type; Step 5: In the application environment, pre-collect the reference image of the person to be verified; Step 6: Fine-tune the single spectrum feature extraction model based on the benchmark image; Step 7: When the person to be verified is undergoing face recognition, a set of actual environmental conditions is collected, and the fine-tuned single spectrum feature extraction model is used to perform face recognition based on the set of actual environmental conditions and the fusion weight calculation strategy.

[0010] The method of pre-collecting the multi-spectral face image set and the environmental condition set is: In the test environment, several groups of multispectral images are collected for each experimenter. Each time a multispectral image is collected, a multispectral camera is used to collect the single spectral images of the experimenter at each wavelength to form a multispectral image, and the corresponding environmental parameter set at that time is collected through the environmental sensor network. All multispectral images constitute a multispectral face image set, and the set of environmental parameters corresponding to all multispectral images constitutes an environmental condition set.

[0011] The method of using a clustering algorithm to classify the environmental condition set into K environmental types includes the following steps: Step 21: for each set of environmental parameters corresponding to the multispectral images, extract all environmental parameters to form an environmental feature vector; Step 22: Treat each environmental feature vector as an M-dimensional discrete point, wherein M is the number of environmental parameters, and the coordinates of the M-dimensional discrete point in each dimension correspond to an environmental parameter in the environmental feature vector; Step 23: Randomly select K M-dimensional discrete points as initial cluster centers, and the remaining M-dimensional discrete points as non-initial cluster centers; Step 24: For each non-initial cluster center, calculate the Euclidean distance to each initial cluster center, and divide each M-dimensional discrete point into the cluster cluster where the nearest initial cluster center is located; Step 25: Calculate the average value of the environmental parameter corresponding to each dimension of the M-dimensional discrete points in each cluster, form a new M-dimensional discrete point with the average value of the environmental parameter corresponding to each dimension, and use the new M-dimensional discrete point as the new cluster center of the cluster; Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new cluster center, and re-divide each M-dimensional discrete point into the cluster where the nearest new cluster center is located; Step 27: Repeat steps 25 and 26 until the M-dimensional discrete points in all divided clusters no longer change, and mark the number of each cluster as k, k=1, 2, 3, ..., K; obtain all the M-dimensional discrete points contained in each cluster, and form a discrete point set from all the M-dimensional discrete points; the k-th environmental cluster includes the M-dimensional space coordinates of the new cluster center of the k-th cluster, and the environmental feature vector corresponding to each M-dimensional discrete point in the k-th cluster.

[0012] For each environment type, the method of constructing and training a single spectrum feature extraction model based on a multispectral face image set is as follows: For each environment type: Screen out all multispectral images of each experimenter in this type of environment as a set of control images; Set a unique personnel number for each experimenter; Setting a sample label for each single-spectrum image in the control image set; the sample label is the personnel number of the experimenter corresponding to the single-spectrum image; For each wavelength the spectral condition is: The single spectral images corresponding to the spectral conditions in the control image set are divided into a training set, a validation set and a test set; Select a convolutional neural network as a single spectrum feature extraction model; A single-spectrum feature extraction model is constructed, whose input layer matches the size and number of channels of the single-spectrum image.

[0013] The single spectrum feature extraction model further includes, in sequence, a convolution layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully connected layer for classification tasks, connecting the output of the convolution layer to the output layer, and an output layer; The output layer uses a Softmax activation function to map the output to the probability distribution of each personnel number; The loss function of the single spectrum feature extraction model is set to multi-classification cross entropy; Select Adam optimizer as the optimizer for the single spectrum feature extraction model; Iteratively train the single spectrum feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision indicators.

[0014] The recognition performance based on the single spectrum feature extraction model is used to design a fusion weight calculation strategy for each environment type: For each environment type: Screen out all multispectral images of each experimenter in this type of environment as a set of control images; Set a unique personnel number for each experimenter; Setting a sample label for each single-spectrum image in the control image set; the sample label is the personnel number of the experimenter corresponding to the single-spectrum image; For each wavelength the spectral condition is: The single spectral images corresponding to the spectral conditions in the control image set are divided into a training set, a validation set and a test set; Select a convolutional neural network as a single spectrum feature extraction model; Construct a single spectrum feature extraction model whose input layer matches the size and number of channels of the single spectrum image; The single spectrum feature extraction model further includes, in sequence, a convolution layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully connected layer for classification tasks, connecting the output of the convolution layer to the output layer, and an output layer; The output layer uses a Softmax activation function to map the output to the probability distribution of each personnel number; The loss function of the single spectrum feature extraction model is set to multi-classification cross entropy; Select Adam optimizer as the optimizer for the single spectrum feature extraction model; Iteratively train the single spectrum feature extraction model on the training set, adjust the hyperparameters based on the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision indicators; The recognition performance based on the single spectrum feature extraction model is used to design a fusion weight calculation strategy for each environment type: For each environment type: Read all single-spectral images in the test set for all experimenters under each wavelength spectral condition; Label each experimenter's number as i; The number of the spectral condition at each wavelength is marked as j; For the spectral condition of the jth wavelength, the single spectrum image of the i-th experimental person in the corresponding test set is input into the single spectrum feature extraction model of the spectral condition to obtain the probability distribution of the person number output by the single spectrum feature extraction model; and the probability of the person number being i in the output probability distribution is read as the accuracy rate of correct recognition; For each environment type, an accuracy fitting function F is constructed, and the accuracy fitting function F is weighted by using the correct recognition accuracy of each single spectrum image in the spectral condition of each wavelength; the accuracy fitting function F is used as a fusion weight calculation strategy.

[0015] In the application environment, the method of pre-collecting the reference image of the person to be verified is: Pre-load all single-spectrum feature extraction models in the face recognition device of the application environment; Before the person to be verified performs face recognition, one or several single-spectrum face images of the person to be verified under various spectral conditions are collected in advance as reference images, and a new person number label is set for each reference image.

[0016] The method of fine-tuning the single spectrum feature extraction model based on the reference image is as follows: Performing data enhancement on the reference image of the person to be verified, and merging the data enhanced reference image with the original multispectral face image set as a new multispectral face image set; Each single-spectral feature extraction model is retrained using the new multispectral face image collection.

[0017] The method of using the fine-tuned single spectrum feature extraction model to perform face recognition based on the actual environmental condition set and fusion weight calculation strategy is as follows: The actual environmental condition set is converted into a new environmental feature vector, thereby obtaining a new M-dimensional discrete point; Calculate the Euclidean distance between the new M-dimensional discrete point and the center points of the K environmental clusters, and use the environmental type corresponding to the environmental cluster with the closest Euclidean distance as the actual environmental type in which the person to be verified is located when performing face recognition; Read all single-spectrum feature extraction models corresponding to the actual environment type, and collect the target single-spectrum images at various wavelengths collected by the face recognition device when performing face recognition; Input each target single spectrum image into the single spectrum feature extraction model of the spectral condition of the corresponding wavelength, and obtain the actual probability distribution of each single spectrum feature extraction model output; Obtain the recognition accuracy of the new personnel number from the actual probability distribution; Input each recognition accuracy of the single spectrum feature extraction model into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F; If the comprehensive accuracy is greater than the preset accuracy coefficient, it is judged as a successful recognition, otherwise it is judged as a failed recognition.

[0018] A multi-spectral image fusion face recognition system is proposed, which includes a sample collection module, an environment clustering module, a recognition model training module, a model fusion module and a face recognition module; wherein each module is electrically connected to each other.

[0019] The sample collection module collects a multispectral face image set and an environmental condition set in advance, and sends the multispectral face image set to the recognition model training module, and sends the environmental condition set to the environmental clustering module.

[0020] The environment clustering module uses a clustering algorithm on the set of environmental conditions to divide K environment clusters, each of which corresponds to an environment type; K is the preset number of environment clusters, and all environment clusters are sent to the face recognition module.

[0021] The recognition model training module, for each environment type, builds and trains a single-spectrum feature extraction model based on a collection of multi-spectral face images, and sends all single-spectrum feature extraction models to the model fusion module.

[0022] The model fusion module designs a fusion weight calculation strategy for each environment type based on the recognition performance of the single spectrum feature extraction model, and sends the fusion weight calculation strategy to the face recognition module.

[0023] The face recognition module collects the reference images of the persons to be verified in advance in the application environment, and fine-tunes the single-spectrum feature extraction model based on the reference images. When the persons to be verified perform face recognition, the module collects the actual environmental condition set, and uses the fine-tuned single-spectrum feature extraction model to perform face recognition based on the actual environmental condition set and the fusion weight calculation strategy.

[0024] An electronic device is proposed, comprising: a processor and a memory, wherein the memory stores a computer program that can be called by the processor; The processor executes the above-mentioned face recognition method of multispectral image fusion by calling the computer program stored in the memory.

[0025] A computer-readable storage medium is provided, on which a rewritable computer program is stored; When the computer program is run on a computer device, the computer device executes the above-mentioned face recognition method of multi-spectral image fusion.

[0026] Compared with the prior art, the present invention has the following beneficial effects: The present invention collects a multi-spectral face image set and an environmental condition set in advance, uses a clustering algorithm on the environmental condition set, divides K environmental clusters, each environmental cluster corresponds to an environmental type, and for each environmental type, constructs and trains a single spectrum feature extraction model based on the multi-spectral face image set, designs a fusion weight calculation strategy for each environmental type based on the recognition performance of the single spectrum feature extraction model, collects a reference image of a person to be verified in advance in an application environment, fine-tunes the single spectrum feature extraction model based on the reference image, collects an actual environmental condition set when the person to be verified performs face recognition, uses the fine-tuned single spectrum feature extraction model, and performs face recognition according to the actual environmental condition set and the fusion weight calculation strategy; first, for different environmental conditions, the environment is divided into different types using a clustering algorithm. Under each environmental type, the single spectrum feature extraction model is trained respectively to extract the corresponding spectral features. Then, by designing a weight strategy, the fusion ratio of different spectral features is adjusted according to the real-time environmental conditions, thereby improving the recognition accuracy and the adaptability of the system, and significantly improving the accuracy and stability of the face recognition system in various complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a flow chart of the face recognition method of multispectral image fusion in Example 1 of the present invention; Figure 2 This is a module connection diagram of the face recognition system for multispectral image fusion in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] Example 1

[0030] like Figure 1 As shown, the face recognition method of multispectral image fusion includes the following steps: Step 1: Collect a set of multispectral face images and a set of environmental conditions in advance; Step 2: Use a clustering algorithm to divide the environmental condition set into K environmental clusters, each of which corresponds to an environmental type; K is the number of preset environmental clusters; Step 3: For each environment type, build and train a single spectrum feature extraction model based on a collection of multispectral face images; Step 4: Based on the recognition performance of the single spectrum feature extraction model, a fusion weight calculation strategy is designed for each environment type; Step 5: In the application environment, pre-collect the reference image of the person to be verified; Step 6: Fine-tune the single spectrum feature extraction model based on the benchmark image; Step 7: When the person to be verified is undergoing face recognition, a set of actual environmental conditions is collected, and the fine-tuned single spectrum feature extraction model is used to perform face recognition based on the set of actual environmental conditions and the fusion weight calculation strategy.

[0031] The method of pre-collecting the multi-spectral face image set and the environmental condition set is as follows: In the test environment, several groups of multispectral images are collected for each experimenter. Each time a multispectral image is collected, a multispectral camera is used to collect the single spectral images of the experimenter at each wavelength to form a multispectral image, and the corresponding environmental parameter set at that time is collected through the environmental sensor network. All multispectral images constitute a multispectral face image set, and the set of environmental parameters corresponding to all multispectral images constitutes an environmental condition set.

[0032] Preferably, the test environment is: By setting controlled conditions in the laboratory, various lighting, background and reflective environments are simulated to systematically and controllably collect test sample data; or: Data is collected under various actual conditions such as public places and office environments, covering different weather, time and congestion levels, in order to fully restore the face recognition needs in real-life scenarios.

[0033] Furthermore, the multispectral image includes spectral images under spectral conditions of different wavelengths, and the spectral conditions include but are not limited to: visible light wavelength, near infrared wavelength, mid-infrared wavelength and far infrared wavelength; wherein the visible light image refers to a standard RGB image, usually within the wavelength range of 400-700 nanometers; the wavelength range of the near infrared image is usually 700-900 nanometers, providing good facial features under low light or night conditions; mid-infrared and far-infrared images are used to provide different facial features under extreme light or temperature conditions.

[0034] The multi-spectral camera is a camera that can capture images of different spectra simultaneously or separately, and can capture images of different spectra at the same time point to achieve better feature alignment.

[0035] Furthermore, the environmental sensor network is composed of sensors corresponding to several environmental parameters deployed in the test environment, and the environmental parameters include but are not limited to lighting conditions, weather conditions and time information. The lighting conditions include but are not limited to light intensity and source direction (such as natural light, indoor lights, etc.), and the weather conditions include but are not limited to sunny days, cloudy days, rainy days, etc.; it can be understood that under different lighting conditions and weather conditions, the face recognition accuracy of different spectra is different. For example, in low-light or no-light environment, infrared spectrum can be used to capture facial information, while under high-light conditions, visible light can provide detailed features; therefore, under different environmental conditions, different spectral recognition models can be used in combination to improve the recognition accuracy.

[0036] Furthermore, the step of using a clustering algorithm to classify the set of environmental conditions into K environmental types includes the following steps: Step 21: for each set of environmental parameters corresponding to the multispectral images, extract all environmental parameters to form an environmental feature vector; Step 22: Treat each environmental feature vector as an M-dimensional discrete point; wherein M is the number of environmental parameters, and the coordinates of the M-dimensional discrete point in each dimension correspond to an environmental parameter in the environmental feature vector; it can be understood that one M-dimensional discrete point corresponds to one M-dimensional spatial coordinate; Step 23: Randomly select K M-dimensional discrete points as initial cluster centers, and the remaining M-dimensional discrete points as non-initial cluster centers; Step 24: For each non-initial cluster center, calculate the Euclidean distance to each initial cluster center, and divide each M-dimensional discrete point into the cluster cluster where the nearest initial cluster center is located; Step 25: Calculate the average value of the environmental parameter corresponding to each dimension of the M-dimensional discrete points in each cluster, form a new M-dimensional discrete point with the average value of the environmental parameter corresponding to each dimension, and use the new M-dimensional discrete point as the new cluster center of the cluster; Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new cluster center, and re-divide each M-dimensional discrete point into the cluster where the nearest new cluster center is located; Step 27: Repeat steps 25 and 26 until the M-dimensional discrete points in all divided clusters no longer change, and mark the number of each cluster as k, k=1, 2, 3, ..., K; obtain all the M-dimensional discrete points contained in each cluster, and form a discrete point set from all the M-dimensional discrete points; the k-th environmental cluster includes the M-dimensional space coordinates of the new cluster center of the k-th cluster, and the environmental feature vector corresponding to each M-dimensional discrete point in the k-th cluster.

[0037] It is understandable that each set of environmental conditions in each environmental cluster has some environmental similarities to a certain extent, that is, it corresponds to an environmental type, and the face photos taken in this environmental type also have a certain degree of similarity in light and shadow, so the same recognition model can be used for face recognition.

[0038] Furthermore, for each environment type, based on the multispectral face image set, the method of constructing and training a single spectrum feature extraction model is as follows: For each environment type: Screen out all multispectral images of each experimenter in this type of environment as a set of control images; Set a unique personnel number for each experimenter; Setting a sample label for each single-spectrum image in the control image set; the sample label is the personnel number of the experimenter corresponding to the single-spectrum image; For each wavelength the spectral condition is: The single spectral images corresponding to the spectral conditions in the control image set are divided into a training set, a validation set and a test set; generally, the division ratios of the training set, the validation set and the test set are: 70%, 15% and 15%; Select a convolutional neural network as a single spectrum feature extraction model; preferably, the convolutional neural network includes but is not limited to ResNet, VGG, MobileNet, etc.; A single-spectrum feature extraction model is constructed, whose input layer matches the size and number of channels of the single-spectrum image.

[0039] The single spectrum feature extraction model also includes, in sequence, a convolution layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully connected layer for classification tasks that connects the output of the convolution layer to the output layer, and an output layer.

[0040] The output layer uses a Softmax activation function to map the output to the probability distribution of each personnel number; The loss function of the single spectrum feature extraction model is set to multi-classification cross entropy; Select Adam optimizer as the optimizer for the single spectrum feature extraction model; Iteratively train the single spectrum feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision indicators.

[0041] It can be understood that after the training process of the above-mentioned single spectrum feature extraction model, a face recognition model is trained for each wavelength spectral condition in each environment type. For example, a face recognition model is trained for visible light wavelength, near-infrared wavelength, mid-infrared wavelength and far-infrared wavelength under outdoor high light conditions, and the recognition accuracy of each face recognition model is obviously different. Therefore, the recognition accuracy needs to be fused and calibrated.

[0042] To this end, the recognition performance based on the single spectrum feature extraction model is used to design a fusion weight calculation strategy for each environment type as follows: For each environment type: Read all single-spectral images in the test set for all experimenters under each wavelength spectral condition; Label each experimenter's number as i; The number of the spectral condition at each wavelength is marked as j; For the spectral condition of the jth wavelength, the single spectrum image of the i-th experimental personnel in the corresponding test set is input into the single spectrum feature extraction model of the spectral condition to obtain the probability distribution of the personnel number output by the single spectrum feature extraction model; and the probability pij of the personnel number i in the output probability distribution is read; it can be understood that the probability pij is the accuracy rate of correct recognition of the single spectrum feature extraction model in the spectral condition of the jth wavelength; For each environment type, an accuracy fitting function F is constructed, and the accuracy fitting function F is weighted by using the correct recognition accuracy of each single spectrum image in the spectral condition of each wavelength; the accuracy fitting function F is used as a fusion weight calculation strategy.

[0043] Specifically, the accuracy fitting function F can be set as: ; Where Pj is the correct recognition accuracy of the single spectrum feature extraction model in the spectral condition of the jth wavelength, wj is the accuracy weight of the spectral condition of the jth wavelength, so the accuracy fitting function F can be understood as the sum of the recognition accuracy rates under the fusion of each wavelength condition.

[0044] Furthermore, the weighted fitting of the accuracy fitting function F refers to using the correct recognition accuracy of each single spectrum image in the spectral condition of each wavelength to fit each accuracy weight wj in the accuracy fitting function F in the solution.

[0045] Specifically, the method of weight fitting the accuracy fitting function F is: Constructing the accuracy loss function , the accuracy loss function .

[0046] By using the gradient descent algorithm to Perform iterative optimization to obtain the accuracy loss function The value of the minimum accuracy weight wj of the spectral condition for each wavelength.

[0047] It can be understood that by fitting the accuracy fitting function F, the appropriate accuracy weights for the spectral conditions of different wavelengths in the corresponding environment type can be calculated, and the accuracy of face recognition in this environment type can be improved through weighted calculation of the accuracy weights.

[0048] Furthermore, in the application environment, the method of pre-collecting the reference image of the person to be verified is: Pre-load all single-spectrum feature extraction models in the face recognition device of the application environment; Before the person to be verified performs face recognition, one or several single-spectrum face images of the person to be verified under various spectral conditions are collected in advance as reference images, and a new person number label is set for each reference image.

[0049] Furthermore, the method of fine-tuning the single spectrum feature extraction model based on the reference image is as follows: Performing data enhancement on the reference image of the person to be verified, and merging the reference image after data enhancement with the original multispectral face image set as a new multispectral face image set; the data enhancement includes but is not limited to horizontal flipping, slight rotation, scaling, etc., to increase diversity; Each single-spectral feature extraction model is retrained using the new multispectral face image collection.

[0050] Furthermore, the method of collecting the actual environmental condition set when the person to be verified performs face recognition is to collect an environmental parameter set composed of actual values ​​of various environmental parameters when performing face recognition.

[0051] The method of using the fine-tuned single spectrum feature extraction model to perform face recognition based on the actual environmental condition set and fusion weight calculation strategy is as follows: The actual environmental condition set is converted into a new environmental feature vector, thereby obtaining a new M-dimensional discrete point; Calculate the Euclidean distance between the new M-dimensional discrete point and the center points of the K environmental clusters, and use the environmental type corresponding to the environmental cluster with the closest Euclidean distance as the actual environmental type in which the person to be verified is located when performing face recognition; Read all single-spectrum feature extraction models corresponding to the actual environment type, and collect the target single-spectrum images at various wavelengths collected by the face recognition device when performing face recognition; Input each target single spectrum image into the single spectrum feature extraction model of the spectral condition of the corresponding wavelength, and obtain the actual probability distribution of each single spectrum feature extraction model output; Obtain the recognition accuracy of the new personnel number from the actual probability distribution; Input each recognition accuracy of the single spectrum feature extraction model into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F; If the comprehensive accuracy is greater than the preset accuracy coefficient, it is judged as a successful recognition, otherwise it is judged as a failed recognition.

[0052] Example 2

[0053] like Figure 2 As shown, the multi-spectral image fusion face recognition system includes a sample collection module, an environment clustering module, a recognition model training module, a model fusion module and a face recognition module; wherein each module is electrically connected to each other.

[0054] The sample collection module collects a multispectral face image set and an environmental condition set in advance, and sends the multispectral face image set to the recognition model training module, and sends the environmental condition set to the environmental clustering module.

[0055] The environment clustering module uses a clustering algorithm on the set of environmental conditions to divide K environment clusters, each of which corresponds to an environment type; K is the preset number of environment clusters, and all environment clusters are sent to the face recognition module.

[0056] The recognition model training module, for each environment type, builds and trains a single-spectrum feature extraction model based on a collection of multi-spectral face images, and sends all single-spectrum feature extraction models to the model fusion module.

[0057] The model fusion module designs a fusion weight calculation strategy for each environment type based on the recognition performance of the single spectrum feature extraction model, and sends the fusion weight calculation strategy to the face recognition module.

[0058] The face recognition module collects the reference images of the persons to be verified in advance in the application environment, and fine-tunes the single-spectrum feature extraction model based on the reference images. When the persons to be verified perform face recognition, the module collects the actual environmental condition set, and uses the fine-tuned single-spectrum feature extraction model to perform face recognition based on the actual environmental condition set and the fusion weight calculation strategy.

[0059] Example 3

[0060] According to another aspect of the present application, an electronic device is provided. The electronic device may include one or more processors and one or more memories. The memories store computer-readable codes, and when the computer-readable codes are run by the one or more processors, the face recognition method of multispectral image fusion as described above may be executed.

[0061] The electronic device may include a bus, one or more CPUs, ROM, RAM, a communication port connected to a network, an input / output component, a hard disk, etc. A storage device in the electronic device, such as a ROM or a hard disk, may store the face recognition method of multispectral image fusion provided in the present application.

[0062] Furthermore, the electronic device may also include a user interface. Of course, this architecture is only exemplary, and when implementing different devices, one or more components in the electronic device may be omitted according to actual needs.

[0063] Example 4

[0064] A computer-readable storage medium according to an embodiment of the present application. Computer-readable instructions are stored on the computer-readable storage medium. When the computer-readable instructions are executed by the processor, the face recognition method for multispectral image fusion according to the embodiment of the present application described with reference to the above figures can be executed. The computer-readable storage medium includes, but is not limited to, for example, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0065] In addition, according to the implementation of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, the non-transitory machine-readable storage medium stores machine-readable instructions, the machine-readable instructions can be run by a processor to execute instructions corresponding to the method steps provided by the present application, and when the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0066] The methods, apparatuses, and devices of the present application may be implemented in many ways. For example, the methods, apparatuses, and devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers recording media storing programs for executing the method according to the present application.

[0067] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0068] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

[0069] The above preset parameters or preset thresholds are all set by technicians in this field according to actual conditions or obtained by simulating a large amount of data.

[0070] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A face recognition method based on multispectral image fusion, characterized in that: The following steps are involved: Step 1: Collect a set of multispectral face images and a set of environmental conditions in advance; Step 2: Use a clustering algorithm to divide the environmental condition set into K environmental clusters, each of which corresponds to an environmental type; K is the number of preset environmental clusters; Step 3: For each environment type, build and train a single spectrum feature extraction model based on a collection of multispectral face images; Step 4: Based on the recognition performance of the single spectrum feature extraction model, a fusion weight calculation strategy is designed for each environment type; Step 5: In the application environment, pre-collect the reference image of the person to be verified; Step 6: Fine-tune the single spectrum feature extraction model based on the benchmark image; Step 7: When the person to be verified is undergoing face recognition, a set of actual environmental conditions is collected, and the fine-tuned single spectrum feature extraction model is used to perform face recognition based on the set of actual environmental conditions and the fusion weight calculation strategy.

2. The face recognition method based on multispectral image fusion according to claim 1, characterized in that: The method of pre-collecting the multi-spectral face image set and the environmental condition set is: In the test environment, several groups of multispectral images are collected for each experimenter. Each time a multispectral image is collected, a multispectral camera is used to collect the single spectral images of the experimenter at each wavelength to form a multispectral image, and the corresponding environmental parameter set at that time is collected through the environmental sensor network. All multispectral images constitute a multispectral face image set, and the set of environmental parameters corresponding to all multispectral images constitutes an environmental condition set.

3. The face recognition method based on multispectral image fusion according to claim 2, characterized in that: The method of using a clustering algorithm to classify the environmental condition set into K environmental types includes the following steps: Step 21: for each set of environmental parameters corresponding to the multispectral images, extract all environmental parameters to form an environmental feature vector; Step 22: Treat each environmental feature vector as an M-dimensional discrete point, wherein M is the number of environmental parameters, and the coordinates of the M-dimensional discrete point in each dimension correspond to an environmental parameter in the environmental feature vector; Step 23: Randomly select K M-dimensional discrete points as initial cluster centers, and the remaining M-dimensional discrete points as non-initial cluster centers; Step 24: For each non-initial cluster center, calculate the Euclidean distance to each initial cluster center, and divide each M-dimensional discrete point into the cluster cluster where the nearest initial cluster center is located; Step 25: Calculate the average value of the environmental parameter corresponding to each dimension of the M-dimensional discrete points in each cluster, form a new M-dimensional discrete point with the average value of the environmental parameter corresponding to each dimension, and use the new M-dimensional discrete point as the new cluster center of the cluster; Step 26: Recalculate the Euclidean distance from each M-dimensional discrete point to each new cluster center, and re-divide each M-dimensional discrete point into the cluster where the nearest new cluster center is located; Step 27: Repeat steps 25 and 26 until the M-dimensional discrete points in all divided clusters no longer change, and mark the number of each cluster as k, k=1, 2, 3, ..., K; obtain all the M-dimensional discrete points contained in each cluster, and form a discrete point set from all the M-dimensional discrete points; the k-th environmental cluster includes the M-dimensional space coordinates of the new cluster center of the k-th cluster, and the environmental feature vector corresponding to each M-dimensional discrete point in the k-th cluster.

4. The face recognition method based on multispectral image fusion according to claim 3, characterized in that: For each environment type, the method of constructing and training a single spectrum feature extraction model based on a multispectral face image set is as follows: For each environment type: Screen out all multispectral images of each experimenter in this type of environment as a set of control images; Set a unique personnel number for each experimenter; Setting a sample label for each single-spectrum image in the control image set; the sample label is the personnel number of the experimenter corresponding to the single-spectrum image; For each wavelength the spectral condition is: The single spectral images corresponding to the spectral conditions in the control image set are divided into a training set, a validation set and a test set; Select a convolutional neural network as a single spectrum feature extraction model; Construct a single spectrum feature extraction model whose input layer matches the size and number of channels of the single spectrum image; The single spectrum feature extraction model further includes, in sequence, a convolution layer for extracting spatial features, a pooling layer for dimensionality reduction and feature aggregation, a fully connected layer for classification tasks, connecting the output of the convolution layer to the output layer, and an output layer; The output layer uses a Softmax activation function to map the output to the probability distribution of each personnel number; The loss function of the single spectrum feature extraction model is set to multi-classification cross entropy; Select Adam optimizer as the optimizer for the single spectrum feature extraction model; Iteratively train the single spectrum feature extraction model on the training set, adjust the hyperparameters according to the performance on the validation set, regularly test the model performance on the validation set, and record the accuracy, recall, and precision indicators.

5. The face recognition method based on multispectral image fusion according to claim 4, characterized in that: The recognition performance based on the single spectrum feature extraction model is used to design a fusion weight calculation strategy for each environment type: For each environment type: Read all single-spectral images in the test set for all experimenters under each wavelength spectral condition; Label each experimenter's number as i; The number of the spectral condition at each wavelength is marked as j; For the spectral condition of the jth wavelength, the single spectrum image of the i-th experimental person in the corresponding test set is input into the single spectrum feature extraction model of the spectral condition to obtain the probability distribution of the person number output by the single spectrum feature extraction model; and the probability of the person number being i in the output probability distribution is read as the accuracy rate of correct recognition; For each environment type, an accuracy fitting function F is constructed, and the accuracy fitting function F is weighted by using the correct recognition accuracy of each single spectrum image in the spectral condition of each wavelength; the accuracy fitting function F is used as a fusion weight calculation strategy.

6. The face recognition method based on multispectral image fusion according to claim 5, characterized in that: In the application environment, the method of pre-collecting the reference image of the person to be verified is: Pre-load all single-spectrum feature extraction models into the face recognition device in the application environment; Before the person to be verified performs face recognition, one or several single-spectrum face images of the person to be verified under various spectral conditions are collected in advance as reference images, and a new person number label is set for each reference image.

7. The face recognition method based on multispectral image fusion according to claim 6, characterized in that: The method of fine-tuning the single spectrum feature extraction model based on the reference image is as follows: Performing data enhancement on the reference image of the person to be verified, and merging the data enhanced reference image with the original multispectral face image set as a new multispectral face image set; Each single-spectral feature extraction model is retrained using the new multispectral face image collection.

8. The face recognition method based on multispectral image fusion according to claim 7, characterized in that: The method of using the fine-tuned single spectrum feature extraction model to perform face recognition based on the actual environmental condition set and fusion weight calculation strategy is as follows: The actual environmental condition set is converted into a new environmental feature vector, thereby obtaining a new M-dimensional discrete point; Calculate the Euclidean distance between the new M-dimensional discrete point and the center points of the K environmental clusters, and use the environmental type corresponding to the environmental cluster with the closest Euclidean distance as the actual environmental type in which the person to be verified is located when performing face recognition; Read all single-spectrum feature extraction models corresponding to the actual environment type, and collect the single-spectrum images of the target at each wavelength collected by the face recognition device when performing face recognition; Input each target single spectrum image into the single spectrum feature extraction model of the spectral condition of the corresponding wavelength, and obtain the actual probability distribution of each single spectrum feature extraction model output; Obtain the recognition accuracy of the new personnel number from the actual probability distribution; Input each recognition accuracy of the single spectrum feature extraction model into the accuracy fitting function F to obtain the comprehensive accuracy output by the accuracy fitting function F; If the comprehensive accuracy is greater than the preset accuracy coefficient, it is judged as a successful recognition, otherwise it is judged as a failed recognition.

9. A multi-spectral image fusion face recognition system, which is used to implement the multi-spectral image fusion face recognition method according to any one of claims 1 to 8, characterized in that: It includes a sample collection module, an environment clustering module, a recognition model training module, a model fusion module and a face recognition module; wherein each module is electrically connected; A sample collection module collects a multispectral face image set and an environmental condition set in advance, and sends the multispectral face image set to the recognition model training module, and sends the environmental condition set to the environmental clustering module; The environment clustering module uses a clustering algorithm on the set of environmental conditions to divide K environment clusters, each of which corresponds to an environment type; K is the preset number of environment clusters, and all environment clusters are sent to the face recognition module; The recognition model training module builds and trains a single-spectrum feature extraction model for each environment type based on a multi-spectral face image collection, and sends all single-spectrum feature extraction models to the model fusion module; The model fusion module designs a fusion weight calculation strategy for each environment type based on the recognition performance of the single spectrum feature extraction model, and sends the fusion weight calculation strategy to the face recognition module; The face recognition module collects the reference images of the persons to be verified in advance in the application environment, and fine-tunes the single-spectrum feature extraction model based on the reference images. When the persons to be verified perform face recognition, the module collects the actual environmental condition set, and uses the fine-tuned single-spectrum feature extraction model to perform face recognition based on the actual environmental condition set and the fusion weight calculation strategy.

10. An electronic device, characterized in that: include: A processor and a memory, wherein: The memory stores a computer program that can be called by the processor; The processor executes the face recognition method for multispectral image fusion according to any one of claims 1 to 8 in the background by calling the computer program stored in the memory.

11. A computer-readable storage medium, characterized in that: A rewritable computer program is stored thereon; When the computer program is executed on a computer device, the computer device executes the face recognition method of multispectral image fusion according to any one of claims 1 to 8 in the background.

Images