Intelligent cabinet multi-identity identification information fusion method and system based on information entropy

Abstract

The invention relates to an intelligent cabinet multi-identity recognition information fusion method and system based on information entropy, and the method comprises the steps: carrying out the photographing of a human face image and a finger vein image through terminal equipment on an intelligent cabinet, and obtaining the feature vectors of the human face and the finger vein through a feature extraction algorithm; an improved deformable convolution algorithm is used for eliminating the measurement inconsistency of face features and finger vein features, and after feature alignment, the face features and the finger vein features are mapped to the same vector space and are connected in series to form a group of multi-identity feature vectors. And finally, judging whether the multi-identity feature vector is valid or not by using the lowest valid feature dimension obtained by the information entropy, outputting the valid feature vector, comparing the valid feature vector with a feature database, and outputting a recognition result. The face information and the finger vein information are comprehensively utilized, and the recognition precision and accuracy of the intelligent cabinet recognition system in a complex environment are effectively improved.

Description

Technical Field

[0001] The present invention relates to a method and system for fusion of multi-identity recognition information of smart cabinets based on information entropy, specifically to a method for performing information entropy analysis on features extracted from different identity recognition modalities, and optimizing the output decision threshold of a support vector machine according to the average entropy value, thereby providing a more accurate and secure solution for unlocking the smart cabinet, and belonging to the technical field of multi-identity recognition of smart cabinets. Background Art

[0002] With the rapid development of intelligent technology, smart cabinets, as part of modern smart home and smart business systems, are widely used in areas such as item storage, information management, and security control. In traditional smart cabinets, a single identity recognition technology is usually used, such as passwords, RFID cards, or a single biometric feature (such as fingerprints, facial recognition, etc.) for identity verification. However, these single recognition methods are often susceptible to interference when faced with problems such as lighting, environmental changes, or identity forgery, resulting in reduced recognition accuracy or insufficient security. Especially in complex or dynamic environments, the robustness and accuracy of single recognition technologies are significantly reduced. In addition, most existing multimodal recognition systems use a simple weighted fusion method, which fails to fully consider the performance differences of different modalities in different environments. This makes the system overly dependent on a certain modality even when there is high uncertainty, thereby affecting the overall recognition effect. Therefore, how to improve the accuracy and robustness of smart cabinet recognition systems has become a major challenge facing current technology. Summary of the Invention

[0003] To overcome the inaccurate information recognition issues of existing systems, this paper provides a method and system for multi-identity identification in smart lockers based on information entropy. This method intelligently identifies and fuses various modalities, extracting corresponding effective features based on the performance of each modality. This method utilizes information entropy to optimize the decision threshold of a support vector machine (SVM), enabling multi-modal identity recognition. This not only makes smart locker authentication more accurate, but also effectively prevents malicious attacks and improves system security.

[0004] A method for fusing multi-identity identification information of a smart cabinet based on information entropy includes the following steps:

[0005] Step 1: Use the terminal device built into the smart cabinet to collect facial and finger vein images, and use the YOLO algorithm to detect the key points of the finger vein and face to obtain the key point information of the finger vein and face;

[0006] Step 2: Extract finger vein and facial feature point information through LBP and Dlib library, extract feature values of relevant feature points by region, perform feature vector fusion, and obtain a comprehensive recognition feature vector;

[0007] Step 3: Based on the finger vein and facial feature key point information, the support vector machine is trained to draw the ROC curve corresponding to the feature points. The information entropy is used to adjust and optimize the judgment threshold to determine whether the comprehensive recognition feature vector is valid.

[0008] Step 4: Perform detection based on the finger vein information and facial information provided by the identification person to obtain the final detection result.

[0009] The step 1 specifically includes: detecting whether a person appears in the image according to the YOLO algorithm; if a person appears, identifying and detecting the entire image; if no person appears, determining that the image is invalid and not performing subsequent recognition; and collecting facial images and finger vein images through the information collection device provided by the smart cabinet to obtain facial images and finger vein images.

[0010] The second step specifically includes:

[0011] Step 2.1: Divide the finger vein features and facial images into blocks using masks. The image is divided into blocks by masks corresponding to different feature points. Each mask is implemented by a binary image (mask). Only pixels in the target area are retained, and the rest of the area is ignored. The area is divided into blocks according to the feature point type.

[0012] Step 2.2: Multi-scale fusion strategy, through the improved deformable convolution algorithm, combines the information of finger vein and facial features to dynamically generate offsets, breaking the traditional limitation of being based on a single modality and solving the spatial inconsistency problem of different modal features. For a given input image, the finger vein feature map X is extracted through two branches respectively. v and facial feature map X f , generate a joint feature map X by splicing c ,Based on the joint feature map, a cross-attention module is introduced, so that the information of one modality can be used as a query, and the information of the other modality can be used as the key and value to calculate the attention weight.

[0013] The weight of each sampling point is dynamically adjusted by the modulation function γ(·), and the final output eigenvalue is calculated as:

[0014]

[0015] X cross =A·V

[0016] Δ=Conv(X cross )

[0017] Where k is the convolution kernel size, A is the cross-modal attention weight, d is the dimension of the vector, X cross is the cross-modal feature, Δx, Δy are the offset vectors of each sampling position;

[0018] The feature maps of different regions are fused. Let V1, V2, and V3 be the feature outputs of each region of the finger vein feature, and let F1, F2, F3, F4, and F5 be the feature outputs of each region of the face feature. The output of each stage is sampled to the same target size using the improved deformable convolution algorithm and then integrated.

[0019] Step 2.3: Extract the finger vein feature value through the local binary pattern. The calculated LBP value represents the local texture pattern of the area, which is recorded as v i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0, and the feature value groups detected under different finger vein mask blocks are recorded as the feature vector of the area [v i ,v i+1 ,…,v j ];

[0020] The input face image is detected by YOLO, the coordinates of the key points of face recognition are extracted using the Dlib library, the spatial relationship between the key points of the face is calculated, and a set of adjacent point pairs (p i ,p j ), calculate the Euclidean distance: The calculated distance value represents the local texture pattern of the area, denoted as f i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0, and the feature value groups detected under different face mask blocks are recorded as the feature vectors of the area [f i ,f i+1 ,…,f j ];

[0021] Step 2.4: For each area of the mask block, concatenate the feature vectors extracted from each area to generate a comprehensive recognition feature vector and number it. The comprehensive recognition feature vector is [f1,f2,…,f n ,v1,v2,…,v m ].

[0022] The step three specifically includes:

[0023] Step 3.1: Preprocess the data by collecting a large amount of labeled sample data. Use Dlib to extract facial features and LBP to extract finger vein features, and divide the data into training and test sets.

[0024] The sample data labeling rules trained by SVM are:

[0025] For finger vein features, the error between the extracted LBP value and the feature LBP value in the database does not exceed 85 percent;

[0026] For facial features, a dynamic threshold mechanism is established: set the initial threshold δ0, δ ij =δ0×α, where α is the adaptive coefficient. It is considered that there is an error at this point.

[0027] Step 3.2: Use the grid search method to adjust the key parameters C and gamma, use the RBF kernel to achieve high-dimensional mapping of the data, use the SMO algorithm to optimize the model parameters, use the support vector machine to draw the ROC curve, and use the information entropy to optimize the judgment threshold of the ROC curve output. The entropy value is calculated as follows:

[0028] H(p)=-plog(p)-(1-p)log(1-p)

[0029] p represents the probability that the sample is predicted to be positive, and 1-p represents the probability that the sample is predicted to be negative. All thresholds returned by the ROC curve are traversed to find the threshold that minimizes the average entropy of samples predicted to be positive while the FPR does not exceed the preset value.

[0030] The step 4 specifically includes:

[0031] The finger vein information and face information provided by the identity identification personnel are input for detection to determine whether the number of valid feature points in the comprehensive identification feature vector reaches the judgment threshold. If so, it means that the fused feature vector meets the identification requirements; if not, it means that the fused feature vector does not meet the identification requirements; and the final detection result is obtained.

[0032] An intelligent cabinet multi-identity recognition information fusion system based on information entropy, including an image acquisition module, an identification information key point detection module, and an operation process and feature recognition module;

[0033] The image acquisition module is responsible for obtaining the information of the detection image according to the detection requirements;

[0034] The identification information key point detection module uses the information of the picture collected by the image acquisition module to detect the finger vein and facial key points through the finger vein and facial key point recognition algorithm, and records the center coordinates of the key feature points of the identified person; the operation process and feature recognition module is used to save the feature point recognition results, and judge the fused comprehensive feature vector based on the threshold value obtained by combining the ROC curve graph corresponding to the key feature points.

[0035] The image acquisition module includes: a task assignment and shooting unit: responsible for issuing identification task requirements to personnel and capturing images of the personnel identification process; a judgment unit: used to judge whether the image information recognizes a human body. If a human body is recognized, it enters the execution unit. If no human body is recognized, the image is considered invalid and does not enter the execution unit.

[0036] The identification information key point detection module includes: an execution unit: responsible for detecting each key point of the finger vein and the face, and recording the center coordinates of the key feature points of the identified person, and at the same time, saving data such as the information of each key point of the finger vein and the face.

[0037] The operation process and feature recognition module include: a judgment unit: used to judge whether the key feature points corresponding to the finger vein and face during identity recognition are valid information, using the support vector machine learning method to divide the results into two categories, namely valid features and invalid features.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] The present invention introduces information entropy, adjusts and optimizes the judgment threshold of the support vector machine output, realizes the efficient fusion of multimodal information, and significantly improves the accuracy and robustness of recognition. The system combines multiple identity authentication methods such as face recognition and finger vein recognition, and upgrades the fusion features from fixed template matching to adaptive feature decoupling, optimizes data processing efficiency, and effectively copes with changing environments such as complex lighting and posture occlusion. At the same time, the multimodal fusion of finger vein features and facial features improves anti-counterfeiting capabilities and security, breaking through the limitations of traditional simple weighted fusion methods and ensuring the stable operation of the system in scenarios with high security requirements. This method is scientific and reasonable, highly practical, and suitable for finance, logistics, medical care and other fields. It has important innovative value and application prospects.

[0040] The present invention uses a terminal device on a smart cabinet to capture facial and finger vein images, respectively. A feature extraction algorithm is used to obtain facial and finger vein feature vectors. An improved deformable convolution algorithm is used to eliminate inconsistencies in the measurement of facial and finger vein features. After feature alignment, the features are mapped to the same vector space and concatenated into a set of multi-identity feature vectors. Finally, the minimum valid feature dimension obtained using information entropy is used to determine whether the multi-identity feature vector is valid. The valid feature vector is output, compared with a feature database, and the recognition result is output. The present invention utilizes a combination of facial and finger vein information to effectively improve the recognition precision and accuracy of the smart cabinet recognition system in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0042] Figure 1This is a system principle block diagram of a method and system for fusion of multi-identity identification information of smart cabinets based on information entropy according to the present invention;

[0043] Figure 2 It is a flowchart of the present invention;

[0044] Figure 3 This is a diagram of the improved feature key point extraction and fusion module for combining finger vein and face;

[0045] Figure 4 This is a flow chart of the present invention using support vector machine (SVM) to obtain the judgment threshold;

[0046] Figure 5 This is the ROC curve diagram after the present invention uses information entropy to adjust and optimize the decision threshold. DETAILED DESCRIPTION

[0047] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0048] The following reference Figure 1 The embodiments of the present invention will be described. The embodiments described below are merely examples of the present invention and are not intended to limit the scope of protection of the present invention.

[0049] Figure 1 The figure shows a principle block diagram of a multi-identity recognition information fusion method and system for a smart cabinet based on information entropy. The system includes an image acquisition module, an identification information key point detection module, an operation process and feature recognition module.

[0050] The image acquisition module is responsible for obtaining the information of the detection image according to the detection requirements.

[0051] The identification information key point detection module uses the information of the picture collected by the image acquisition module to detect the finger vein and facial key points through the finger vein and facial key point recognition algorithm, and records the center coordinates of the key feature points of the identified person.

[0052] The operation process and feature recognition module are used to save the feature point recognition results, and judge the fused comprehensive feature vector based on the threshold value obtained by combining the ROC curve graph corresponding to the key feature points.

[0053] Figure 2This is a flowchart for illustrating the multi-modal feature fusion judgment of the identification personnel in this embodiment, a method for fusion of multi-identity identification information of smart cabinets based on information entropy, and its workflow is as follows:

[0054] Step 1: Obtain image data. Obtain the image data of the person being identified through the terminal.

[0055] The smart cabinet's built-in terminal device is used to capture facial and finger vein images, including geometric transformation of the images to correct geometric distortion caused by angle or distance; denoising filtering to eliminate environmental noise, and finger vein and facial key point detection using the YOLO algorithm to obtain finger vein and facial key point information;

[0056] Step 2: Divide the feature points into blocks by region using a mask. Divide the masks into different regions according to the feature point type and extract different types of feature points separately.

[0057] Finger vein and facial feature point information is extracted through LBP and Dlib library, the feature values of relevant feature points are extracted by region, and feature vectors are fused to obtain a comprehensive recognition feature vector;

[0058] Finger vein and facial keypoint detection is performed using the YOLO algorithm to obtain finger vein and facial keypoint information. We first precisely locate key feature points in the finger vein and facial images. Then, based on the type of feature points, we generate a corresponding binary mask for each target region. Specifically, each mask is a binary image where the pixel value within the target region is set to 1, while the pixel value elsewhere is set to 0, retaining only the pixel information in the target region. Morphological processing is then performed to remove noise and fill holes, ensuring smooth mask edges and accurate regional coverage. Using these masks, we can finely segment the original image. Specifically, we isolate information related to finger vein texture or key facial features according to the regions defined by each feature point, while ignoring irrelevant background. This segmentation process not only improves feature extraction accuracy but also provides a more clear and effective foundation for subsequent feature fusion and matching. Finger vein images are divided into geometric feature point regions, centerline feature point regions, and topological feature point regions; facial image feature points are divided into contour regions, eye regions, nose regions, eyebrow regions, and mouth regions.

[0059] Step 3: Extract the feature values of finger veins and key feature points of the face.

[0060] Based on the finger vein and facial feature key point information, the support vector machine is trained to draw the ROC curve corresponding to the feature points. The information entropy is used to adjust and optimize the judgment threshold to determine whether the comprehensive recognition feature vector is valid.

[0061] The image data is divided into non-overlapping sub-block areas at the edge end. For each sub-block, a central pixel is selected and then compared with the neighboring pixels of the central pixel. The calculated LBP value represents the local texture pattern of the area, which is recorded as v i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0. The feature value groups detected under different finger vein mask blocks are recorded as the feature vectors of the area [v i ,v i+1 ,…,v j ].

[0062] The input face image is detected by YOLO, and the coordinates of key points of face recognition are extracted using the Dlib library. The spatial relationship between the key points of the face is calculated, and a set of adjacent point pairs (p i ,p j ), calculate the Euclidean distance: The calculated distance value represents the local texture pattern of the area, denoted as f i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0. The feature value groups detected under different face mask blocks are recorded as the feature vectors of the area [f i ,f i+1 ,…,f j ].

[0063] Perform feature fusion to obtain a comprehensive recognition feature vector. In the previous steps, we have extracted feature images of their respective regions from the finger vein and face images. The finger vein image is divided into geometric feature point area, centerline feature point area, and topological feature point area; the feature points of the face image are divided into contour area, eye area, nose area, eyebrow area, and mouth area. A multi-scale fusion strategy is adopted. Through the improved deformable convolution algorithm, the information of finger vein and face features is combined to dynamically generate offsets, breaking the traditional limitation of being based on a single modality and effectively solving the spatial inconsistency problem of different modal features. First, for a given input image, the finger vein feature map X is extracted through two branches respectively. v and facial feature map X f , then, generate a joint feature map X by splicing c Based on the joint feature map, a cross-attention module is introduced, so that the information of one modality can be used as a query, and the information of the other modality can be used as the key and value to calculate the attention weight. The specific steps are as follows:

[0064] X v The projection is the query vector Q, and X fThey are projected into key vector K and value vector V respectively. The similarity between Q and K is calculated using dot product and normalized by softmax to obtain the cross-modal attention weight A:

[0065]

[0066] Where d is the dimension of the vector and is used for scaling.

[0067] Use weight A to perform weighted summation on V to obtain the fused cross-modal feature representation X cross :

[0068] X cross =A·V

[0069] Get the fused cross-modal feature X cross After that, further feature extraction is performed through multiple convolutional layers, and the offset vector Δx, Δy of each sampling position is finally output. This process can be expressed as:

[0070] Δ=Conv(X cross )

[0071] Where Δ contains the offset information of each spatial position.

[0072] The image size of each divided region is modified by the improved deformable convolution algorithm to meet the needs of feature fusion. The feature values of the finger vein and key feature points of the face are extracted using step S3. The feature vectors extracted from each region are concatenated to generate a comprehensive recognition feature vector and numbered. The comprehensive recognition feature vector is [f1, f2, ..., f n ,v1,v2,…,v m ].

[0073] The integrated recognition feature vector is run through a support vector machine to determine the number of valid eigenvalues within it. This number is then compared with a threshold. If the number is greater than the threshold, the feature vector is considered valid. The threshold is obtained by optimizing the ROC curve output using information entropy. The entropy value is calculated as: H(p) = -plog(p) - (1-p)log(1-p), where p represents the probability of a sample being predicted as positive, and 1-p represents the probability of a sample being predicted as negative. The ROC curve is constructed by connecting all (FPR, TPR) points under each threshold in descending order. Ideally, the closer the curve is to the upper left corner (FPR = 0, TPR = 1), the better the model performance. Each threshold corresponds to a set of (FPR, TPR) coordinate points.

[0074] Figure 3 It is used to illustrate the improved part of the model in the finger vein and facial feature point detection and fusion model.

[0075] By utilizing an improved finger vein and facial feature extraction network and employing a feature-driven dynamic mask segmentation mechanism, a spatial distribution model of feature points is constructed to achieve adaptive segment size adjustment. A 0.5mm precision rasterized mask is designed based on the topological structural features of the finger vein, and morphological closing operations are used to fill tiny holes. For the facial region, segment size is dynamically adjusted based on the spatial relationships between key points, and cross-scale feature alignment is achieved through an improved deformable convolution algorithm. This design upgrades the fusion process from fixed template matching to adaptive feature decoupling, improving feature extraction accuracy under complex conditions.

[0076] The feature maps of different regions are fused. Let V1, V2, and V3 be the feature outputs of each region of the finger vein feature, and let F1, F2, F3, F4, and F5 be the feature outputs of each region of the face feature. The output of each stage is sampled to the same target size through the deformable convolution algorithm and then integrated.

[0077] Deformable Convolution Algorithm:

[0078] Feature fusion requires modifying the size of the high-level feature map first. Therefore, the deformable convolution algorithm is used to modify the size of the high-level feature map. The principle of the deformable convolution algorithm sampling method is as follows: In the original feature map, the offset of the basic sampling grid is generated by the learnable parameters Δx, Δy, and the neighborhood sampling position of the target sampling point P(x, y) is Q i,j =(x+Δx i,j ,y+Δy i,j ), where i and j represent the sampling point index of the convolution kernel. The weight of each sampling point is dynamically adjusted by the modulation function γ(·), and the final output eigenvalue is calculated as:

[0079]

[0080] Where k is the convolution kernel size.

[0081] While the deformable convolution algorithm increases computational effort during upsampling, its weighted fusion of pixel neighborhoods effectively maintains the integrity of key features. While improving image resolution, this method significantly reduces the risk of blurring feature edges and maintains the continuous variation of grayscale values, facilitating the extraction of finger vein and facial features.

[0082] After the improved finger vein and facial feature point detection model, the model's recognition accuracy increased from 84.5% to 92.3%, thereby improving the recognition accuracy of finger vein and facial feature points and improving the effect of identity recognition.

[0083] Figure 4This is a flowchart for using information entropy to train a support vector machine to draw a receiver operating characteristic (ROC) curve and obtain the judgment threshold. The core idea of a support vector machine (SVM) is to achieve classification or regression by finding an optimal hyperplane that maximizes the geometric separation between samples of different categories (i.e., the shortest distance from the support vector to the hyperplane). First, a large amount of labeled sample data is collected, including matching face and finger vein image pairs (positive samples) and mismatched image pairs (negative samples).

[0084] The annotation of samples follows the following rules: for finger vein features, the error between the extracted LBP value and the feature LBP value in the database shall not exceed 85%; for facial features, a dynamic threshold mechanism is established: the initial threshold δ0, δ ij =δ0×α, where α is the adaptive coefficient. It is considered that there is an error in this point. These samples are preprocessed and feature extracted (using Dlib to extract facial features and LBP to extract finger vein features), and the features are standardized to eliminate dimensional differences. The feature points of finger veins vary from person to person. The number of endpoints formed when the veins are not sufficiently transmitted inside the finger is usually 5-15. The number of bifurcation points formed at the branches of the veins is about 10-30. The number of adjacent bifurcation point combinations is about 5-10. Therefore, we selected 118 feature points (68 facial key point features and 50 finger vein key point features) as training samples, with labels of +1 and -1 (-1 represents a negative sample, +1 represents a positive sample). In order to evaluate the generalization ability of the model, we divided the data into training and test sets, with a division ratio of 70% training and 30% testing.

[0085] The decision function of SVM can be expressed as: Where: f(x) represents the classification output of the input sample x, which is a label (+1 or -1), sign is the sign function, according to φ is the radial basis function, c i is the center point of the i-th hidden layer neuron, w i is the output layer weight, and b is the bias term. Traverse all thresholds returned by the ROC curve, and for each threshold, calculate the average entropy of the test sample predicted as the positive class. Then, for these positive prediction samples, calculate the information entropy corresponding to their predicted probabilities, and then obtain an average entropy value. For a given threshold τ∈T, all predicted probabilities p i Samples greater than or equal to τ are considered to be positive, that is: {i|p i ≥τ}; average the entropy values of all positive samples:

[0086]

[0087] where Nτ is the total number of samples predicted as positive under the threshold τ. This average entropy value reflects the confidence of the model in predicting the positive class under this threshold: the lower the entropy, the higher the confidence.

[0088] For the current threshold thr, determine whether the predicted probability y_proba of each test sample is greater than or equal to thr, and obtain a binary prediction result. For each sample, we already have the predicted probability p (i.e., the value in y_proba), so we directly calculate the entropy value of the sample. Using the result of y_pred_thr, select the entropy value corresponding to the samples predicted as the positive class, calculate the average entropy of these samples, and record the average entropy value under this threshold for subsequent comparison. From the candidate thresholds that meet FPR ≤ 0.1, select the one with the smallest average entropy as the final decision threshold. Based on the FPR array calculated by the ROC curve, select the indices corresponding to all thresholds with FPR less than or equal to 0.1. The thresholds corresponding to these indices meet our FPR requirements. Compare the average entropy values of these candidate thresholds. Select the threshold with the smallest average entropy value and use it for the final prediction and evaluation. Finally, with FPR = 0.06, TPR = 0.78, and a threshold of 0.52, not only can the low error requirement be met, but the model's predictions for the positive class are also more accurate. The ROC curve result is: Figure 5 shown.

[0089] The output threshold of the ROC curve is converted into a specific feature value to measure the impact of each feature on the decision function output. Each feature is roughly considered to have a "contribution value." These contribution values are accumulated. When the cumulative sum reaches or exceeds the decision threshold, it is considered that there is sufficient feature information to support a positive prediction. The "gradient" is used to measure the impact of each feature on the decision function output for the current sample.

[0090] For a given test sample x, the decision function f(x) is about each feature x i The partial derivative of It can reflect the local sensitivity of the feature. The absolute value is taken as the contribution index and recorded as The contribution values of all features are accumulated until the cumulative value reaches or exceeds the decision threshold T.

[0091] For the RBF kernel, the kernel function is

[0092] K(x,x')=exp(-γ||x-x'|| 2 )

[0093] x i Finding partial derivatives involves

[0094]

[0095] This allows for a gradient approximation.

[0096] Here, the feature contribution value of each feature point is about 0.0085, and the threshold of 0.52 corresponds to 62 feature points. When the concatenated feature vector [f1,f2,…,f n ,v1,v2,…,v m ] When the valid feature points in the vector are greater than 62, the feature vector can be determined to be a valid vector. The 118-dimensional feature vector identified by the tester through fusion is [116.10172.59100.48121.580106.33139.4084.38132.9757.06148.85237.4594.00192.8069.35123.340178.65141.0079.00155.32254.32135.3985.66165.85122.3028.36180.880115.8267.52154.62 135.44289.10101.74180.6159.76108.2064.97193.170145.0081.27159.27121.65 267.89191.08103.440245.670139.17217.89122.320104.50253.44169.320142.64 222.2264.120136.07289.01168.890127.47274.5560.300138.50232.78141.880 The data was fed into the machine learning SVM classifier, which identified 75 valid eigenvalues. Those greater than the threshold were considered valid eigenvectors. The evaluation result reporting module concluded that identity recognition was possible.

[0097] Step 4: Perform detection based on the finger vein information and facial information provided by the identification person to obtain the final detection result.

[0098] The finger vein information and face information provided by the identity identification personnel are input for detection to determine whether the number of valid feature points in the comprehensive identification feature vector reaches the judgment threshold. If so, it means that the fused feature vector meets the identification requirements; if not, it means that the fused feature vector does not meet the identification requirements; and the final detection result is obtained.

[0099] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations to these embodiments may be made without departing from the principles and spirit of the present invention, and these changes and modifications still fall within the scope of protection of the present invention.

Claims

1. A method for fusion of multi-identity identification information of smart cabinets based on information entropy, characterized by: The following steps are involved: Step 1: Use the terminal device built into the smart cabinet to collect facial and finger vein images, and use the YOLO algorithm to detect the key points of the finger vein and face to obtain the key point information of the finger vein and face; Step 2: Extract finger vein and facial feature point information through LBP and Dlib library, extract feature values of relevant feature points by region, perform feature vector fusion, and obtain a comprehensive recognition feature vector; Step 3: Based on the finger vein and facial feature key point information, the support vector machine is trained to draw the ROC curve corresponding to the feature points. The information entropy is used to adjust and optimize the judgment threshold to determine whether the comprehensive recognition feature vector is valid. Step 4: Perform detection based on the finger vein information and facial information provided by the identification person to obtain the final detection result.

2. The method for fusion of multi-identity identification information of smart cabinets based on information entropy according to claim 1 is characterized by: The step one specifically includes: detecting whether a person appears in the image according to the YOLO algorithm; if a person appears, identifying and detecting the entire image; if no person appears, determining that the image is invalid and not performing subsequent recognition; and collecting facial images and finger vein images through the information collection device provided by the smart cabinet to obtain facial images and finger vein images.

3. The method for fusion of multi-identity identification information of smart cabinets based on information entropy according to claim 1 is characterized by: The second step specifically includes: Step 2.1: Divide the finger vein features and facial images into blocks using masks. The images are divided into blocks by masks corresponding to different feature points. Each mask is implemented using a binary image. Only pixels in the target area are retained, and the rest of the area is ignored. The regions are divided into blocks according to the feature point type. Step 2.2: Multi-scale fusion strategy, through the improved deformable convolution algorithm, combines the information of finger vein and facial features to dynamically generate offsets, breaking the traditional limitation of being based on a single modality and solving the spatial inconsistency problem of different modal features. For a given input image, the finger vein feature map X is extracted through two branches respectively. v and facial feature map X f , generate a joint feature map X by splicing c ,Based on the joint feature map, a cross attention module is introduced, so that the information of one modality can be used as a query, and the information of the other modality can be used as the key and value to calculate the attention weight. The weight of each sampling point is dynamically adjusted by the modulation function γ(·), and the final output eigenvalue is calculated as: X cross =A·V Δ=Conv(X cross ) Where k is the convolution kernel size, A is the cross-modal attention weight, d is the dimension of the vector, X cross is the cross-modal feature, Δx, Δy are the offset vectors of each sampling position; The feature maps of different regions are fused. Let V1, V2, and V3 be the feature outputs of each region of the finger vein feature, and let F1, F2, F3, F4, and F5 be the feature outputs of each region of the face feature. The output of each stage is sampled to the same target size using the improved deformable convolution algorithm and then integrated. Step 2.3: Extract the finger vein feature value through the local binary pattern. The calculated LBP value represents the local texture pattern of the area, which is recorded as v i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0, and the feature value groups detected under different finger vein mask blocks are recorded as the feature vector of the area [v i ,v i+1 ,…,v j ]; The input face image is detected by YOLO, the coordinates of the key points of face recognition are extracted using the Dlib library, the spatial relationship between the key points of the face is calculated, and a set of adjacent point pairs (p i ,p j ), calculate the Euclidean distance: The calculated distance value represents the local texture pattern of the area, denoted as f i For areas where features are missing or cannot be extracted, the feature values are uniformly recorded as 0, and the feature value groups detected under different face mask blocks are recorded as the feature vectors of the area [f i ,f i+1 ,…,f j ]; Step 2.4: For each area of the mask block, concatenate the feature vectors extracted from each area to generate a comprehensive recognition feature vector and number it. The comprehensive recognition feature vector is [f1,f2,…,f n ,v1,v2,…,v m ].

4. The method for fusion of multi-identity identification information of smart cabinets based on information entropy according to claim 1 is characterized by: The step three specifically includes: Step 3.1: Preprocess the data by collecting a large amount of labeled sample data. Use Dlib to extract facial features and LBP to extract finger vein features, and divide the data into training and test sets. Step 3.2: Use the grid search method to adjust the key parameters C and gamma, use the RBF kernel to achieve high-dimensional mapping of the data, use the SMO algorithm to optimize the model parameters, use the support vector machine to draw the ROC curve, and use the information entropy to optimize the judgment threshold of the ROC curve output. The entropy value is calculated as follows: H(p)=-plog(p)-(1-p)log(1-p) p represents the probability that the sample is predicted to be positive, and 1-p represents the probability that the sample is predicted to be negative. All thresholds returned by the ROC curve are traversed to find the threshold that minimizes the average entropy of samples predicted to be positive while the FPR does not exceed the preset value.

5. The method for fusion of multi-identity identification information of smart cabinets based on information entropy according to claim 1 is characterized by: The step 4 specifically includes: The finger vein information and face information provided by the identity identification personnel are input for detection to determine whether the number of valid feature points in the comprehensive identification feature vector reaches the judgment threshold. If so, it means that the fused feature vector meets the identification requirements; if not, it means that the fused feature vector does not meet the identification requirements; and the final detection result is obtained.

6. A fusion system for the smart cabinet multi-identity recognition information fusion method based on information entropy according to any one of claims 1 to 5, characterized in that: It includes image acquisition module, recognition information key point detection module, operation process and feature recognition module; The image acquisition module is responsible for obtaining the information of the detection image according to the detection requirements; The identification information key point detection module uses the information of the picture collected by the image acquisition module to detect the finger vein and facial key points through the finger vein and facial key point recognition algorithm, and records the center coordinates of the key feature points of the identified person; The operation process and feature recognition module are used to save the feature point recognition results, and judge the fused comprehensive feature vector based on the threshold value obtained by combining the ROC curve graph corresponding to the key feature points.

Images