Palmprint recognition method and system based on multi-level feature bilinear fusion network

Abstract

The application provides a palmprint recognition method and system based on a multi-level feature bilinear fusion network, comprising: a fusion step of fusing palmprint deep-level features and palmprint shallow-level features extracted from the multi-level feature bilinear fusion network, and then performing multi-receptive field bilinear pooling on the features; a lightweight step of performing lightweight processing on the multi-level feature bilinear fusion network after the multi-receptive field bilinear pooling; and an encoding position step of embedding a coordinate attention mechanism into a network model of the multi-level feature bilinear fusion network after the lightweight processing. The multi-level feature bilinear fusion network is adopted to deeply mine the detail information of the palmprint, thereby improving the recognition accuracy.

Description

Technical Field

[0001] This invention relates to the field of biometric recognition technology, specifically to a palmprint recognition method and system based on a multi-level feature bilinear fusion network. Background Technology

[0002] Differences between palmprint images can be influenced by factors such as pose, angle, and position, thus palmprint image recognition places high demands on visual feature representation capabilities. Conventional neural networks can only capture a single-scale receptive field, making accurate recognition of multi-scale targets like palmprints difficult. To address these issues, and considering the small differences between palmprint classes, this invention designs a multi-level feature bilinear fusion network.

[0003] Chinese invention patent application CN110210288A discloses a palmprint recognition method and device. Its technical solution is based on a palmprint neural network model to predict the target user ID corresponding to the palmprint region image. By combining palmprint recognition technology with neural network technology, it can quickly and accurately identify the target user matching the detected palmprint, and can perform palmprint recognition well under various shooting conditions. However, the large number of network layers leads to a significant increase in computational complexity and parameter quantity, which is detrimental to subsequent hardware implementation. This application involves a multi-level feature bilinear fusion network that deeply mines the detailed information of the palmprint with fewer network layers. Furthermore, it employs lightweight processing, greatly reducing computational complexity and parameter quantity. Finally, it embeds a coordinate attention mechanism to further improve recognition accuracy, achieving better overall performance. Summary of the Invention

[0004] In view of the shortcomings of the prior art, the purpose of this invention is to provide a palmprint recognition method and system based on a multi-level feature bilinear fusion network.

[0005] A palmprint recognition method based on a multi-level feature bilinear fusion network provided by the present invention includes:

[0006] Fusion steps: The deep and shallow palm print features in the multi-level feature bilinear fusion network are fused to obtain a fusion function, and the fusion function is subjected to multi-receptive field bilinear pooling.

[0007] Lightweighting step: Lightweighting is performed on the multi-level feature bilinear fusion network after multi-receptive field bilinear pooling;

[0008] Encoding position step: Based on the lightweight multi-level feature bilinear fusion network, the coordinate attention mechanism is embedded into the multi-level feature bilinear fusion network model.

[0009] Preferably, the multi-level feature bilinear fusion network comprises two network branches with different receptive fields. One network branch has a convolution dilation rate set to 1 (i.e., ordinary convolution), while the other network branch has a convolution dilation rate set to 2. The shallow and deep palmprint features from these two different receptive field network branches are fused using the concat function. The fusion function formula is as follows:

[0010] f1 = concat(f(A), f(B))

[0011] f2 = concat(f(C), f(D))

[0012] Where A represents the deep features in the first branch; B represents the deep features in the second branch; C represents the shallow features in the first branch; D represents the shallow features in the second branch; f1 represents the deep features after merging two deep features; and f2 represents the shallow features after merging two shallow features.

[0013] The fused shallow and deep palm print features are then subjected to multi-receptor field bilinear pooling, with the specific implementation formula being:

[0014] b(I,f1,f2)=f1 T (I)f2(I)

[0015] x=∑b(I,f1,f2)

[0016]

[0017] z = y / ||y||2

[0018] b(I,f1,f2) represents the matrix obtained by multiplying the two features f1 and f2; f1 and f2 are the features obtained by the above two formulas; x represents the matrix obtained by performing sum pooling on matrix b; y represents the moment normalization of x; z is the result of L2 normalization of feature y.

[0019] Preferably, the multi-level feature bilinear fusion network includes a shallow network and a deep network. The shallow network has a small receptive field and contains palm print details. The deep network has an increased receptive field and extracts the overall contour features of the palm print.

[0020] Preferably, the network lightweighting uses depthwise separable convolutions and changes the dilation rate in the convolutions to reduce computational cost;

[0021] The depthwise separable convolution decomposes ordinary convolution into depthwise convolution and pointwise convolution. The depthwise convolution layer is responsible for convolution calculation, using a single convolution kernel for each input channel; the pointwise convolution is responsible for combining, using a 1×1 convolution kernel to calculate a linear combination of the input channels, combining the depthwise convolution and the output.

[0022] Set the dimensions of the input feature map to W×H×C. I The kernel size is D K ×D K The quantity is C O indivual.

[0023] Preferably, the coordinate attention mechanism encodes horizontal and vertical features using pooling kernels (H,1) and (1,W) on the input; and performs feature transformation using 1×1 convolution, batch normalization, and non-linear activation.

[0024] The coordinate attention mechanism ensures that each weight includes channel information, horizontal spatial information, and vertical spatial information.

[0025] A palmprint recognition system based on a multi-level feature bilinear fusion network includes:

[0026] Fusion module: The deep and shallow palm print features in the multi-level feature bilinear fusion network are fused to obtain a fusion function, and the fusion function is subjected to multi-receptive field bilinear pooling.

[0027] Lightweight module: Performs lightweight processing on the multi-level feature bilinear fusion network after multi-receptive field bilinear pooling;

[0028] Encoding Location Module: Based on the lightweight multi-level feature bilinear fusion network, the coordinate attention mechanism is embedded into the multi-level feature bilinear fusion network model.

[0029] Preferably, the multi-level feature bilinear fusion network comprises two network branches with different receptive fields. One network branch has a convolution dilation rate set to 1 (i.e., ordinary convolution), while the other network branch has a convolution dilation rate set to 2. The shallow and deep palmprint features from the two different receptive field network branches are fused using the concat function, as shown in the following formula:

[0030] f1 = concat(f(A), f(B))

[0031] f2 = concat(f(C), f(D))

[0032] Where A represents the deep features in the first branch; B represents the deep features in the second branch; C represents the shallow features in the first branch; D represents the shallow features in the second branch; f1 represents the deep features after merging two deep features; and f2 represents the shallow features after merging two shallow features.

[0033] The fused shallow and deep palm print features are then subjected to multi-receptor field bilinear pooling, with the specific implementation formula as follows:

[0034] b(I,f1,f2)=f1 T (I)f2(I)

[0035] x=∑b(I,f1,f2)

[0036]

[0037] z = y / ||y||2

[0038] b(I,f1,f2) represents the matrix obtained by multiplying the two features f1 and f2; f1 and f2 are the features obtained by the above two formulas; x represents the matrix obtained by performing sum pooling on matrix b; y represents the moment normalization of x; z is the result of L2 normalization of feature y.

[0039] Preferably, the multi-level feature bilinear fusion network includes a shallow network and a deep network. The shallow network has a small receptive field and contains palm print details. The deep network has an increased receptive field and extracts the overall contour features of the palm print.

[0040] Preferably, the network lightweighting uses depthwise separable convolutions and changes the dilation rate in the convolutions to reduce computational cost;

[0041] The depthwise separable convolution decomposes ordinary convolution into depthwise convolution and pointwise convolution. The depthwise convolution layer is responsible for convolution calculation, using a single convolution kernel for each input channel; the pointwise convolution is responsible for combining, using a 1×1 convolution kernel to calculate a linear combination of the input channels, combining the depthwise convolution and the output.

[0042] Set the dimensions of the input feature map to W×H×C. I The kernel size is D K ×D K The quantity is C O indivual.

[0043] Preferably, the coordinate attention mechanism encodes horizontal and vertical features using pooling kernels (H,1) and (1,W) on the input; and performs feature transformation using 1×1 convolution, batch normalization, and non-linear activation.

[0044] The coordinate attention mechanism ensures that each weight includes channel information, horizontal spatial information, and vertical spatial information.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] 1. This invention improves recognition accuracy by employing a multi-level feature bilinear fusion network to deeply mine the detailed information of palm prints.

[0047] 2. This invention combines depthwise separable convolution and dilated convolution to replace traditional convolution, which greatly reduces the computational complexity and number of parameters of the network model.

[0048] 3. This invention compensates for the shortcomings of lightweight processing by embedding a coordinate attention mechanism, and enhances the capture of location and channel information, thereby further improving network performance. Attached Figure Description

[0049] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0050] Figure 1 This is a schematic diagram of a multi-level feature bilinear fusion network;

[0051] Figure 2 This is a schematic diagram of depthwise convolution;

[0052] Figure 3 This is a schematic diagram of pointwise convolution;

[0053] Figure 4 This is a schematic diagram of the coordinate attention mechanism. Detailed Implementation

[0054] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0055] This invention proposes a multi-level feature bilinear fusion network for palmprint recognition. The multi-level feature bilinear fusion network can extract second-order features with richer details from the palmprint. Then, depthwise separable convolution and dilated convolution are combined to reduce network parameters and training time. Finally, a coordinate attention mechanism is embedded to strengthen the correlation between channel and spatial location information, improving the network's ability to extract palmprint features in both space and channels.

[0056] Differences between palmprint images can be influenced by factors such as pose, angle, and position, thus palmprint image recognition places high demands on visual feature representation capabilities. Typical neural networks can only capture a single-scale receptive field, making accurate recognition of multi-scale targets like palmprints difficult. To address these issues, and considering the small differences between palmprint classes, this invention designs a multi-level feature bilinear fusion network, such as... Figure 1 The network framework diagram.

[0057] (1) Multi-level feature bilinear fusion network

[0058] The network has two identical branches; one branch has a convolution dilation rate set to 1 (i.e., ordinary convolution), and the other branch has a convolution dilation rate set to 2. From... Figure 1 It can be seen that the two branches are completely identical except for their expansion rates. The shallow and deep features of different receptive field network branches are fused using the concat function, as shown in the following formula:

[0059] f1 = concat(f(A), f(B))

[0060] f2 = concat(f(C), f(D))

[0061] The fused features are then subjected to bilinear pooling to obtain higher-order features. The specific formula for multi-receptive-field bilinear pooling is as follows:

[0062] b(I,f1,f2)=f1 T (I)f2(I)

[0063] x=∑b(I,f1,f2)

[0064]

[0065] z = y / ||y||2

[0066] First, the shallow and deep features of the network are merged. The shallow network has a small receptive field and its features contain more palmprint details. The deep network has an increased receptive field and its features extract the overall contour features of the palmprint. The merged features are then multiplied by an outer product to calculate the correlation between the two feature paths, resulting in a second-order feature containing more information. Since fully connected layers do not accept multidimensional tensors, the second-order features are processed into a one-dimensional feature tensor. The resulting tensor is then subjected to a square root operation and L2 normalization to avoid slow computation and overfitting issues in the fully connected layer. Finally, the processed one-dimensional tensor is fed into the fully connected layer for palmprint image classification and prediction.

[0067] (2) Network lightweighting

[0068] Convolutional neural networks (CNNs) generally suffer from a large number of parameters and large model size, making them unsuitable for environments with limited computing resources. Therefore, lightweight network design is essential. This invention uses depthwise separable convolutions and modifies the dilation rate within the convolutions to replace conventional convolutions, thereby reducing computational cost and complexity.

[0069] Depthwise separable convolution decomposes ordinary convolution into depthwise convolution and pointwise convolution, which greatly reduces the number of parameters and computational cost.

[0070] Figure 2 and Figure 3 These are schematic diagrams of depthwise convolution and pointwise convolution, respectively. The depthwise convolution layer is responsible for convolution calculation, using a single convolution kernel for each input channel. The pointwise convolution layer is responsible for combining, using a 1×1 convolution kernel to calculate a linear combination of the input channels, combining the depthwise convolution and the output.

[0071] If the dimensions of the input feature map are W×H×C I The kernel size is D K ×D K The quantity is C O For a regular convolutional layer, the output size is W×H×C. O W, H, C I This refers to the dimension of the input feature map, where x, y, and z are the feature vectors calculated by the network, W×H×C. I After being processed layer by layer, the input feature map of varying sizes yields features x, y, z, namely W, H, and C. I The input dimensions correspond one-to-one with x, y, and z. In this case, the number of parameters in the convolution kernel is D. K ×D K ×C I ×C O The computational complexity is D. K ×D K ×W×H×C I ×C O For depthwise convolution, the number of parameters in its kernel is D. K ×D K ×C I There are 1, and the computational complexity is D. K ×D K ×C I ×W×H. For point convolution, the number of parameters is C. I ×C O One, the computational complexity is W×H×C I ×C O .

[0072]

[0073]

[0074] As can be seen from the above equation, depthwise separable convolution significantly reduces both computational resources and the number of parameters compared to conventional convolution. The convolution kernel size used in this invention is 3×3, thus reducing computational resources and the number of parameters to approximately 1 / 9 of the previous method.

[0075] (3) Coordinate attention mechanism

[0076] Because depthwise convolution and pointwise convolution are computed separately, channel information is not effectively utilized, affecting the transmission of feature information. To compensate for the shortcomings brought about by lightweight design, and to encode more accurate positional information, a coordinate attention mechanism is embedded into the network model.

[0077] Figure 4 This is a block diagram of the coordinate attention mechanism. As shown in the diagram, the coordinate attention mechanism first encodes horizontal and vertical features in the input using pooling kernels (H,1) and (1,W). The basic idea of ​​pooling is to divide the feature map into several sub-regions and perform statistical summaries on each sub-region. The pooling kernel is the size of this region. This captures positional information while preserving spatial information. Then, 1×1 convolution, batch normalization, and non-linear activation are used for feature transformation. Using the coordinate attention mechanism ensures that each weight contains channel information, horizontal spatial information, and vertical spatial information, while considering channel relationships and positional information. This helps the network accurately locate palmprint information with only a small increase in computational complexity, enhancing recognition capabilities.

[0078] Working principle

[0079] First, the multi-level feature bilinear fusion network uses two convolutional paths with different dilation rates to obtain receptive fields of different sizes from the palmprint image. It then merges the extracted deep and shallow palmprint features and uses bilinear pooling to extract second-order features containing richer details from the merged features. Next, the network is lightweighted by combining depthwise separable convolution and dilated convolution, reducing network parameters, training time, and facilitating hardware implementation. Finally, a coordinate attention mechanism is embedded, which not only compensates for the shortcomings of lightweight processing but also strengthens the correlation between channel and spatial location information, improving the network's ability to extract palmprint features in both space and channels.

[0080] The present invention also provides a multi-level feature bilinear fusion network palmprint recognition system. The multi-level feature bilinear fusion network palmprint recognition system can be implemented by executing the process steps of the multi-level feature bilinear fusion network palmprint recognition method. That is, those skilled in the art can understand the palmprint recognition method based on the multi-level feature bilinear fusion network as a preferred embodiment of the multi-level feature bilinear fusion network palmprint recognition system.

[0081] Specifically, a palmprint recognition system based on a multi-level feature bilinear fusion network includes:

[0082] Fusion module: Fuses deep and shallow palm print features extracted from the multi-level feature bilinear fusion network, and then performs multi-receptive field bilinear pooling on them; Lightweight module: Lightweights the multi-level feature bilinear fusion network after multi-receptive field bilinear pooling; Encoding position module: Embeds the coordinate attention mechanism into the network model of the lightweight multi-level feature bilinear fusion network.

[0083] The multi-level feature bilinear fusion network consists of two network branches with different receptive fields. One branch has a convolutional dilation rate of 1, while the other branch has a convolutional dilation rate of 2. The shallow and deep palmprint features from these two branches are fused using the concat function, as shown in the following formula:

[0084] f1 = concat(f(A), f(B))

[0085] f2 = concat(f(C), f(D))

[0086] The fused shallow and deep palm print features are then subjected to multi-receptor field bilinear pooling, with the specific implementation formula as follows:

[0087] b(I,f1,f2)=f1 T (I)f2(I)

[0088] x=∑b(I,f1,f2)

[0089]

[0090] z = y||y||2

[0091] The multi-level feature bilinear fusion network consists of shallow and deep networks. The shallow network has a small receptive field and contains palm print details. The deep network has an increased receptive field and extracts the overall contour features of the palm print.

[0092] The network is lightweight by using depthwise separable convolutions and changing the dilation rate in the convolutions to reduce computational cost. Depthwise separable convolutions decompose ordinary convolutions into depthwise convolutions and pointwise convolutions. The depthwise convolutional layers handle the convolution computation, using a single kernel for each input channel. The pointwise convolutions handle the combination, using 1×1 kernels to compute a linear combination of the input channels, combining the depthwise convolutions and the output. The input feature map dimensions are set to W×H×C. I The kernel size is D K ×D K The quantity is C O indivual.

[0093] The coordinate attention mechanism encodes horizontal and vertical features using pooling kernels (H,1) and (1,W) on the input; it uses 1×1 convolution, batch normalization, and non-linear activation for feature transformation; the coordinate attention mechanism ensures that each weight contains channel information, horizontal spatial information, and vertical spatial information.

[0094] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0095] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A palmprint recognition method based on a multi-level feature bilinear fusion network, characterized in that, include: Fusion steps: The deep and shallow palm print features in the multi-level feature bilinear fusion network are fused to obtain a fusion function, and the fusion function is subjected to multi-receptive field bilinear pooling. Lightweighting step: Lightweighting is performed on the multi-level feature bilinear fusion network after multi-receptive field bilinear pooling; Encoding position steps: Based on the lightweight multi-level feature bilinear fusion network, the coordinate attention mechanism is embedded into the multi-level feature bilinear fusion network model; The multi-level feature bilinear fusion network comprises two network branches with different receptive fields. One branch has a convolution dilation rate of 1 (i.e., ordinary convolution), while the other branch has a convolution dilation rate of 2. The shallow and deep palmprint features from these two different receptive field network branches are fused using the concat function. The fusion function formula is as follows: Where A represents the deep-level features in the first branch; B represents the deep-level features in the second branch; C represents the shallow-level features in the first branch; and D represents the shallow-level features in the second branch. This represents the deep feature resulting from the merging of two deep features. This represents the shallow feature resulting from merging two shallow features.

2. The palmprint recognition method based on a multi-level feature bilinear fusion network according to claim 1, characterized in that, The fused shallow and deep palm print features are then subjected to multi-receptor field bilinear pooling, with the specific implementation formula being as follows: express and The matrix obtained by multiplying two features; and The features are obtained from the fusion function formula; x represents matrix x obtained by performing sum pooling on matrix b; y represents the moment normalization of x; z is obtained by performing L2 normalization on the feature y.

3. The palmprint recognition method based on a multi-level feature bilinear fusion network according to claim 2, characterized in that, The multi-level feature bilinear fusion network includes a shallow network and a deep network. The shallow network has a small receptive field and contains palm print details. The deep network has an increased receptive field and extracts the overall contour features of the palm print.

4. The palmprint recognition method based on a multi-level feature bilinear fusion network according to claim 1, characterized in that, The network lightweighting uses depthwise separable convolutions and changes the dilation rate in the convolutions to reduce computational cost. The depthwise separable convolution decomposes ordinary convolution into depthwise convolution and pointwise convolution. The depthwise convolution layer is responsible for convolution calculation, using a single convolution kernel for each input channel; the pointwise convolution is responsible for combining, using a 1×1 convolution kernel to calculate a linear combination of the input channels, combining the depthwise convolution and the output. Set the dimensions of the input feature map to W×H×C. I The kernel size is D K ×D K The quantity is C O indivual.

5. The palmprint recognition method based on a multi-level feature bilinear fusion network according to claim 1, characterized in that, The coordinate attention mechanism encodes horizontal and vertical features using pooling kernels (H,1) and (1,W) on the input; and performs feature transformation using 1×1 convolution, batch normalization, and non-linear activation. The coordinate attention mechanism ensures that each weight includes channel information, horizontal spatial information, and vertical spatial information.

6. A palmprint recognition system based on a multi-level feature bilinear fusion network, characterized in that, include: Fusion module: The deep and shallow palm print features in the multi-level feature bilinear fusion network are fused to obtain a fusion function, and the fusion function is subjected to multi-receptive field bilinear pooling. Lightweight module: Performs lightweight processing on the multi-level feature bilinear fusion network after multi-receptive field bilinear pooling; Encoding position module: Based on the lightweight multi-level feature bilinear fusion network, the coordinate attention mechanism is embedded into the multi-level feature bilinear fusion network model; The multi-level feature bilinear fusion network comprises two network branches with different receptive fields. One network branch has a convolution dilation rate of 1, i.e., ordinary convolution; the other network branch has a convolution dilation rate of 2. The shallow and deep palm print features from the two different receptive field network branches are fused using the concat function. The fusion function formula is as follows: Where A represents the deep-level features in the first branch; B represents the deep-level features in the second branch; C represents the shallow-level features in the first branch; and D represents the shallow-level features in the second branch. This represents the deep feature resulting from the merging of two deep features. This represents the shallow feature resulting from merging two shallow features.

7. The palmprint recognition system based on a multi-level feature bilinear fusion network according to claim 6, characterized in that, The fused shallow and deep palm print features are then subjected to multi-receptor field bilinear pooling, with the specific implementation formula as follows: express and The matrix obtained by multiplying two features; and The features are obtained from the fusion function formula; x represents matrix x obtained by performing sum pooling on matrix b; y represents the moment normalization of x; z is obtained by performing L2 normalization on the feature y.

8. The palmprint recognition system based on a multi-level feature bilinear fusion network according to claim 7, characterized in that, The multi-level feature bilinear fusion network includes a shallow network and a deep network. The shallow network has a small receptive field and contains palm print details. The deep network has an increased receptive field and extracts the overall contour features of the palm print.

9. The palmprint recognition system based on a multi-level feature bilinear fusion network according to claim 6, characterized in that, The network lightweighting uses depthwise separable convolutions and changes the dilation rate in the convolutions to reduce computational cost. The depthwise separable convolution decomposes ordinary convolution into depthwise convolution and pointwise convolution. The depthwise convolution layer is responsible for convolution calculation, using a single convolution kernel for each input channel; the pointwise convolution is responsible for combining, using a 1×1 convolution kernel to calculate a linear combination of the input channels, combining the depthwise convolution and the output. Set the dimensions of the input feature map to W×H×C. I The kernel size is D K ×D K The quantity is C O indivual.

10. The palmprint recognition system based on a multi-level feature bilinear fusion network according to claim 6, characterized in that, The coordinate attention mechanism encodes horizontal and vertical features using pooling kernels (H,1) and (1,W) on the input; and performs feature transformation using 1×1 convolution, batch normalization, and non-linear activation. The coordinate attention mechanism ensures that each weight includes channel information, horizontal spatial information, and vertical spatial information.

Images