Three-dimensional palmprint region-of-interest extraction method based on CMV3-TFNet lightweight model

Abstract

The invention discloses a three-dimensional palm print region-of-interest extraction method based on a CMV3-TFNet lightweight model. The method comprises the following steps: firstly, acquiring twelve deformed fringe patterns through a four-step phase shift method by using a three-dimensional palm print acquisition system, and generating a two-dimensional palm print depth map; preprocessing the collected palm image data; a CMV3-TFNet model architecture is used to carry out palm print center positioning and rotation angle prediction on a preprocessed image, a polar coordinate transformation network PTN is introduced into a model structure to enhance rotation robustness, depth separable convolution and a lightweight Transform module are used to improve calculation efficiency, and a CBAM module is used to enhance feature extraction capability. And finally, cutting the palmprint image after rotation correction, and extracting a region of interest of the palmprint. According to the method, good performance is obtained on various data sets, and the capability of accurately extracting the palmprint region of interest in a rotating complex scene is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing methods, and in particular to a method for extracting a region of interest of a three-dimensional palmprint based on a CMV3-TFNet lightweight model. Background Art

[0002] With the rapid development of biometric technology, palmprint recognition has gradually become a research hotspot in the field of identity authentication and security due to its unique biometric features and contactless acquisition method. The region of interest (ROI) in a palmprint image usually includes important texture features, including main lines, detail lines, and point features, and its accurate extraction is crucial for identity recognition. However, traditional palmprint recognition algorithms face complex background interference and various noise challenges, especially when the rotation or displacement of the palmprint image changes, it is even more difficult to ensure high accuracy and low missed detection rate.

[0003] To solve these problems, the collection and analysis of 3D palmprint data has gradually attracted attention. 3D palmprint data includes richer depth information and can provide more comprehensive palmprint features than 2D images, thereby improving recognition accuracy. However, due to the scarcity and complexity of 3D palmprint datasets, it is difficult to promote the direct use of 3D palmprint data for training and recognition in practical applications. Summary of the invention

[0004] The technical problem to be solved by the present invention is how to provide a method for improving the feature extraction capability of the model under a rotating complex background, thereby improving the accuracy of extracting the region of interest of the palmprint.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is: a method for extracting regions of interest of three-dimensional palmprints based on a CMV3-TFNet lightweight model, comprising the following steps:

[0006] Collecting three-dimensional palm print texture data of the target palm, and converting the three-dimensional palm print data into a two-dimensional depth map;

[0007] Preprocess the generated two-dimensional depth map;

[0008] Use CMV3-TFNet model to extract palmprint ROI from preprocessed image data;

[0009] Identify and locate the center point of the palm print area to provide a benchmark for subsequent feature extraction;

[0010] Predict the bounding box size of the palmprint area to determine the scope of the region of interest;

[0011] Estimate the rotation angle of the palm print area to achieve rotation correction;

[0012] Predict the offset of the palmprint area for accurate positioning and alignment of the image;

[0013] According to the predicted palmprint center, frame size, rotation angle and offset, the corresponding palmprint ROI area is cropped to complete the extraction of the palmprint feature area.

[0014] The CMV3-TFNet model includes:

[0015] Input layer: The size of the input image is 512×512×3, which is used to receive the RGB image and pass it to the polar coordinate transformation module of the next layer;

[0016] Polar coordinate transformation module PTN: used to perform polar coordinate transformation on the basis of the input image to achieve rotation invariance. The transformed feature map retains the input size (512,512,3) and passes the result to the initial convolution layer;

[0017] Initial convolution layer: This layer uses a 3×3 convolution kernel with a stride of 2, including BatchNorm2d and Hardswish activation functions, and the size of the output feature map is 256×256×16;

[0018] Inverted residual module 1: The input is a feature map of size 256×256×16, using 3×3 depthwise separable convolution, ReLU and 1×1 pointwise convolution, the number of channels is increased to 24, and the size of the output feature map is 128×128×24;

[0019] Inverted residual module 2: The input is a feature map of size 128×128×24, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 40, and the CBAM attention module is embedded, and the size of the output feature map is 64×64×40;

[0020] Inverted residual module 3: The input is a feature map of size 64×64×40, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains 40, and the size of the output feature map is 64×64×40;

[0021] Inverted residual module 4: The input is a feature map of size 64×64×40, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 80, and the CBAM attention module is embedded, and the size of the output feature map is 32×32×80;

[0022] Inverted residual module 5: The input is a feature map of size 32×32×80, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 80, and the size of the output feature map is 32×32×80;

[0023] Inverted residual module 6: The input is a feature map of size 32×32×80, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 160, and the CBAM attention module is embedded, and the size of the output feature map is 16×16×160;

[0024] Inverted residual module 7: The input is a feature map of size 16×16×160, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 160, and the size of the output feature map is 16×16×160;

[0025] Inverted residual module 8: The input is a feature map of size 16×16×160, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 960, and the CBAM attention module is embedded, and the size of the output feature map is 8×8×960;

[0026] Lightweight Transformer module: located in the later stage of feature extraction, used to further enhance the global feature expression, using multi-head self-attention, and the output feature size is 8×8×160;

[0027] Upsampling layer 1: The input is a feature map of size 8×8×960, and a 3×3 deconvolution kernel is used to upsample the feature map to 16×16×256 and output a feature map of size 16×16×256;

[0028] Upsampling layer 2: The input is a feature map of size 16×16×256. A 3×3 deconvolution kernel is used to upsample the feature map to 32×32×128, and a feature map of size 32×32×128 is output.

[0029] Upsampling layer 3: The input is a feature map of size 32×32×128. A 3×3 deconvolution kernel is used to upsample the feature map to 64×64×64, and a feature map of size 64×64×64 is output.

[0030] Detection head: Use multiple convolutional layers to input the deconvolution feature map into different detection head branches, and output the center point position, bounding box, center point offset and rotation angle of the target, including:

[0031] Heatmap Head: Generates a heatmap to detect the center point of an object and outputs a 64×64×1 feature map to represent the heatmap of the detection area.

[0032] Size Head: Regresses the width and height of the object to generate an accurate bounding box and outputs a 64×64×2 feature map to predict the width and height of the target object.

[0033] Offset Head: predicts the offset of the center point of the object and outputs a feature map of size 64×64×2;

[0034] Angle Head: used to regress the rotation angle and output a feature map of size 64×64×1.

[0035] The beneficial effect of adopting the above technical solution is that when constructing the CMV3-TFNet model, the method converts the three-dimensional palmprint data into a two-dimensional depth map for training, and combines the Polar Transformer Network (PTN) module to improve rotation invariance and solve the detection accuracy problem caused by the rotation change of the palmprint image. In addition, the CenterNet backbone is integrated with MobileNetV3 and the CBAM module is introduced to enhance the model's ability to focus on features and improve detection efficiency and accuracy. Finally, by adding a lightweight Transformer module, it can effectively capture global features in complex backgrounds and ensure robustness in diverse environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0037] Figure 1 is a flow chart of the method described in an embodiment of the present invention;

[0038] Figure 2 It is a fringe projection diagram of a three-dimensional palmprint collection system in an embodiment of the present invention;

[0039] Figure 3 is a network structure diagram of CenterNet in an embodiment of the present invention;

[0040] Figure 4 is a deep convolution module diagram in an embodiment of the present invention;

[0041] Figure 5 is a point-by-point convolution module diagram in an embodiment of the present invention;

[0042] Figure 6 CBAM channel and spatial attention module diagram in an embodiment of the present invention;

[0043] Figure 7 : is a model structure diagram of CMV3-TFNet in an embodiment of the present invention;

[0044] Figure 8 This is the IITD palmprint dataset extraction effect in the embodiment of the present invention. Figure 1 ;

[0045] Fig. 9This is the IITD palmprint dataset extraction effect in the embodiment of the present invention. Figure 2 ;

[0046] Fig.10 This is a diagram showing the extraction effect of the Tongji palmprint dataset in an embodiment of the present invention;

[0047] Fig.11 This is a diagram showing the extraction effect of the COEP palmprint dataset in an embodiment of the present invention;

[0048] Fig.12 This is a diagram showing the effect of extracting the REST palmprint dataset in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0051] Overall, such as Figure 1 As shown, the embodiment of the present invention discloses a method for extracting a region of interest of a three-dimensional palmprint based on a CMV3-TFNet lightweight model, and the method comprises the following steps:

[0052] First, if Figure 2 As shown in the figure, four deformed fringe images with the same number of fringe are added together to form a two-dimensional palmprint texture image. Subsequently, a low-pass filter is used to filter the image to remove noise and generate a smooth palmprint image.

[0053] Secondly, the collected two-dimensional palmprint depth images are standardized, including size adjustment, pixel normalization, etc., to ensure the consistency of the data and suitability for subsequent processing steps.

[0054] Finally, the CMV3-TfNet model is used to predict the center position of the palmprint, estimate the rotation angle of the palmprint, and perform rotation correction on the palmprint ROI area. By cropping the corrected image, the ROI of the palmprint region of interest is extracted.

[0055] CenterNet is widely used in various computer vision tasks, such as object detection and pose estimation. CenterNet uses key point detection to determine the location of an object by predicting the center point of the object. This anchor-free method avoids the reliance on anchors in traditional target detection, making the network structure simpler and more efficient.

[0056] The network structure of CenterNet is as follows Figure 3 As shown in the figure, it is mainly divided into two parts: the backbone network and the head network. The backbone network usually uses classic convolutional neural networks such as ResNet or DLA to extract the features of the input image. Through multi-layer convolution operations, the backbone network outputs a feature map, which will be used for subsequent target detection tasks.

[0057] Based on the backbone network, the head network completes the target detection task through three branches:

[0058] Heatmap Head: Generates heatmaps to detect the center point of each object. Offset Head: Predicts the sub-pixel offset of the center point of the object to improve positioning accuracy. Size Head: Regresses the width and height of the object to generate an accurate bounding box.

[0059] Through this structural design, CenterNet can effectively fuse multi-scale information, significantly reducing computational complexity while maintaining detection accuracy. This model performs well in many practical applications, especially in scenarios with high real-time requirements, such as autonomous driving and intelligent monitoring systems.

[0060] During inference, CenterNet locates objects by finding the local maximum (center point) in the heat map, and uses the regressed bounding box size and offset to accurately locate the bounding box of each object. CenterNet provides an efficient and concise object detection method by simplifying the object detection task to a key point detection problem and predicting the size and position of the object through regression. The core idea is to detect the center point of the object through the heat map and combine the regression method to infer the complete bounding box information.

[0061] CMV3-TFNet network:

[0062] In traditional object detection tasks, most algorithms can only detect objects in the horizontal or vertical direction (i.e., axis-aligned bounding boxes). However, in railway security inspection scenarios, the palm may be rotated, which makes it difficult for traditional axis-aligned bounding boxes to accurately extract the ROI of the palm print. Therefore, using a rotated bounding box is particularly important in this scenario, which can capture the key information of the palm more accurately.

[0063] In order to deal with the problem of palm rotation angle, CenterNet was improved by adding an additional rotation angle branch to predict the rotation angle of the target. This branch usually outputs an angle value to represent the rotation angle of the target relative to the horizontal line. By introducing this rotation angle branch, the representation of the bounding box is changed from an ordinary axis-aligned rectangle (horizontal or vertical) to a rotated rectangle. The rotated rectangle can be represented by five parameters: the two coordinates of the center point (x, y), the width (w), the height (h), and the rotation angle (θ). The rotation angle constraint angle regression loss L is added to the loss function of CenterNet. ang , used for regression of rotation angles, usually using L1 loss or Smooth L1 loss.

[0064]

[0065] The final total loss function is:

[0066] L det =L k +λ size L syze +λ off L off +λ ang L ang

[0067] Where L k is the heat map loss, λ size is the bounding box size loss function L size The weight parameter, λ off is the offset regression loss function L off The weight parameter, λ ang is the loss function L for angle regression ang The weight parameter of .

[0068] Heatmap regression: CenterNet uses heatmaps to represent the center point position of an object. Each pixel of the heatmap corresponds to a position in the input image, and the pixel value represents the probability that the position is the center of the object. The heatmap is generated by taking the center point of the object as the center of the Gaussian distribution. A heatmap of a Gaussian distribution is generated, where the center point of each object is at the peak of the distribution. The Gaussian kernel calculation formula is:

[0069]

[0070] Among them, Y xyc The heatmap value at position (x, y) corresponds to the number of categories c. is the center point of the object. p is the standard deviation of object size adaptation.

[0071] Heatmap loss uses weighted mean square error (MSE) or focal loss to train the heatmap to maximize the probability of the center point of the real object. k Expressed as

[0072]

[0073] Offset regression: Since the resolution of the feature map is usually lower than that of the original image, in order to improve positioning accuracy, CenterNet also regresses the sub-pixel offset of each center point in the original image. The introduction of this offset can significantly improve the accuracy of the detection results. Since the heat map output is low-resolution, in order to improve positioning accuracy, CenterNet also compensates for the quantization error by regressing the offset of each center point, and uses L1 loss to regress the offset regression loss L of the object center point. off as follows:

[0074]

[0075] in, is the predicted offset, and p is the actual center point position. To predict the center point position, R is the downsampling multiple of the output feature map.

[0076] Bounding box size regression, for the detected center point, CenterNet uses a regression model to predict the width and height of the object corresponding to the center point. This regression operation is done directly on the feature map, so there is no need for complex region proposals or anchor point adjustments. After determining the center point, CenterNet predicts the width and height of the object through regression, using the width and height loss for bounding box size regression, usually using L1 or L2 loss. Bounding box size regression loss L suze as follows.

[0077]

[0078] Where N is the number of objects in the image, is the predicted bounding box size, S k is the actual bounding box size.

[0079] The rotation change of palmprint images is a common problem in palmprint recognition systems, especially in contactless collection, where the angle and position of the palm may change due to gestures and shooting angles, resulting in inaccurate extracted regions of interest (ROI). In order to enhance the robustness of the model to rotation changes, this application introduces Polar Transformer Network (PTN) at the front end of the model structure as a preprocessing unit to convert the input image to polar coordinate space. PTN achieves rotation invariance through the following steps:

[0080] Coordinate transformation: PTN first transforms the coordinate system of the input palmprint image, mapping the image from the Cartesian coordinate system to the polar coordinate system. In the polar coordinate space, the rotation of the image will mainly affect the radial direction, but will not affect the angular features, so that the local features of the palmprint remain relatively stable.

[0081] Adaptive rotation invariance: The polar coordinate converted image can standardize the different rotation forms of the palm print, ensuring that the palm print image features under different angles and gestures can be more consistently represented. The image feature representation under polar coordinates has higher rotation invariance, thus reducing the difficulty of the subsequent network to process rotation features during the learning process.

[0082] Preprocessing module: PTN is the initial processing unit of the model. It preprocesses the input image before it flows into the backbone network. This not only reduces the computational burden of the backbone network, but also enables the rotation invariant features to be captured and expressed more efficiently in the backbone network. Through this preprocessing module, it can ensure that the subsequent network can extract the region of interest (ROI) of the palmprint more stably.

[0083] The introduction of PTN effectively improves the stability and accuracy of the model in palmprint ROI extraction at different rotation angles, especially in the contactless palmprint database.

[0084] Compared with traditional convolutional neural networks, MobileNetV3 achieves a balance between high precision and low computational complexity by combining automated neural network architecture search (NAS) and manually optimized neural network layer structures. Its core modules include Depthwise Separable Convolution, which can significantly reduce the computational complexity of the model. The inverted residual structure improves computational efficiency by expanding and compressing the number of channels; the SE module (Squeeze-and-Excitation) enhances feature representation capabilities by adaptively assigning weights to each channel. In addition, MobileNetV3-Large also uses two nonlinear activation functions, Swish and ReLU6, which are reasonably selected and used in different network layers to improve the expressiveness and efficiency of the model.

[0085] Traditional convolution usually processes spatial (height×width) and channel information simultaneously through a single convolution kernel. Although this operation can extract complex features, it will generate a large number of parameters and calculations, especially when the number of channels in the input feature map is large, the calculation overhead increases sharply.

[0086] To solve this problem, the depthwise separable convolution decomposes the convolution into two steps: Depthwise Convolution Figure 4 As shown in , each input channel is convolved separately, so that features are extracted only in space without fusion between channels; pointwise convolution is as follows Figure 5 As shown, a 1×1 convolution is used to linearly combine the channels at each position to aggregate the feature information of different channels. This decomposition method significantly reduces the number of parameters and calculations of the convolution layer while retaining the feature extraction capability.

[0087] In the network structure designed in this application, in order to further improve the performance and efficiency of the model, the backbone network is replaced with MobileNetV3, and deep separable convolution is widely used in the convolution operations of each layer in the backbone network MobileNetV3. While maintaining a high feature extraction capability, it significantly reduces the number of parameters and computational cost of the model. Specifically, by adopting a deep separable convolution design in the convolution layer of the backbone network, the model can ensure the extraction of effective palmprint region of interest (ROI) features while being lightweight. This lightweight design greatly improves the practicality of the model in embedded systems and mobile applications.

[0088] In order to further improve the ROI extraction effect of the model in complex backgrounds, this application introduces the CBAM module such as Figure 6 As shown in the figure, the ability to extract palmprint detail features is effectively enhanced. The CBAM module combines channel attention and spatial attention, allowing the model to focus more on important feature areas and significantly improve recognition accuracy.

[0089] The global pooling operation in the CBAM module is combined with spatial attention, which enables the model to consider the combination of global information and local details when processing images, making the network more adept at dealing with multi-scale and multi-complexity tasks.

[0090] The specific structure of the CMV3-TFNet model is as follows Figure 7 As shown, the CMV3-TFNet model includes:

[0091] Input layer: The size of the input image is 512×512×3. The input layer receives the RGB image and passes it to the polar coordinate transformation module of the next layer.

[0092] Polar Transformer Network (PTN): The PTN module performs polar coordinate transformation on the input image to achieve rotation invariance. The transformed feature map retains the input size (512,512,3) and the result is passed to the first convolutional layer.

[0093] Initial convolution layer: This layer uses a 3×3 convolution kernel with a stride of 2, including BatchNorm2d and Hardswish activation functions, and the size of the output feature map is 256×256×16;

[0094] Inverted residual module 1: The input is a feature map of size 256×256×16, using 3×3 depthwise separable convolution, ReLU and 1×1 pointwise convolution, the number of channels is increased to 24, and the size of the output feature map is 128×128×24;

[0095] Inverted residual module 2: The input is a feature map of size 128×128×24, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 40, and the CBAM attention module is embedded, and the size of the output feature map is 64×64×40;

[0096] Inverted residual module 3: The input is a feature map of size 64×64×40, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains 40, and the size of the output feature map is 64×64×40;

[0097] Inverted residual module 4: The input is a feature map of size 64×64×40, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 80, and the CBAM attention module is embedded, and the size of the output feature map is 32×32×80;

[0098] Inverted residual module 5: The input is a feature map of size 32×32×80, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 80, and the size of the output feature map is 32×32×80;

[0099] Inverted residual module 6: The input is a feature map of size 32×32×80, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 160, and the CBAM attention module is embedded, and the size of the output feature map is 16×16×160;

[0100] Inverted residual module 7: The input is a feature map of size 16×16×160, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 160, and the size of the output feature map is 16×16×160;

[0101] Inverted residual module 8: The input is a feature map of size 16×16×160, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 960, and the CBAM attention module is embedded, and the size of the output feature map is 8×8×960;

[0102] Lightweight Transformer module: Located in the later stage of feature extraction, it is used to further enhance the global feature expression. It uses multi-head self-attention and the output feature size is 8×8×160;

[0103] Upsampling layer 1: The input is a feature map of size 8×8×960, and a 3×3 deconvolution kernel is used to upsample the feature map to 16×16×256 and output a feature map of size 16×16×256;

[0104] Upsampling layer 2: The input is a feature map of size 16×16×256. A 3×3 deconvolution kernel is used to upsample the feature map to 32×32×128, and a feature map of size 32×32×128 is output.

[0105] Upsampling layer 3: The input is a feature map of size 32×32×128. A 3×3 deconvolution kernel is used to upsample the feature map to 64×64×64, and a feature map of size 64×64×64 is output.

[0106] Detection head: Use multiple convolutional layers to input the deconvolution feature map into different detection head branches, and output the center point position, bounding box, center point offset and rotation angle of the target, including:

[0107] Heatmap Head: Generates a heatmap to detect the center point of an object and outputs a 64×64×1 feature map to represent the heatmap of the detection area.

[0108] Size Head: Regresses the width and height of the object to generate an accurate bounding box and outputs a 64×64×2 feature map to predict the width and height of the target object.

[0109] Offset Head: predicts the offset of the center point of the object and outputs a feature map of size 64×64×2;

[0110] Angle Head: used to regress the rotation angle and output a feature map of size 64×64×1.

[0111] As shown in Table 1, which is a summary table of the model, the total number of parameters of the model is about 4.94M, and the computational complexity of the model is 0.52GFLOPs. The model size is estimated to be 64.67MB, including: input size: 0.19MB, memory required for forward / backward propagation: 45.63MB, and memory occupied by parameters: 18.85MB.

[0112] Table 1CMV3-TFNet model structure

[0113]

[0114] Experimental test:

[0115] 1) Palmprint Dataset

[0116] This experiment is tested on four contactless palmprint databases, namely IITD, COEP, CASIA and REST palmprint databases.

[0117] IITD contactless palmprint database was developed by Indian Institute of Technology Delhi (IIT Delhi) and is one of the most representative datasets in the field of contactless palmprint recognition. The database includes 2601 palmprint images from 230 individuals. The images are collected by a contactless method, that is, there is no contact between the palm and the camera, mainly by long-distance camera. This acquisition method simulates the scenarios in real applications, such as public security inspection and barrier-free identity authentication. The image acquisition is carried out under natural light, which means that the lighting conditions in the image are variable, which can better simulate the actual application environment. The database includes palmprint images of both left and right hands, and different palm postures are considered during the acquisition process to increase the diversity of the dataset. This dataset is widely used to study contactless palmprint recognition algorithms, especially when dealing with practical problems such as ambient lighting changes, palm posture changes, and background interference, providing valuable data support.

[0118] The COEP palmprint database was created by the Indian Institute of Engineering, Pune, and is mainly used for the research of contactless palmprint recognition. The COEP database includes 1,400 palmprint images from about 350 individuals. During the collection process, a non-contact method is also used, covering different lighting and background conditions, which is very challenging.

[0119] The Tongji palmprint database was developed by Tongji University and includes 12,000 images collected from 300 individuals. The database covers images collected in different time periods. The REST database includes 1,945 images collected from 179 individuals. Table 2 shows the division of the data set used for training and testing.

[0120] Table 2 Division of four data sets

[0121]

[0122] Data preprocessing:

[0123] In the data preprocessing stage, the input palmprint image is first scaled to a fixed size (such as 512x512). This step is implemented by the get_affine_transform function, which generates an affine transformation matrix based on the image center, scale, and rotation angle, and performs an affine transformation on the image through the cv2.warpAffine function. During the affine transformation process, the original aspect ratio of the image is maintained, thereby avoiding feature loss caused by image deformation. In addition, the input image is standardized and the image is normalized using the mean and standard deviation of ImageNet to enhance the model's robustness to lighting changes.

[0124] Then, MobileNetV3 extracts multiple layers of feature maps. Then, these feature maps undergo multiple deconvolution operations. The introduction of deconvolution layers ensures that the network can effectively integrate low-level detail information during upsampling, thereby improving the ability to detect targets. Upsampling is then performed to restore spatial resolution, and finally a prediction map for category, center offset, bounding box regression, and angle is generated.

[0125] In the processing of annotation information, the annotation file is loaded through the COCO API, and the new annotation information is calculated according to the transformation of the image. After the annotation information is affine transformed, it is mapped to the new image coordinate system to ensure the accuracy of the annotation.

[0126] Model Training

[0127] 1) Training experimental environment

[0128] The experiment uses Windows 11 64-bit operating system, Intel(R) Core(TM) i7-12650H CPU, 16 GB RAM, and NVIDIA GeForce RTX 4060 graphics card. The specific environment is shown in Table 3.

[0129] Table 3 Training hardware environment

[0130]

[0131] In this experiment, four public contactless palmprint databases were used for training and validation. All images were uniformly resized to 512×512 pixels and standardized to ensure that the model can process palmprint images of different resolutions and under different conditions.

[0132] The implementation of the model is based on the PyTorch framework, and the entire experimental process is carried out on a server equipped with a GPU. During the training process, the Adam optimizer is used, which can provide stable convergence in complex training processes due to its adaptive learning rate adjustment mechanism. The initial learning rate is set to 0.0001, and is gradually reduced according to the predetermined number of epochs during the training process to ensure that the model can gradually approach the optimal solution. In addition, a training cycle of 150 epochs is set, and the performance of the model on the validation set is evaluated after each epoch to dynamically adjust the learning rate. For the same model, different thresholds will affect the final detection accuracy. Therefore, experiments were conducted with thresholds of 0.8, 0.6, and 0.5, respectively. The test accuracies are 93.1%, 98.3%, and 99%, respectively. Therefore, in this application, a threshold of 0.5 is selected.

[0133] During the training process, a custom loss function CtdetLoss is used. This loss function combines the losses of heatmap, width, height, angle, and regression, and assigns different weights to each of them. These weights are set to balance the contribution of each loss item to the final model, so that the model can both ensure accuracy and improve stability when extracting ROI. In order to further improve the training effect of the model, we designed a custom loss function L det , which combines heatmap loss, bounding box size loss, angle loss, and center offset loss, and ensures the balance of each loss during training through reasonable weight distribution. In this way, the stability and accuracy of the model are effectively improved, enabling it to extract palmprint ROI more accurately. The final training accuracy reached 98.25%, while the test accuracy reached 96.33%.

[0134] Extraction effect: Figure 8-Figure 12 This is the effect diagram of extracting the palmprint region of interest on four non-contact palmprint databases.

[0135] Evaluation indicators: In the object detection network, IOU is a measure of the degree of overlap between the model detection box and the real object box. The IOU value range is 0 to 1, which indicates the degree of overlap between the predicted box and the actual target box. When the IOU is higher than the set threshold, it means that the prediction is accurate, otherwise it means that the prediction is incorrect. Based on practical considerations, the threshold is usually selected between 0.5 and 0.7.

[0136] In this application, the proposed palmprint ROI extraction method not only needs to perform well in the extraction stage, but also needs to prove its effectiveness in subsequent recognition tasks. To this end, two classic palmprint recognition networks: PalmNet and CO3Net were used to verify the extracted ROI. The experiment mainly focuses on indicators such as recognition accuracy, EER (Equal Error Rate), and AUC (Area Under Curve) to comprehensively evaluate the performance of the extracted ROI in practical applications.

[0137] PalmNet is a classic palmprint recognition network that has been widely used in small-scale embedded systems. When identifying the extracted ROI, PalmNet achieved an average accuracy of 99.62% in multiple experiments, and achieved 100% recognition accuracy in some tests. This shows that the ROI extraction method proposed in this application can well preserve the key features in the palmprint image, so that PalmNet can accurately distinguish different palmprint images during recognition.

[0138] CO3Net is a more advanced palmprint recognition network with strong feature extraction and matching capabilities. In the experiment, CO3Net showed extremely high accuracy in identifying the ROI extracted by this application. Especially in terms of EER and AUC indicators, CO3Net achieved an EER of 0.0023% and an AUC of 0.9920%.

[0139] Ablation experiment

[0140] 1) Ablation experiment of backbone network selection

[0141] The original Resnet is used as the backbone network without any module replacement.

[0142] 2) Attention mechanism ablation experiment

[0143] Experiment 1: Use the SE module of the original MobileNetV3.

[0144] Experiment 2: Completely remove the CBAM module and use only the basic MobileNetV3 network.

[0145] Experiment 3: Replace the CBAM module with another type of attention mechanism ECA module.

[0146] As shown in Table 4, the extracted model accuracy is 95.4%, the inference time is 17.6ms, the computational complexity is 0.52GFLOPs, and the total number of parameters is about 4.94M.

[0147] Table 4 Ablation experiment

[0148]

[0149] 3) Impact of palmprint ROI extraction on recognition effect

[0150] As shown in Table 5, the recognition accuracy of PalmNet and CO3Net is tested on the extraction results after removing or changing a certain module to quantify the impact of the module on the actual recognition effect.

[0151] Table 5 Recognition results of ablation experiments

[0152]

[0153]

[0154] 4) Comparison with other methods

[0155] The comparison with other deep learning networks is shown in Table 6:

[0156] Table 6 Ablation experiment recognition results

[0157]

[0158] As shown in Table 6, the final model size is 64.67MB, the computational complexity of the model is 0.52GFLOPs, and the total number of parameters is about 4.94M, including: input size: 0.19MB, memory required for forward / backward propagation: 45.63MB, and memory occupied by parameters: 18.85MB.

[0159] Comparison with other ROI extraction methods

[0160] In order to evaluate the lightweight palmprint ROI extraction method proposed in this application, this application is compared with some classic methods. Each method has its unique advantages, but also has certain limitations.

[0161] Maximum inscribed circle algorithm: The region of interest (ROI) is determined by calculating the maximum inscribed circle in the palm print image, avoiding interference from background noise, with simple operation and fast calculation speed. This method is not robust to changes in the rotation angle and palm posture in the image, and is prone to failure when the background is complex and the palm contour is unclear.

[0162] The Contcode boundary contour encoding method proposed by Wong et al. has good robustness, can cope with certain noise and image deformation, and is suitable for extracting ROI on palmprint images with clear boundaries. It mainly relies on the boundary information of the palmprint image, so it is easy to produce false detection or missed detection when the boundary is fuzzy or the background is complex, and it is insensitive to posture and illumination changes.

[0163] U-Net can accurately segment the palmprint area of ​​interest by combining skip connections with multi-scale features, and performs well in processing complex texture images. However, the network is large in scale, computationally complex, and takes a long time to train and infer, making it unsuitable for use in real-time applications or resource-constrained devices.

[0164] The deep residual network ResNet successfully solves the gradient vanishing problem in deep networks by introducing residual modules, and can better extract complex palmprint features, and performs particularly well on large-scale data sets. However, its computational cost is high, the model parameters are large, and it runs slowly in resource-constrained environments, which cannot meet the needs of real-time applications.

[0165] The method proposed in this application combines the lightweight design of MobileNetV3 and CenterNet, which can effectively deal with the complex background and diverse palm postures of palmprint images, while significantly reducing the amount of calculation and model parameters. It is suitable for mobile devices and embedded devices. In addition, the use of the CBAM module further improves the ability to extract palmprint detail features. Due to the lightweight design of the model, in some specific high-precision tasks, it may not be as capable of extracting details as more complex large models.

[0166] Although each method has its own advantages in different scenarios, the method proposed in this application reduces the computational complexity and resource usage of the model while ensuring detection accuracy, and is more suitable for lightweight requirements in practical applications. The following Table 7 compares it with other extraction methods in terms of accuracy, inference time, computational complexity and model parameter quantity.

[0167] Table 7 Comparison of other extraction methods

[0168]

[0169] Through experimental verification, the method proposed in this application was tested under different lighting conditions and complex backgrounds to evaluate the recognition rate, processing speed and anti-interference ability of the method, and achieved significant improvements in accuracy and speed. The experimental results show that the average accuracy of the model constructed in this application on the test set reached 95.4%, and the inference time on the mobile device was 5.95ms. Compared with other traditional methods, it has higher practicality in resource-constrained environments and has significant advantages in extracting palmprint areas of interest.

Claims

1. A method for extracting regions of interest from three-dimensional palmprints based on a CMV3-TFNet lightweight model is characterized by The steps include: Collecting three-dimensional palm print texture data of the target palm and converting the three-dimensional palm print data into a two-dimensional depth map; Preprocess the generated two-dimensional depth map; Use CMV3-TFNet model to extract palmprint ROI from preprocessed image data; Identify and locate the center point of the palm print area to provide a benchmark for subsequent feature extraction; Predict the bounding box size of the palmprint area to determine the scope of the region of interest; Estimate the rotation angle of the palm print area to achieve rotation correction; Predict the offset of the palmprint area for accurate positioning and alignment of the image; According to the predicted palmprint center, frame size, rotation angle and offset, the corresponding palmprint ROI area is cropped to complete the extraction of the palmprint feature area.

2. The method for extracting regions of interest of three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 1, characterized in that: The method for preprocessing palm image data comprises the following steps: normalizing and resizing the image, and scaling the input palm print image to a fixed size.

3. The method for extracting regions of interest of three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 1, characterized in that: The CMV3-TFNet model includes: Input layer: The size of the input image is 512×512×3, which is used to receive the RGB image and pass it to the polar coordinate transformation module of the next layer; Polar coordinate transformation module PTN: used to perform polar coordinate transformation on the basis of the input image to achieve rotation invariance. The transformed feature map retains the input size (512,512,3) and passes the result to the initial convolution layer; Initial convolution layer: This layer uses a 3×3 convolution kernel with a stride of 2, including BatchNorm2d and Hardswish activation functions, and the size of the output feature map is 256×256×16; Inverted residual module 1: The input is a feature map of size 256×256×16, using 3×3 depthwise separable convolution, ReLU and 1×1 pointwise convolution, the number of channels is increased to 24, and the size of the output feature map is 128×128×24; Inverted residual module 2: The input is a feature map of size 128×128×24, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 40, and the CBAM attention module is embedded, and the size of the output feature map is 64×64×40; Inverted residual module 3: The input is a feature map of size 64×64×40, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains 40, and the size of the output feature map is 64×64×40; Inverted residual module 4: The input is a feature map of size 64×64×40, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 80, and the CBAM attention module is embedded, and the size of the output feature map is 32×32×80; Inverted residual module 5: The input is a feature map of size 32×32×80, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 80, and the size of the output feature map is 32×32×80; Inverted residual module 6: The input is a feature map of size 32×32×80, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 160, and the CBAM attention module is embedded, and the size of the output feature map is 16×16×160; Inverted residual module 7: The input is a feature map of size 16×16×160, using 3×3 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels remains at 160, and the size of the output feature map is 16×16×160; Inverted residual module 8: The input is a feature map of size 16×16×160, using 5×5 depthwise separable convolution, ReLU and 1×1 point-by-point convolution, the number of channels is increased to 960, and the CBAM attention module is embedded, and the size of the output feature map is 8×8×960; Lightweight Transformer module: located in the later stage of feature extraction, used to further enhance the global feature expression, using multi-head self-attention, and the output feature size is 8×8×160; Upsampling layer 1: The input is a feature map of size 8×8×960, and a 3×3 deconvolution kernel is used to upsample the feature map to 16×16×256 and output a feature map of size 16×16×256; Upsampling layer 2: The input is a feature map of size 16×16×256. A 3×3 deconvolution kernel is used to upsample the feature map to 32×32×128, and a feature map of size 32×32×128 is output. Upsampling layer 3: The input is a feature map of size 32×32×128. A 3×3 deconvolution kernel is used to upsample the feature map to 64×64×64, and a feature map of size 64×64×64 is output. Detection head: Use multiple convolutional layers to input the deconvolution feature map into different detection head branches, and output the center point position, bounding box, center point offset and rotation angle of the target, including: Heatmap Head: Generates a heatmap to detect the center point of an object and outputs a 64×64×1 feature map to represent the heatmap of the detection area. Size Head: Regresses the width and height of the object to generate an accurate bounding box and outputs a 64×64×2 feature map to predict the width and height of the target object. Offset Head: predicts the offset of the center point of the object and outputs a feature map of size 64×64×2; Angle Head: used to regress the rotation angle and output a feature map of size 64×64×1.

4. The method for extracting regions of interest of three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 1, characterized in that: The total loss function of the CMV3-TFNet model is: Ldet=Lk+λsizeLsize+λoffLoff+λangLang Where Lk is the heat map loss, λsize is the weight parameter of the bounding box size loss function Lsize, λoff is the weight parameter of the offset regression loss function Loff, and λang is the weight parameter of the angle regression loss function Lang.

5. The method for extracting regions of interest of three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 4, characterized in that: The heat map loss Lk is calculated by the following method: CenterNet uses heat maps to represent the center point position of an object. Each pixel of the heat map corresponds to a position in the input image, and the pixel value represents the probability that the position is the center of the object. The heat map is generated by taking the center point of the object as the center of the Gaussian distribution to generate a heat map of a Gaussian distribution, where the center point of each object is located at the peak of the distribution. The Gaussian kernel calculation formula is: Among them, Yxyc is the heat map value at the position (x, y), corresponding to the number of categories c; is the center point position of the object, σp is the standard deviation of the object size adaptation; The heat map loss uses weighted mean square error MSE or focal loss FocalLoss to train the heat map to maximize the probability of the center point of the real object. The heat map loss Lk is expressed as:

6. The method for extracting regions of interest from three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 4, characterized in that: The offset regression loss Loff of the object center point is regressed using the L1 loss as follows: in is the predicted offset, and p is the actual center point position. To predict the center point position, R is the downsampling multiple of the output feature map.

7. The method for extracting regions of interest from three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 4, characterized in that: The bounding box size regression loss Lsize is as follows: Where N is the number of objects in the image, is the predicted bounding box size, and Sk is the actual bounding box size.

8. The method for extracting regions of interest of three-dimensional palmprints based on the CMV3-TFNet lightweight model as claimed in claim 4, characterized in that: The constrained angle regression loss Lang is used for the regression of the rotation angle. The calculation formula is as follows: Among them, pi·mi and ti·mi respectively represent the predicted angle and the result of multiplying the predicted angle and the true angle with their corresponding valid mask mi; pi represents the predicted angle of the i-th sample; ti represents the true angle of the i-th sample; mi represents the mask identifier of the i-th sample, which is used to determine which sample points are valid in the loss calculation.

Images