A robust palm region of interest positioning method in natural scenes

Abstract

The application discloses a robust palm region of interest positioning method in a natural scene, and comprises the following steps: acquiring an input picture, detecting a palm region and a palm contour region in the input picture, and obtaining a palm region and contour region feature map; inputting the feature map into a palm region of interest positioning network model, performing regression on the feature map through a positioning network in the model, obtaining a posture correction parameter, performing spatial change on the palm region feature map, performing down-sampling and finger root amplification on the image, and generating an adjusted feature map; performing key point coordinate fusion regression on the adjusted feature map, obtaining a first key point coordinate, inversely transforming the coordinate into a coordinate system of the original input picture, obtaining a second key point coordinate; and extracting a palm region of interest image based on the second key point coordinate. The application does not need to remove the background through infrared imaging, can process problems such as translation and rotation in a large range, and realizes the non-contact palm region of interest positioning demand in a natural scene.

Description

Technical Field

[0001] This invention relates to the field of image processing, and more particularly to a robust method, apparatus, feature extraction and recognition device for locating the region of interest of a hand in natural scenes. Background Technology

[0002] Palmprint recognition technology can automatically identify a user based on the biometric features of the human palm. Palmprint recognition includes the following parts: palm image acquisition, palmprint region of interest (ROI) localization, palmprint feature extraction, feature matching, and recognition. For ROI localization, the background is often complex and the palm posture varies considerably; therefore, ROI localization has become a bottleneck for the recognition system.

[0003] However, current palmprint region of interest localization solutions rely heavily on infrared imaging to remove the background and cannot handle large-scale palm rotation, translation, and scaling. In other words, they cannot meet the non-contact palm region of interest localization requirements in natural scenes, which is inconvenient for users. This is a problem that urgently needs to be solved.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The technical problem this application aims to solve is to provide a robust method, apparatus, and medium for locating the region of interest (ROI) of a hand in natural scenes, addressing the shortcomings of existing technologies. This invention can obtain a feature map of the relevant region of the hand through a deep network, distinguish between the hand background image and the feature map of the relevant region of the hand, and simultaneously perform pose correction on the obtained feature map and obtain the coordinates of key points in the hand region through coordinate regression. Thus, the high-definition ROI image of the hand can be extracted from the coordinates, making this invention robust to complex backgrounds and natural hand poses, and able to successfully locate the ROI from hand images taken by different devices in natural scenes.

[0006] To address the shortcomings of the existing technology, the first aspect of this application provides a robust method for locating the region of interest (ROI) of a hand in a natural scene, the method comprising:

[0007] The process involves acquiring an input image of a hand, detecting the hand region and the hand contour region in the input image, and obtaining feature maps of the hand region and the contour region.

[0008] The palm region and contour region feature maps are input into a palm region of interest localization network model trained by a weakly supervised training strategy. The localization network of the trained palm region of interest localization network model regresses the palm region and contour region feature maps to obtain the palm pose correction parameters. Based on the pose correction parameters, the palm region feature map is spatially transformed, and the image is downsampled and the finger root is magnified to generate an adjusted feature map.

[0009] The adjusted feature map is fused and regressed to obtain the first key point coordinates of the fused regression. These coordinates are then inversely transformed back to the coordinate system of the original input image based on the aforementioned palm pose correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image.

[0010] Based on the obtained coordinates of the second key point, the region of interest image of the palm is extracted.

[0011] The training steps of the weakly supervised training strategy for training the palm region of interest localization network model include: using a weak palm region segmenter, a binary classifier trained from a set of monochrome background palm images is used to classify the pixels in the input first palm image, extract the palm region, and perform edge detection on the palm region to obtain the palm contour.

[0012] The palm contour is processed by an iterative palm region of interest localization algorithm to obtain the first annotation information of the input first palm image;

[0013] Based on the palm region of the first palm image, palm pixels are extracted, and the palm pixels are combined with natural background images from the natural image library to synthesize the background and generate a second palm image in a natural environment.

[0014] The second palm image and the first annotation information obtained above are simultaneously subjected to the same random spatial transformation operation to obtain the third palm image and the second annotation information. Then, the image quality of the third palm image is randomly perturbed to obtain the fourth palm image. The resulting data augmented training samples contain the fourth palm image and the second annotation information.

[0015] The obtained training samples are used to train the palm region of interest localization network model.

[0016] The iterative palm region of interest localization algorithm specifically includes:

[0017] The extracted palm region is subjected to key point detection, and the number of key points is verified.

[0018] Based on the number of verified key points, when all 5 fingertip points and 4 finger valley points are detected, the palm region of interest localization method based on the distance extreme points is executed;

[0019] If the verification does not obtain all 5 fingertip points and 4 finger valley points, then perform finger edge detection based on line scanning; if 4 fingertip points and 3 finger valley points are detected, then execute the palm region of interest localization method based on line scanning.

[0020] If no four fingertips and three finger valleys are detected, move to the next scanning point and perform line-scan-based finger edge detection. The detection is then used to determine whether four fingertips and three finger valleys have been detected. If four fingertips and three finger valleys are detected, the palm region of interest localization method based on line scanning is executed. If no four fingertips and three finger valleys are detected, move to the next scanning position and perform line-scan-based finger edge detection again.

[0021] The detection will exit when the scan triggers the stop condition.

[0022] The step of acquiring the input image and detecting the palm region and palm contour region in the input image to obtain the feature maps of the palm region and contour region specifically includes:

[0023] Primary features are extracted from images using a backbone network with a multi-scale pyramid structure.

[0024] Based on the acquired primary features, a semantic segmentation network is used to process the primary features to generate a background region map, a palm region map, and a palm contour region. Figure 3 The three-channel outputs are region feature maps of each channel, and their semantic segmentation performance is supervised by focus loss.

[0025] The generated three-channel region feature map is preprocessed to generate a feature map containing six channels.

[0026] The iterative palm region of interest localization algorithm and the line-scan-based finger edge detection further include:

[0027] The iterative palm region of interest localization algorithm obtains the palm region of interest through a localization method and then performs anomaly detection on the palm region of interest.

[0028] The line-scan-based finger edge detection vertically detects the brightness value of the palm region image from top to bottom in the input palm region. When a complete change pattern is detected, the normal four-finger edge is detected. If no complete change pattern is detected, the vertical scan fails, and the device moves one step to the right to continue the next vertical scan.

[0029] When the scan exceeds the preset scan area, the scan termination condition is triggered, and the scan detection is exited.

[0030] The process of fusing and regressing the key point coordinates of the adjusted feature map specifically includes:

[0031] The key point features are perceived and extracted by a multi-head attention network, and then the coordinates of the key points are globally regressed by a multilayer perceptron to obtain the key point coordinates predicted by the global regression.

[0032] The adjusted feature map is processed by a Vision Transformer network for attention perception and feature extraction, then local key points are detected by a CNN convolutional network, and finally the key point response is mapped to the local regression key point coordinates by a differentiable space-to-numerical transformation.

[0033] The keypoint coordinates obtained from global regression and local regression are fused using the fusion weights predicted from the current sample features to obtain the first keypoint coordinates of the fused regression.

[0034] The finger root edge distance loss is calculated, and the finger root edge distance loss is defined as follows: d i It is the Euclidean distance sequence between the edge of the i-th finger root and the key point of the i-th finger root, L edge This refers to the loss due to the distance from the root edge.

[0035] The step of extracting the region of interest image of the palm based on the obtained second key point coordinates specifically includes:

[0036] The training samples are judged based on a pre-set threshold;

[0037] When the sample size reaches the threshold, the coordinates of the corner points of the region of interest on the palm are output using the coordinates of the second keypoint.

[0038] When the sample size does not reach the threshold, the coordinates of the center point of the palm region of interest and the key point of the finger root are used to establish a coordinate system and locate the square palm region of interest.

[0039] Perform a projective transformation on the obtained corner coordinates of the region of interest of the palm to extract the image of the region of interest of the palm.

[0040] A second aspect of this application provides a robust palm region of interest localization device for natural scenes, the robust palm region of interest localization device for natural scenes comprising:

[0041] The palm region and contour extraction module is used to acquire the input palm image, detect the palm region and palm contour region in the input image, and obtain the feature maps of the palm region and contour region.

[0042] The hand pose adjustment module is used to input the feature maps of the hand region and the contour region into the hand region of interest localization network model trained by a weakly supervised training strategy. The localization network of the hand region of interest localization network model trained by the weakly supervised training strategy regresses the feature maps of the hand region and the contour region to obtain the hand pose correction parameters. Based on the pose correction parameters, the palm region feature map is spatially transformed, and the image is downsampled and the finger root is magnified to generate the adjusted feature map.

[0043] The palm key point coordinate fusion and regression module is used to perform key point coordinate fusion and regression on the adjusted feature map to obtain the first key point coordinates of the fused regression, and then inversely transforms these coordinates back to the coordinate system of the original input image based on the above palm posture correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image.

[0044] The palm region of interest image extraction module is used to extract the palm region of interest image based on the obtained second keypoint coordinates.

[0045] A third aspect of this application provides a feature extraction and recognition device, characterized in that the feature extractor performs palm biometric detection on palm biometric images obtained by the robust palm region of interest localization method in natural scenes as described above, and realizes user identity recognition and verification based on the detected palm biometric information.

[0046] A fourth aspect of this application provides a terminal device, including a memory, a processor, and a robust palm region of interest localization program for natural scenes stored in the memory and executable on the processor, to implement the steps in the robust palm region of interest localization method for natural scenes as described above.

[0047] Beneficial Effects: Compared with existing technologies, this application provides a robust method, apparatus, feature extraction and recognition device for locating the region of interest (ROI) of a hand in natural scenes. The method includes acquiring an input hand image, detecting the hand region and hand contour region in the input image to obtain feature maps of the hand region and contour region; inputting the hand region and contour region feature maps into a trained hand ROI localization network model, regressing the hand region and contour region feature maps through the localization network of the trained hand ROI localization network model to obtain hand posture correction parameters; performing spatial transformation on the hand region feature maps based on the posture correction parameters, and performing downsampling and finger root magnification operations on the image to generate an adjusted feature map; performing key point coordinate fusion regression on the adjusted feature map to obtain the first key point coordinates of the fused regression, and inversely transforming these coordinates back to the coordinate system of the original input image based on the aforementioned hand posture correction parameters to obtain the second key point coordinates of the hand ROI corresponding to the input hand image; and extracting the hand ROI image based on the obtained second key point coordinates. This invention enables the differentiation of palm region and palm contour region feature maps through a semantic segmentation network. The resulting image feature maps undergo pose adjustment and correction to normalize different palm poses, providing high-quality, pose-consistent input samples for subsequent network modules. Then, the pose-corrected image feature maps undergo coordinate-weighted regression based on global and local information, fusing global and local information to achieve robustness in coordinate regression. Finally, a projective transformation is performed on the coordinates to obtain the palm region of interest image, enabling the processing of palm images with complex backgrounds without the need for an infrared camera. Furthermore, during feature extraction, the network focuses on the finger root edge region, deriving the finger root edge distance loss for keypoint regression, providing high-quality feature input for subsequent coordinate regression and improving regression accuracy and robustness. In addition, this invention improves the training of palm region of interest localization. In the network modeling process, an iterative palm region of interest (ROI) localization algorithm is employed, which incorporates an anomaly detection module and utilizes both distance extreme point detection-based and line scan-based palm ROI localization methods. This approach not only eliminates erroneous palm ROI localizations but also improves the overall success rate of palm ROI localization through iterative search. After obtaining calibration information using the iterative palm ROI localization algorithm, this invention employs a natural scene palm dataset augmentation method. This involves synthesizing a background image from the original image and then augmenting the resulting image, thereby providing more realistic and richer training samples for the localization network and further enhancing its generalization ability. Simultaneously, this invention performs palm biometric detection on the palm ROI images obtained using the robust palm ROI localization method in natural scenes, enabling high-precision user identification and verification. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 A flowchart illustrating the robust region of interest localization method for a hand in natural scenes provided by the present invention;

[0050] Figure 2 A reference diagram showing the configuration of the palm region of interest localization network structure provided in an embodiment of the present invention;

[0051] Figure 3 A logical block diagram of the palm key point coordinate regression network provided in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram of training sample region labels and key point annotations provided in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of key point markings on the palm and location of the region of interest on the palm provided in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of edge distance loss provided in an embodiment of the present invention;

[0055] Figure 7 A block diagram of a weakly supervised training strategy provided in an embodiment of the present invention;

[0056] Figure 8 A flowchart of the iterative palm region of interest localization algorithm provided in an embodiment of the present invention;

[0057] Figure 9 Schematic diagrams of two traditional methods for locating regions of interest in the palm provided in embodiments of the present invention;

[0058] Figure 10 The image of the region of interest of the palm is obtained by extracting the key point coordinates based on the obtained fusion regression, as provided in the embodiment of the present invention.

[0059] Figure 11 The annotation effect diagrams automatically generated on different datasets by the iterative palm region of interest localization algorithm provided in the embodiments of the present invention;

[0060] Figure 12 A schematic diagram illustrating the data augmentation process provided in this embodiment of the invention;

[0061] Figure 13 This is a diagram illustrating spatial changes and edge sensing effects provided in an embodiment of the present invention.

[0062] Figure 14 This is a diagram showing the localization effect of the network model provided in this embodiment of the invention on a natural hand image;

[0063] Figure 15 A schematic diagram of a robust palm region of interest localization device in natural scenes provided in an embodiment of the present invention. Detailed Implementation

[0064] This application provides a robust method, apparatus, and medium for locating regions of interest (ROIs) of a hand in natural scenes. To make the objectives, technical solutions, and effects of this application clearer and more explicit, the following detailed description is provided with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit the scope of this application.

[0065] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0066] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0067] Furthermore, if the embodiments of this invention involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by this invention.

[0068] Palmprint recognition technology can automatically identify users based on the biometric features of the human palm surface (such as epidermal ridges, superficial textures, and subcutaneous vein distribution), offering advantages such as fast verification speed and objective accuracy. Currently, electronic sentinel systems based on biometric recognition technology have a promising development opportunity. For existing systems, facial recognition poses privacy and security risks, while fingerprint recognition, being a contact-based method, carries the risk of disease transmission. In contrast, contactless palmprint recognition technology offers advantages such as natural interaction, low privacy sensitivity, rich features, high recognition rate, and low risk of disease transmission, thus gradually becoming a popular research and industrialization direction in biometric recognition.

[0069] Palmprint recognition involves several steps: palm image acquisition, region of interest (ROI) localization (ROI as referred to below), palmprint feature extraction, feature matching, and recognition. For non-contact image acquisition, the background is often complex, and hand poses vary considerably. Therefore, ROI localization becomes a bottleneck for the recognition system. To enhance the robustness of palmprint recognition to environmental conditions, current commercial systems often employ infrared cameras combined with infrared light sources for imaging to remove the background. After obtaining the infrared image of the palm, a series of digital image processing methods (such as adaptive binary segmentation, palm edge root trace, and keypoint extraction) can be used to locate and extract the ROI. Such solutions require specially customized hardware. However, the above methods also have many problems. First, the hardware cost is high; second, the systems of different manufacturers are not interchangeable, which limits the popularization and promotion of palmprint recognition applications; third, for various internet-based applications (such as internet finance, online government services, etc.), the identity verification function of their login pages can only rely on the mobile phone camera; therefore, the need to use mobile phones to capture palm images in natural scenes and perform identity recognition is becoming increasingly urgent. However, existing ROI location methods cannot well meet the current needs. Currently, there are two types of existing ROI location methods. The first type is the traditional method, which is based on a monochrome background palm image (such as infrared-assisted imaging, or artificially arranged monochrome background) to segment the palm region. After obtaining the palm outline, the following different methods can be used to locate the ROI:

[0070] Method 1: A line-scan-based method for locating the region of interest (ROI) of the palm. This method detects finger edges and determines the location of key points at the finger roots by detecting the tangents at the finger root edges, thereby establishing a palm coordinate system and locating the ROI. However, this method has strict requirements on palm posture and is applicable only in limited scenarios. This type of method is designed for palm ROI localization problems under constrained environments. The determination of the line scan position and direction, as well as the selection of palm root reference points, are based on fixed palm posture and position. These are fixed during system design and used as rules to constrain the user's hand placement. Therefore, this type of method is highly sensitive to palm rotation, translation, scaling, and background complexity; even slight deviations will render it unusable.

[0071] Method 2: Palm Region of Interest (ROI) localization method based on distance extrema. First, a reference point at the base of the palm is determined; the distances between this reference point and points in a counter-clockwise order along the palm edge are calculated; key points at the fingertips and bases are detected by analyzing local maxima and minima of distance changes; the results are then denoised and refined through post-processing to obtain more accurate key point coordinates; after obtaining the key points at the bases, a local coordinate system for the palm can be established and the ROI located, similar to Method 1. However, this method is unstable and susceptible to noise. Traditional methods detect key point positions based on local extrema (such as inflection points of local direction changes and inflection points of local distance changes). Due to the lack of global information, this type of method is easily affected by noise in binarization segmentation (such as background interference, sleeves, accessories, uneven or excessively dark lighting). Therefore, this type of method is not robust in real-world applications.

[0072] Method 3: Similar to the two methods above, this method also establishes rectangular ROI regions based on finger root keypoints. The difference is that the ROI region intersects with the palm contour; that is, it searches for the largest rectangular ROI within the palm region that passes through the finger root keypoints. This method processes the four finger root keypoints (concave points) based on distance thresholds, extracting the keypoints corresponding to the index-middle finger and ring-little finger, and applying the same algorithm to both hands. However, this rule-based method assumes an overly ideal palm shape. In actual processing, some users may have their thumb finger root keypoint too close to the finger root keypoints of the other four fingers. In this case, the rule-based method cannot effectively distinguish the order of the finger root keypoints and may locate the wrong ROI.

[0073] The second category is methods based on deep convolutional networks. These methods often directly detect and regress key points of the ROI or the contour points of the palm, and then extract the ROI image based on the key points. However, such methods are prone to overfitting. The palm contour points, especially those on the left and right sides and at the base of the palm, are easily affected by finger posture, sleeves, and ornaments. Therefore, the positioning accuracy and robustness of such methods heavily depend on the size of the training samples. With limited training data, it is difficult to learn the true model parameters, easily leading to overfitting and convergence to individual poses or interference, resulting in poor performance in real-world scenarios. In addition, current solutions often require the five fingers to be spread apart, limiting the application scenarios of palmprint recognition. At the same time, the method has a limited field of view, and robustness and accuracy cannot be balanced. Coordinate regression based on the global field of view does not have spatial invariance, and it often fails to correctly regress coordinates after the samples are translated, rotated, or scaled. Numerical coordinate mapping network models based on local responses have spatial invariance, but lack a global field of view, are susceptible to noise interference, and their positioning accuracy is easily affected. Furthermore, the manual annotation of data in this method is labor-intensive, and manual operation lacks objective standards, which can easily introduce random errors when fatigued.

[0074] As can be seen from the above, many existing technologies rely on infrared imaging to remove backgrounds and cannot handle issues such as large-scale hand rotation, translation, scaling, and finger closure. They also cannot meet the requirements for non-contact hand ROI positioning in natural scenes.

[0075] To address the problems in existing technologies, this invention provides a robust palm region of interest localization method in natural scenes. This method can be executed by a robust palm region of interest localization device in natural scenes. The device can be implemented in software or hardware and can be applied to smart terminal devices with operating systems, such as tablets and computers. In the embodiments of the invention, after acquiring an input palm image, a deep network is used to detect the palm region and palm edge region, generating palm region and palm contour region segmentation feature maps. Palm region segmentation is achieved without the assistance of infrared imaging. Then, for these obtained palm-related region feature maps, a localization network is used to detect the palm posture, and the palm region feature maps are standardized to reduce the influence of rotation and translation. Simultaneously, the interdigital and finger root regions are magnified to ensure the accuracy of subsequent coordinate regression. Subsequently, key point coordinate fusion regression is performed on the adjusted feature maps. The fused regression coordinates obtained through global and local regression possess both robustness (spatial invariance) and regression accuracy. Finally, based on the obtained key point coordinates, the hand contour is extracted. The system generates a palm region of interest (ROI) image. Furthermore, it trains a robust palm ROI localization method for natural scenes by using a weak classifier to extract the palm region from a standard palm image and generate the palm edge region. Then, it detects finger roots and key points of the palm ROI using a traditional palm ROI localization method to obtain the annotation information required for training the network. Finally, it combines image synthesis technology to generate simulated palm images with complex backgrounds, and uses random spatial geometric transformations and image quality perturbations to generate rich training samples. This allows the palm ROI localization network model to utilize existing public palmprint databases combined with data augmentation techniques for fully supervised network training, avoiding the arduous work of manual annotation and enhancing the network's generalization ability.

[0076] Instance methods

[0077] This invention provides a robust method for locating the region of interest (ROI) of a hand in natural scenes, which can be applied to smart terminal devices.

[0078] In a specific implementation, the method described in this invention can be based on a trained palm region of interest localization network model, wherein, as shown in the example... Figure 2As shown, the palm region of interest localization network model includes a backbone network, a palm region and contour extraction module, a palm pose adjustment module, a palm key point coordinate fusion and regression module, and a palm region of interest image extraction module. The specific configuration process of the palm region of interest localization network structure is as follows: A palm image is obtained as input to the network; primary features of the palm image are extracted through the backbone network (①); the extracted primary features are input to the palm region and contour extraction module, which specifically includes a semantic segmentation module and a channel-to-channel softmax operation. The semantic segmentation module (②) extracts the palm region and contour region, and the extracted feature map (BRE) is processed... The six-channel feature map is obtained by performing a pixel-based softmax operation (③) on the map. The obtained feature map is input into the palm pose adjustment module, which includes a localization network for palm pose adjustment and adaptive scaling through deformation and downsampling. The localization network (④) performs spatial transformation adjustment on the image pose and performs downsampling and image deformation (enlargement of the finger root region) to achieve finger pose adjustment in the image (⑤) to obtain the feature map after palm pose adjustment, namely the BRE attention feature map. The BRE attention feature map is input into the palm key point coordinate fusion regression module. The key point coordinate regression module performs key point coordinate regression based on local information (⑥) and key point coordinate regression based on global information (⑦). After fusion regression (⑧) of the obtained coordinates, the coordinates of the palm region of interest are obtained through coordinate transformation (⑨). Finally, the target image is obtained through the finger region of interest image acquisition module. In addition, loss calculation is performed on the feature map after palm region segmentation in the whole network process. Various losses such as the mean square error (MSE) loss of the obtained coordinate regression and the minimum distance of the edge are calculated to obtain the loss term and optimize the network.

[0079] like Figure 1 As shown, this embodiment of the invention provides a robust method for locating the region of interest (ROI) of a hand in a natural scene, comprising the following steps:

[0080] Step S10: Obtain the input palm image, detect the palm region and palm contour region in the input image, and obtain the feature maps of the palm region and contour region.

[0081] In this embodiment of the invention, when the method is implemented, after acquiring the input image, the palm region and palm contour region in the input image are detected through a series of networks to obtain the feature maps of the palm region and contour region.

[0082] Furthermore, the step of acquiring the input image and detecting the palm region and palm contour region in the input image to obtain the feature maps of the palm region and contour region specifically includes:

[0083] Primary features are extracted from images using a backbone network with a multi-scale pyramid structure.

[0084] Based on the acquired primary features, a semantic segmentation network is used to process the primary features to generate a background region map, a palm region map, and a palm contour region. Figure 3 The three-channel outputs are region feature maps of each channel, and their semantic segmentation performance is supervised by focus loss.

[0085] The generated three-channel region feature map is preprocessed to generate a feature map containing six channels.

[0086] Specifically, after obtaining the input image, various commonly used backbone network models with multi-scale pyramid structures, such as HRFormer, are used to extract primary features. This aims to preserve as much detail as possible about the hand contour, improve the accuracy of keypoint coordinates, and prepare for subsequent semantic segmentation and keypoint coordinate regression. The primary features extracted by the backbone network model contain a total of 224 channels. In addition, to improve network speed, lightweight backbone network models, such as ResNet, can also be used to extract primary features.

[0087] After the backbone model obtains the primary features, a semantic segmentation network is used to further process these features. The resulting primary features with 224 channels are then used to output region feature maps through semantic segmentation. Here, the region feature map contains three channels, corresponding to the background region map, the palm region map, and the palm outline region map, respectively.

[0088] The generated 3-channel region feature map is preprocessed into a 6-channel feature map. The specific preprocessing method is as follows: perform a pixel-based softmax operation between channels on the original input 3-channel region feature map to generate a new 3-channel feature map. This operation stretches the region category (similar to binarization stretching) to determine whether the current position belongs to the background, the inside of the palm, or the palm outline. Then, the original 3-channel region feature map and the newly generated 3-channel region feature map are concatenated to form the preprocessed region feature map, which has a total of 6 channels.

[0089] The robust hand region of interest localization method operates across the entire natural scene, using classification loss to supervise the operation of each pixel location in the region feature map. A pixel location corresponds to three categories: background pixel, hand region pixel, and hand edge pixel; the category labels are 0, 1, and 2, respectively. The three channels of the feature map output represent the probabilities of the current pixel location belonging to the three categories. The pixel's three-channel output is processed by a focal loss (hereinafter referred to as focal loss), which primarily focuses on the error of hard samples in the training samples and supervises the semantic segmentation effect of the pixel. This segmentation loss is denoted as L. seg The segmentation loss consists of the classification loss for each pixel in the region feature map; for each pixel, the classification loss is: Among them, L pix (t) represents the classification loss of the corresponding region feature map, where W and H are the width and height of the palm region feature map, respectively; C is the total number of channels in the region feature map, here C = 3; we define three region categories: background region (categorical 0), palm region (categorical 1), and palm edge region (categorical 2), corresponding to the three feature channels of the feature map; c is the index of the feature map channel; i is the index of the pixel in the feature map; γ is the hyperparameter defined in focal loss, with a value of 2.0; y represents the true class label of pixel i in feature channel c, expressed as a one-hot vector; t represents the region feature map; p t (c,i) represents the class prediction probability value of pixel i in feature map channel c, and its specific calculation method is as follows: This transformation is a softmax operation between feature channels, i.e., p t (c,i) represents the probability that pixel i in channel c of the region feature map belongs to category c, where c∈{0,1,2}; the segmentation loss L seg It consists of a weighted average of pixel classification losses from the region feature maps. Different network structures exhibit different specific representations. When only one set of palm region feature maps (denoted as BRE) is used, then L... seg =L pix (BRE) When there are multiple sets of palm region feature maps, such as when a coarse segmentation feature map of the palm region is first predicted (denoted as S), and then S is further refined to obtain a new palm region feature map (denoted as BRE), if both region feature maps are included in the loss for supervision, then... Where t∈{S,BRE}.

[0090] Step S20: Input the palm region and contour region feature maps into the trained palm region of interest localization network model. Regress the palm region and contour region feature maps through the localization network of the palm region of interest localization network model trained by the weak supervision training strategy to obtain the palm pose correction parameters. Based on the pose correction parameters, perform spatial transformation on the palm region feature map, and perform downsampling and finger root magnification operations on the image to generate the adjusted feature map.

[0091] In this embodiment of the invention, the palm region and contour region feature maps are regressed by a localization network to obtain the palm posture correction parameters. Then, based on the posture correction parameters, the palm region feature map is spatially transformed, and the image is downsampled and the finger root is magnified to generate an adjusted feature map.

[0092] Specifically, after obtaining the feature maps of the palm and contour regions, we can regress the coordinates of the finger roots and ROI keypoints. Because the palm's posture is quite flexible during non-contact palmprint acquisition, tilting, rotation, translation, and being too close or too far away are common occurrences. To ensure output accuracy, we adjust the palm's posture before coordinate regression. Therefore, a localization network is used to regress the palm's posture correction parameter theta. This localization network structure consists of three convolutional layers followed by two fully connected layers, outputting theta. The localization network simultaneously senses the geometric posture of the palm and the geometric scale of the finger root region to regress the palm posture correction parameter theta. After obtaining the palm posture correction parameter theta, the spatial transformation network module performs spatial transformations (such as projective and affine transformations) on the input 6-channel region feature map based on theta, thus achieving the aforementioned palm posture correction purpose and generating an adjusted finger root region feature map. The spatial transformation of the feature map is implemented using basic operations of the PyTorch network. After adjusting the feature map, it is downsampled so that the size of the feature map after pose transformation is reduced to 1 / 4 of the input feature map (i.e., the horizontal and vertical dimensions are reduced by half). This reduces the computational load in the backend coordinate regression process based on global information. While adjusting the hand pose, the finger root region also needs to be enlarged to ensure the accuracy of the final key point coordinate regression and the resolution of the finger root region input to the backend.

[0093] Different hand pose correction and normalization provide high-quality and pose-consistent input samples for subsequent network modules, reducing the learning difficulty of the subsequent key point coordinate regression network module, reducing the learning space of the coordinate regression module, reducing its overfitting risk, and ensuring the overall generalization ability of the model.

[0094] Step S30: Perform key point coordinate fusion regression on the adjusted feature map to obtain the first key point coordinates of the fused regression, and inversely transform the coordinates back to the coordinate system of the original input image based on the above-mentioned palm posture correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image; In this embodiment of the invention, perform key point coordinate fusion regression on the feature map obtained after step S20 to obtain the first key point coordinates of the fused regression, and inversely transform the coordinates back to the coordinate system of the original input image based on the above-mentioned palm posture correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image;

[0095] Furthermore, the robust hand region of interest localization method in natural scenes is characterized by the following step: fusing and regressing the keypoint coordinates of the adjusted feature map, specifically including:

[0096] The key point features are perceived and extracted by a multi-head attention network, and then the coordinates of the key points are globally regressed by a multilayer perceptron to obtain the key point coordinates predicted by the global regression.

[0097] The adjusted feature map is subjected to attention perception and feature extraction through the VisionTransformer network, local key points are detected through the CNN convolutional network, and finally the key point response is mapped to the local regression key point coordinates through the differentiable space to numerical transformation.

[0098] The keypoint coordinates obtained from global regression and local regression are fused using the fusion weights predicted from the current sample features to obtain the first keypoint coordinates of the fused regression.

[0099] The finger root edge distance loss is calculated, and the finger root edge distance loss is defined as follows: d i It is the Euclidean distance sequence between the edge of the i-th finger root and the key point of the i-th finger root, L edge This refers to the loss due to the distance from the root edge.

[0100] Specifically, Differentiable Spatial to Numerical Transform (DSNT) is a network that converts local keypoint responses into their corresponding numerical coordinates. Since the convolutional response encodes the distribution characteristics of local pixel regions, and the convolutional weights are shared across the entire image, the DSNT regressor possesses spatial invariance. That is, even when the hand shifts or rotates, the trained network model can still correctly regress the keypoint coordinates. However, local views cannot perceive global information and are easily affected by region segmentation noise. When the hand region predicted by the network front-end is not ideal, the coordinates are prone to regressing to noise points. Furthermore, when determining the ROI location based on finger root keypoints, it is necessary to first clarify the left and right hand markings, thereby determining the order of the finger root keypoints (which one is the thumb), and then selecting two finger root keypoints—ring-little finger and index-middle finger—to establish the local coordinate system of the hand. Therefore, global information about the hand distribution helps improve the accuracy of keypoint localization. Based on this, the coordinate regression of this scheme includes two parts: local regression and global regression. These two parts are then fused to obtain the coordinates of the key points. The fusion of global and local information enables the coordinate regression network module to have both robustness (spatial invariance) and regression accuracy.

[0101] The global regression specifically includes: first, extracting keypoint features using a multi-head attention network module; then, regressing the keypoint coordinates using a fully connected network. Specifically, the global regression module is a VisionTransformer structure (ViT), a commonly used network model containing a multi-head attention module (MHSA) and a multilayer perceptron (MLP) module. The MHSA module has 6 input feature map channels, a region partitioning window size of 7, 6 output channels, and 3 heads. After MHSA processing, the generated two-dimensional feature map is stretched into a one-dimensional vector and input to the subsequent MLP module. The MLP module includes an input layer, an activation layer, a dropout layer, and an output layer. The output of the output layer is the predicted keypoint coordinates. The coordinates obtained by the global regression network module are denoted as X. g X g ∈R 14 It includes the x and y coordinates of keypoints t1 and t2, as well as the center and corner points of the ROI.

[0102] X g =[x t1 ,y t1 ,xt2 ,y t2 ,x c ,y c ,x r1 ,y r1 ,x r2 ,y r2 ,x r3 ,y r3 ,x r4 ,y r4 ].

[0103] The local regression specifically includes: first, using a Vision Transformer network module to perform attention perception and feature extraction on the pose adjustment feature map generated in step S20; then, using a CNN convolutional module to detect local keypoints; and finally, using DSNT to map the keypoint responses to keypoint coordinates. Specifically, the Vision Transformer module has 6 input channels, 6 output channels, 3 heads, and a window size of 7; the CNN convolutional module structure is: BN+Conv2D+ReLU+Conv2D. The obtained coordinates are denoted as X. l X l ∈R 14 It includes the xy coordinates of keypoints t1 and t2, as well as the center and corner points of the ROI.

[0104] X l =[x t1 ,y t1 ,x t2 ,y t2 ,x c ,y c ,x r1 ,y r1 ,x r2 ,y r2 ,x r3 ,y r3 ,x r4 ,y r4 ].

[0105] Furthermore, the keypoint coordinates obtained from global regression and local regression are fused using fusion weights predicted from the current sample features. The resulting fused regression keypoint coordinates specifically include:

[0106] The two types of coordinates are fused using a weight w. The final coordinates are: X = w·X g +(1-w)·X lAnd w∈[0,1]. The fusion weight w here can be predicted by the network based on the features of the current sample, or it can be fixed to a value, such as 0.5. The prediction method for the fusion weight w is to predict the fusion weight w based on global information while performing keypoint coordinate regression based on global information, and to modify the output of the global regression to X. g ∈R 15 It includes the x and y coordinates of key points t1 and t2, as well as the center and corner points of the ROI, and the initial value w_ of the fusion weight prediction, i.e.

[0107] X g =[x t1 ,y t1 ,x t2 ,y t2 ,x c ,y c ,x r1 ,y r1 ,x r2 ,y r2 ,x r3 ,y r3 ,x r4 ,y r4 ,w_].

[0108] To ensure that w is between 0 and 1, we perform the following processing on w_:

[0109]

[0110] Where s is the function curve shape adjustment parameter, which is learned through the network and has an initial value of 0.

[0111] Furthermore, this invention calculates the loss caused by region segmentation, keypoint localization, finger root edge distance region perception, and DSNT network computation during the training process, specifically including:

[0112] The keypoints predicted by the entire network ultimately include the finger root keypoints t1 and t2, the ROI center point c, and the four corner points r1 to r4. For the output keypoint coordinates, we use the MSE (Mean Squared Loss) function for supervision. The localization loss for the finger root keypoints t1 and t2 is denoted as L. t The location loss for key points of ROI is denoted as L. roi Their definitions are as follows:

[0113]

[0114] Where, k t =[x t1 ,y t1 ,x t2 ,y t2[ ] represents the predicted coordinates of the finger root key points, y t Here, i represents the true value of the corresponding coordinates, and i is the coordinate index.

[0115]

[0116] Where, k roi =[x c ,y c ,x r1 ,y r1 ,x r2 ,y r2 ,x r3 ,y r3 ,x r4 ,y r4 ], which represents the predicted coordinates of the ROI key points (ROI center point c and four corner points r1~r4), y roi The corresponding coordinates are the true values.

[0117] This invention also proposes a finger root edge distance loss to enable the network to perceive the finger root edge region during training. The finger root edge distance loss is defined as: Where, d i It is the Euclidean distance sequence between the edge of the i-th finger root and the key point of the i-th finger root. Specifically, as follows... Figure 6 As shown, each distance in the sequence is determined by the regressed finger root keypoint (denoted as t). i ) to the corresponding finger root edge point (denoted as e) j Composed of, i.e.: d i ={distance(t i ,e j )}, h={1,2},j=1,…,n, where n is the total number of points contained in the finger root edge point sequence, where i represents the finger root edge and has the same value. The total number of points contained in the finger root edge point sequence is sufficient to cover the finger root edge. In the figure, i is also the corresponding edge, and the edge distance loss is denoted as L. edge The finger root edge distance loss in keypoint regression prompts the network model to perceive the finger root edge, encouraging it to focus on this region during feature extraction. This provides high-quality feature input for subsequent coordinate regression, improving accuracy and robustness. By perceiving the finger root edge, the network can locate keypoints even when the finger is closed. Ideally, such as... Figure 13 As shown in the figure, the top and bottom columns represent two examples of input image processing. Figure 13(a) is the input image, (b) is the intermediate feature map, (c) is the detected palm region image, (d) is the region feature map after pose adjustment, (e) is the finger root edge perception feature map, and (f) is the predicted key point. The edge distance loss term enables the network to perceive the finger root edge. During pose adjustment, the finger root area can be enlarged, rotated, and translated as needed to protect and enhance the finger root edge features, making the final output ROI key point image clearer and more accurate.

[0118] As mentioned earlier, for the palm region image and the palm contour region image, we perform focal loss for each pixel to supervise and train the parameters of the segmentation module. This segmentation loss is L. seg .

[0119] In DSNT, the loss function for the response heatmap is the same as the official function without expansion, denoted as L. hm .

[0120] In summary, the overall loss function is:

[0121] L=αL seg +βL t +γL roi +δL edge +μL hm

[0122] Where α, β, γ, δ, and μ are non-negative weights used to balance the importance of each loss. We set them according to the priority of different modules as follows: α = 1.0, β = 0.5, γ = 0.1, δ = 0.1, μ = 0.1. That is, first, ensure the network generates an ideal palm region feature map; then, ensure the coordinate regression accuracy of finger root keypoints t1 and t2; finally, calculate the ROI keypoint localization loss L. roi Minimum distance loss L at the root edge edge and the loss L in key point heatmap detection in DSNT hm This is the lowest priority for constraining the model's learning. In practice, before backpropagation of the loss function for each training batch, we need to normalize each loss term in the loss function, i.e.: Where x∈{seg,t,roi,edge,hm} represents each loss term, |L x1 | represents the specific loss value obtained by the loss term in the current training batch. Here, L x It is a variable, |L x1 | is a constant. This allows the present invention to normalize the scale of different types of losses, balance the degree of influence of different loss terms on the network, and optimize the network based on the network loss.

[0123] Step S40: Based on the obtained coordinates of the second key point, extract the image of the region of interest of the palm.

[0124] In this embodiment of the invention, the region of interest image of the palm is extracted based on the coordinates of the second key point obtained in step S30.

[0125] Furthermore, the step of extracting the region of interest image of the palm based on the obtained second keypoint coordinates specifically includes:

[0126] The training samples are judged based on a pre-set threshold;

[0127] When the sample size reaches the threshold, the coordinates of the corner points of the region of interest on the palm are output using the coordinates of the second keypoint.

[0128] When the sample size does not reach the threshold, the coordinates of the center point of the palm's region of interest and the key points of the finger roots are used to establish a coordinate system and locate the square region.

[0129] Perform a projective transformation on the obtained corner coordinates of the region of interest of the palm to extract the image of the region of interest of the palm.

[0130] Specifically, for ROI image extraction, the four ROI corner points obtained from the fusion regression keypoint coordinates in step S30 can be directly used to perform projective transformation to extract the ROI image; alternatively, the coordinates of the four corner points of the rectangular ROI can be solved first based on the finger root keypoints t1, t2 and the ROI center point c, and then projective transformation can be performed to extract the ROI image. The first method, which directly uses the four ROI corner points obtained from the second keypoint coordinates in step S30 to perform projective transformation to extract the ROI image, predicts four corner points that can adapt to the contour of the hand and are not strictly rectangular; therefore, it can adapt to a certain degree of hand deformation. However, there is a risk of overfitting when the amount of training data is insufficient. When the training set is small, the predicted keypoints may be affected by factors such as finger posture, sleeves, and accessories. The first method, based on the interference of elements, results in incorrect predictions. The second method, based on the finger root keypoints t1 and t2 and the ROI center point c, solves for the coordinates of the four corner points of the square ROI. Further, by connecting the finger root keypoints t1 and t2, a local coordinate system is established by extending a perpendicular line from the midpoint of the connecting line l towards the palm. According to the predefined ROI offset d and ROI side length s, the coordinates of the four corner points r1 to r4 of the square ROI are calculated in the image coordinate system. Then, a projective transformation is performed to extract the ROI image. The ROI generated by this method is strictly square, focusing on detecting the finger root keypoint, a region with obvious visual saliency. It has strong generalization ability, but it does not adapt to abnormal palm postures. When the palm is tilted, the predicted ROI is still square, but the palm print inside is deformed.

[0131] In practice, the selection is based on the size of the training samples, and an appropriate sample threshold is set. Here, the training samples refer to hand image samples before virtual background synthesis and data augmentation, i.e., the initial hand images collected, used to train the hand region of interest (ROI) localization network model. Ideally, when the original training sample size is large enough, reaching millions, the coordinates of the ROI corner points are output using fused regression keypoint coordinates; when the original training sample size is less than millions, the coordinates of the ROI center point and finger root keypoints are output using fused regression keypoint coordinates to establish a coordinate system and locate the square ROI region. Specifically, such as... Figure 10 As shown, once the training sample size reaches the threshold, as... Figure 10 -a Using the first method described above, the corner points can be adjusted to adapt to the edge of the palm, while Figure 10 -b Using the second method described above, the corner points still form a square, failing to adapt to the edges of the palm. Figure 10 -c The capacity of the training samples did not reach the threshold, so the ROI constructed using the ROI coordinates obtained by the first method is incorrect.

[0132] The training steps for the weakly supervised training strategy to train the palm region of interest localization network model include:

[0133] The weak segmenter of the palm region uses a binary classifier trained on a set of palm images with a monochrome background to classify the pixels in the first input palm image, extract the palm region, and perform edge detection on the palm region to obtain the palm contour.

[0134] The palm contour is processed by an iterative palm region of interest localization algorithm to obtain the first annotation information of the input first palm image;

[0135] Based on the palm region of the first palm image, palm pixels are extracted, and the palm pixels are combined with natural background images from the natural image library to synthesize the background and generate a second palm image in a natural environment.

[0136] The second palm image and the first annotation information obtained above are simultaneously subjected to the same random spatial transformation operation to obtain the third palm image and the second annotation information. Then, the image quality of the third palm image is randomly perturbed to obtain the fourth palm image. The resulting data augmented training samples contain the fourth palm image and the second annotation information.

[0137] The hand region of interest localization network model was trained using the aforementioned training samples.

[0138] Specifically, the weakly supervised training strategy first uses a simple localization algorithm to automatically annotate the palm region and key points on a constrained ideal palm image. Then, it generates rich virtual palm image samples with natural backgrounds through background synthesis and data augmentation techniques to train the robust palm region of interest localization method in natural scenes. Current publicly available palmprint databases are collected under strict constraints, often using a monochrome background, fixed palm orientation, spread fingers, and suitable lighting. Therefore, traditional ROI localization algorithms can effectively locate ROIs and generate the required annotation information on these constrained databases. The weakly supervised training strategy mainly includes two functional modules: automated annotation and data augmentation. The automated annotation module mainly includes three parts: a weak palm region segmenter, an iterative palm region of interest localization algorithm, and a label generation method.

[0139] The specific implementation of the weak segmenter for the palm region is as follows: Given a set of palm images with a monochrome background, randomly crop the background region image patch and the skin color image patch of the palm region respectively; treat all background pixel values ​​as one class and all skin color pixel values ​​as another class, and train a binary classifier (such as support vector machine SVM, logistic regression LR binary classifier, etc.); use the binary classifier to classify the pixels in the newly input palm image and extract the palm region.

[0140] The iterative palm region of interest localization algorithm specifically includes:

[0141] The extracted palm region is subjected to key point detection, and the number of key points is verified.

[0142] Based on the number of verified key points, when all 5 fingertip points and 4 finger valley points are detected, the palm region of interest localization method based on the distance extreme points is executed;

[0143] If the verification does not obtain all 5 fingertip points and 4 finger valley points, then perform finger edge detection based on line scanning; if 4 fingertip points and 3 finger valley points are detected, then execute the palm region of interest localization method based on line scanning.

[0144] If no four fingertips and three finger valleys are detected, move to the next scanning point and perform line-scan-based finger edge detection. The detection is then used to determine whether four fingertips and three finger valleys have been detected. If four fingertips and three finger valleys are detected, the palm region of interest localization method based on line scanning is executed. If no four fingertips and three finger valleys are detected, move to the next scanning position and perform line-scan-based finger edge detection again.

[0145] The detection will exit when the scan triggers the stop condition.

[0146] The iterative palm region of interest localization algorithm and the line-scan-based finger edge detection further include:

[0147] The iterative palm region of interest localization algorithm obtains the palm region of interest through a localization method and then performs anomaly detection on the ROI.

[0148] The line-scan-based finger edge detection vertically detects the brightness value of the palm region image from top to bottom in the input palm region. When a complete change pattern is detected, the normal four-finger edge is detected. If no complete change pattern is detected, the vertical scan fails, and the device moves one step to the right to continue the next vertical scan.

[0149] When the scan exceeds the preset scan area, the scan termination condition is triggered, and the scan detection is exited.

[0150] More specifically, this scheme employs an automated annotation method to achieve weakly supervised training. In the iterative hand ROI localization algorithm, the traditional ROI localization algorithm is improved by introducing an anomaly detection module. This module uses geometric information such as position, size, and angle, as well as quality information such as effective image area and sharpness, to determine whether the currently detected ROI is a normal ROI region. ROI localization is initiated, and after obtaining the current ROI, it is judged by the anomaly detection module. If it is determined to be a normal ROI, the algorithm returns and exits successfully; otherwise, it iteratively performs the next ROI detection until a termination condition is triggered, at which point the algorithm exits. Here, the termination condition is a pre-set area used to control the line scan range, such as the pre-set 3 / 4 area of ​​the image. That is, when the scanned position exceeds 3 / 4 of the image width, the termination condition is triggered, and the algorithm exits directly. The iterative palm ROI localization algorithm combines two ROI localization methods. The distance extreme point-based method can obtain information on the key points of the fingertips and bases of the five fingers, and has better robustness to palm rotation (within ±30 degrees). However, this method requires the detection of all five fingers and the recognition of the thumb to properly establish a palm coordinate system for ROI localization. For non-contact palm acquisition, this prerequisite is sometimes not met, so this method is not robust enough. On the other hand, the line scan-based ROI localization method only needs to detect four fingers, so it can serve as an effective supplement to the distance extreme point method.

[0151] like Figure 9 As shown, the present invention uses a palm region of interest localization method based on distance extreme points to obtain the palm region of interest. Figure 9The image in -a is specifically processed as follows: First, the palm region is segmented to obtain the palm edge point sequence and the largest inscribed circle of the palm (radius r, center c). Then, a starting reference point is set as sp (using the point on the rightmost boundary of the image that is closest to the center c of the inscribed circle of the palm, the start point in Figure a). The distance between the reference point sp and each point on the palm contour is calculated (starting from the point on the palm contour closest to sp, calculated counterclockwise along the palm contour), forming a distance variation curve (as shown in Figure b). The local maxima (peaks in Figure b) in the curve correspond to the fingertip point, and the local minima (valley points in Figure b) correspond to the finger root key points (at this time, due to possible palm rotation, the fingertip and finger root points determined according to the distance extrema will have a certain slight error compared with their true positions). In order to further reduce the error, the fingertip point of the middle finger is connected to the center c of the inscribed circle, and then extended forward by a distance of 1.2*r to obtain a new reference point rp (reference point in Figure a). The direction of the line rp->c is taken as the current palm direction. Using rp as the new reference point, the distance sequence between rp and points on the palm contour is calculated counterclockwise to obtain a new distance variation curve (as shown in Figure b). The local maxima of the curve are the fingertips, and the local minima are the finger roots. Since rp is set based on the palm direction, the extreme points of its corresponding distance curve are closer to the actual fingertips and finger roots. Let vp1 be the finger root point corresponding to the index finger and middle finger, and vp2 be the finger root point corresponding to the ring finger and middle finger. A palm rectangular coordinate system can be established by connecting vp1 and vp2 and their perpendicular line. Then, the coordinates of the four corner points of the square ROI can be obtained according to the pre-set ROI offset and ROI side length. The specific method for locating the palm region of interest based on line scanning for palm images is as follows: Figure 9As shown in -c and 9-d, the palm region is first segmented to obtain a binarized palm region image (as shown in Figure c). Let's assume the grayscale value of the palm region pixels is 255, and the grayscale value of the background region pixels is 0. Then, a vertical scan line (denoted as ls) is scanned from top to bottom to detect the pixel value at the current position. During the top-to-bottom scan, when a pixel value first changes from 255 to 0, the current pixel is recorded as p1. Continuing the downward scan, when a pixel value changes from 0 to 255, the current pixel is recorded as p2. Continuing the downward scan, when a pixel value changes from 255 to 0, the current pixel is recorded as p3. Continuing the downward scan, when a pixel value changes from 0 to 255, the current pixel is recorded as p4. Continuing the downward scan, when a pixel value changes from 255 to 0, the current pixel is recorded as p5. Continuing the downward scan, when a pixel value changes from 0 to 255, the current pixel is recorded as p4. Point p1 to p6 is denoted as p6. If a single vertical scan successfully detects points p1 to p6, then: the finger valley edges p1-p2 and p5-p6 are obtained through an edge tracking algorithm; the tangents of the two finger valley edge curves p1-p2 and p5-p6 are detected, and the tangent points corresponding to the tangents are denoted as vp1 and vp2, respectively. A hand rectangular coordinate system can be established by connecting vp1 and vp2 and their perpendicular line. Then, the coordinates of the four corner points of the square ROI can be obtained according to the pre-set ROI offset d and ROI side length s. After that, the program returns a success and exits. If a single vertical scan fails to successfully detect the complete points p1 to p6, then: the program moves a fixed step size to the next horizontal coordinate position. If the new position exceeds the preset range (such as 1 / 2 of the image width or other pre-set thresholds), the program returns a failure and exits. If the new position does not exceed the preset range, the program returns to the above steps and performs a vertical scan again to try to locate the position. Figure 9 The image in -d is extracted to obtain Figure 9 ROI image in -e.

[0152] Furthermore, such as Figure 8 The diagram shows a flowchart of the iterative palm region of interest localization algorithm provided in an embodiment of the present invention. This diagram implements the above-mentioned iterative palm ROI localization algorithm process, which executes the following:

[0153] Step S81 specifically includes: performing the following operations in sequence: starting, inputting a palm image, performing palm region segmentation, performing maximum connected component detection, palm center point detection, generating an edge distance queue, detecting the starting point of the palm contour, rearranging the edge distance queue, detecting fingertip key points, detecting palm direction, detecting reference points, re-sorting the edge distance queue again, detecting fingertip and finger valley key points, and verifying the number of key points.

[0154] Step S82: Determine whether 5 fingertip points and 4 finger valley points are detected. If yes, proceed to step S83; otherwise, proceed to step S84.

[0155] Step S83 specifically includes: executing the ROI localization method based on the extreme distance point, detecting the tangent of the finger valley edge, performing ROI localization and image extraction, detecting ROI geometry and image quality anomalies, and determining whether the ROI is abnormal. If yes, proceed to step S84; otherwise, proceed to step S85.

[0156] Step S84 specifically includes: performing finger edge detection based on line scanning on the image, detecting the white-black-white-black-white-black-white change sequence, and determining whether the detection is successful. If successful, proceed to step S86; otherwise, proceed to step S87.

[0157] Step S85: Output ROI image and coordinates, return successfully;

[0158] Step S86 specifically includes: finger valley edge tangent detection. ROI localization and image extraction, ROI geometric and quality anomaly detection, determining whether the ROI is abnormal. If so, proceed to step S87; otherwise, proceed to step S85.

[0159] Step S87: Move the line scan to the next scan position and determine whether the scan stop condition is triggered. If yes, proceed to step S88; otherwise, proceed to step S84.

[0160] Step S88: Exit if it fails.

[0161] Among them, the maximum connected component detection: only the palm region is retained to exclude background noise interference;

[0162] Palm center point detection: The center of the largest inscribed circle of the palm is the palm center point;

[0163] Edge distance queue: This is the distance queue formed between the reference point and the edge points of the palm in counterclockwise order (the initial reference point can be the midpoint of the image border on the palm root side).

[0164] Edge distance queue rearrangement: The sequence of contour points returned by hand contour detection often starts from the top-left corner of the image. The order of extreme points in the resulting distance queue is affected by hand rotation and cannot be uniquely determined. Therefore, to constrain the position of the first contour point, we stipulate that the point on the hand contour closest to the reference point is the first contour point. Then, the hand contour points are arranged in a counter-clockwise order, and the corresponding distance queue is calculated. This process is considered edge distance queue rearrangement.

[0165] Fingertip keypoint detection: The point on the edge of the palm corresponding to the local maximum of the distance queue is the fingertip point;

[0166] Palm orientation detection: The direction of the straight line formed by the tip of the middle finger and the center point of the palm is regarded as the palm orientation;

[0167] Reference point detection: The point where the line formed by the tip of the middle finger and the center point of the palm is extended towards the base of the palm and intersects the contour can be used as a new reference point (considered as an optimization operation on the starting reference point, so that the reference point is farther away from the tips of the four fingers).

[0168] Finger point detection: The point on the edge of the palm corresponding to the local minimum value of the distance queue is the finger point.

[0169] Verify the number of key points: A normal, complete palm image should be able to detect 5 fingertips and 4 phalanges; when the thumb is not captured, it should be able to detect 4 fingertips and 3 phalanges. Therefore, if all 5 fingertips and 4 phalanges can be detected, execute the ROI localization method based on extreme distance points. If only 4 fingertips and 3 phalanges are detected, execute the ROI localization method based on line scanning.

[0170] like Figure 11 The image shown is an annotation effect image automatically generated on different datasets by the iterative palm region of interest localization algorithm provided in the embodiment of the present invention. After the iterative palm ROI localization algorithm of the present invention automatically annotates the constrained ideal palm, it can obtain training labeled images containing clear and complete palm regions, finger root key points and ROI.

[0171] The ROI localization method based on line scanning is as follows:

[0172] Assuming the palm is placed horizontally with fingertips pointing to the left, the brightness values ​​of the palm area are vertically detected from top to bottom at the starting position (e.g., the image origin). When a complete change pattern of (white-black-white-black-white-black-white) is observed, the normal edge of the four fingers is detected. The tangent points t1 and t2 can be located by detecting the tangent line at the edge of the finger valley. If no complete change pattern is detected, the vertical scan fails, and the scanner moves one step to the right to continue the next vertical scan, detecting the change pattern of (white-black-white-black-white-black-white). If the scan continues to fail, the scanner continues to move one step to the right until it exceeds the pre-set scanning area (e.g., 3 / 4 of the image width), at which point the scan is terminated, and the scan exits.

[0173] The label generation method, combined with the aforementioned iterative palm ROI localization algorithm, is as follows: A first palm image I to be labeled is input into the weak palm region segmenter. The weak region segmenter extracts the palm region Seg_palm, and edge detection is performed on the Seg_palm to obtain the palm contour Boundary. The iterative palm ROI localization algorithm is used to process the palm contour Boundary to obtain the coordinates of the finger root keypoints t1 and t2 (where t1 and t2 correspond to the two tangent points of the longest tangent lines of the two finger root edges). The Boundary is then thickened by 5 pixels to obtain the edge region Seg_edge. The iterative palm ROI localization algorithm is then applied to the finger root keypoints t1 and t2 of the index-middle finger and ring-little finger. The effect is as follows: Figure 5 The diagram shows a clear and complete Region of Interest (ROI), capable of determining the coordinates of the four corner points r1 to r4 and the center point c. Based on the finger root keypoints t1 and t2, finger root edge point sequences edge1 and edge2 are generated, with the following effect: Figure 4 The thickened outline appears at the middle finger valley; thus, the annotation information corresponding to the input first palm image I is obtained, specifically: palm region Seg_palm, palm outline region Seg_edge, finger root key point coordinates t1, t2, ROI key point coordinates corner points r1~r4 and center point c, finger root edge point sequence edge1 and edge2; the annotation information obtained from the above is denoted as the first annotation information L, and L=[Seg_palm,Seg_edge,t1,t2,c,r1~r4,edge1,edge2].

[0174] The data augmentation module includes background synthesis and data augmentation. The iterative ROI localization algorithm using the label generation method can automatically generate the first annotation information L of the constrained palm image. However, our goal is to locate the palm ROI in a natural scene. To fully train the localization network model proposed in this scheme, data augmentation methods are used to generate more complex training samples in a natural state using the pose-constrained palm image and corresponding annotation data. The background synthesis specifically includes: inputting a first hand image I with a constrained pose and background, and its corresponding first annotation information L; randomly extracting a natural background image B from a natural image database; extracting the palm pixel PalmPixel from the first hand image I based on the palm region Seg_palm; and pasting the palm pixel PalmPixel into the background image B to generate a second hand image I2 under natural conditions. Data augmentation specifically includes: inputting the second hand image I2 and the first annotation information L; simultaneously performing random spatial transformation operations such as translation, rotation, cropping, scaling, and perspective transformation on the second hand image I2 and the first annotation information L to obtain a third hand image I3 and the second annotation information L2; and performing random image quality perturbations such as contrast, brightness, blurring, color channel conversion, and grayscale conversion on the third hand image I3 to obtain a fourth hand image I4. Through the above background synthesis and data augmentation, a new training sample (fourth hand image I4, second annotation information L2) can be obtained.

[0175] Specifically, such as Figure 12 The diagram shown is a schematic of the data enhancement process provided in an embodiment of the present invention. The data enhancement process of the present invention involves obtaining an RGB palm image with a restricted background, then processing the background scene image with a natural background using a region mask of the palm outline to obtain a background image, and then synthesizing the background image and the palm image to obtain a composite image of the background. After that, random transformations are performed on the image in terms of grayscale ratio, contrast, brightness, saturation, channel displacement, blur, mirroring, cropping, rotation, scaling, etc., to obtain the data-enhanced palm image.

[0176] Furthermore, the weakly supervised training strategy provided by this invention is shown in the figure below. Figure 7As shown, the weakly supervised training of this invention first trains a palm region segmenter. The training is conducted by inputting a set of standard palm images with a pose-constrained monochrome background, cropping background image blocks and palm skin image blocks, and training a binary classifier based on the distribution of two types of pixel values ​​to achieve the palm region segmenter. After training the palm region segmenter, image annotation is performed, i.e., palm region segmentation and palm contour generation, followed by finger root and ROI keypoint localization and finger root edge point sequence localization. Then, data augmentation is performed, i.e., spatial transformation is performed on the image and annotation information after palm region extraction and complex background synthesis, and image quality is perturbed. After data augmentation, training samples and labels are generated, and the palm keypoint regression network is trained using the samples and labels, i.e., the palm region of interest localization network model is trained.

[0177] This invention illustrates the overall palm key point coordinate regression network structure configuration, as shown below. Figure 2 As shown, in this invention, an input palm image is first acquired, and the palm image is processed by the backbone network to obtain primary features; then, semantic segmentation is used to segment the palm image into regions, resulting in a palm region feature map containing three channels. The three channel outputs of the feature map represent the probability that the current pixel position belongs to one of the three categories (i.e., Figure 1 The process begins with the generation of a BREmap (a feature map in the original text), followed by a softmax operation on the generated 3-channel feature map to obtain a 6-channel feature map. Based on this 6-channel feature map, a localization network is used to obtain correction parameters. These parameters are then used to adjust the hand pose of the feature map, followed by downsampling to obtain the BRE attention feature map, i.e., the adjusted feature map. The adjusted feature map is then subjected to regression based on global and local information. Global regression uses a multi-head attention module (MHSA) and a multilayer perceptron (MLP) to obtain predicted coordinates, while local regression uses a VisionTransformer network, a CNN convolutional module, and a DSNT to obtain predicted coordinates. These coordinates are then fused and regressed to obtain the final coordinates, which are then used to transform the ROI image. Furthermore, the process includes calculations for hand and boundary segmentation Focal loss, coordinate regression MSE loss, and minimum edge distance MSE loss, which are then summed using appropriate weights to obtain the final loss term. Additionally, ①-⑨ in the diagram indicate the general operation of the network.

[0178] Furthermore, this invention utilizes the hand key point coordinate regression network logic through... Figure 3The block diagram briefly describes the process: First, a palm image is input into the invention. The palm region and palm contour region are extracted through semantic segmentation, generating a region feature map (containing the palm region Seg palm and the palm contour region Segedge). Hand pose correction parameters are regressed based on the region feature map. Spatial transformation is performed on the region feature map according to the correction parameters, and finger root region scaling and palm pose adjustment are applied. Then, keypoint coordinate regression is performed, and projection and inverse mapping transformation are performed using these keypoint coordinates to achieve ROI image extraction. Further, after inputting the adjusted region feature map, coordinate regression of global and local information is performed, followed by fusion weight prediction based on global information. The coordinates are fused using the weights to obtain the first keypoint coordinates. These coordinates are then inversely transformed back to the coordinate system of the original input image based on the aforementioned palm pose correction parameters to obtain the second keypoint coordinates corresponding to the region of interest (ROI) of the input palm image.

[0179] This invention uses the method described above to extract the Region of Interest (ROI) from multiple palm images to obtain a localization effect diagram of the network model provided in this embodiment on a natural palm image, such as... Figure 14 As shown in the figure, this invention first achieves semantic segmentation of the palm region, generating feature maps of the palm region and palm contour region. Then, pose transformation is performed on the obtained region feature maps, adaptively adjusting and correcting the geometric position, rotation tilt angle, and scaling of the palm, so that the palm feature maps of different users can be normalized in terms of offset, rotation, and scale after processing. The aforementioned palm and contour region feature maps are downsampled to reduce the computational load of the backend MLP fully connected regression network. At the same time, the finger root region of the palm is enlarged to ensure regression accuracy. After adaptive pose correction, the coordinates of key points are weighted and regressed based on global and local information. The adaptive weighting coefficients are predicted by the network based on the region feature maps. The aforementioned key points include finger root key points t1 and t2, ROI center point c, and four corner points r1 to r4. The ROI image generated after obtaining the coordinates is then... Figure 14 This is reflected in the fact that the ROI images it acquires are relatively clear and accurate.

[0180] Exemplary device

[0181] like Figure 15 As shown, based on the robust palm region of interest localization method in natural scenes described above, this embodiment of the invention provides a robust palm region of interest localization device in natural scenes, the device comprising:

[0182] The palm region and contour extraction module 151 is used to acquire the input palm image, detect the palm region and palm contour region in the input image, and obtain the palm region and contour region feature maps.

[0183] The hand pose adjustment module 152 is used to input the feature maps of the hand region and the contour region into the hand region of interest localization network model trained by a weakly supervised training strategy, regress the feature maps of the hand region and the contour region through the localization network of the trained hand region of interest localization network model to obtain the pose correction parameters of the hand, perform spatial transformation on the palm region feature map based on the pose correction parameters, and perform downsampling and finger root magnification operations on the image to generate the adjusted feature map.

[0184] The palm key point coordinate fusion and regression module 153 is used to perform key point coordinate fusion and regression on the adjusted feature map to obtain the first key point coordinate of the fusion and regression, and then inversely transform the coordinates back to the coordinate system of the original input image based on the above palm posture correction parameters to obtain the second key point coordinate of the palm region of interest corresponding to the input palm image.

[0185] The palm region of interest image extraction module 154 is used to extract the palm region of interest image based on the obtained second key point coordinates.

[0186] In a further explanation, the palm region and contour extraction module 151 implements the effects of the backbone network and palm region and contour extraction module in the network model corresponding to the method of the present invention; the palm posture adjustment module 152 implements the effects of the palm posture adjustment module in the network model corresponding to the method of the present invention; the palm key point coordinate fusion and regression module 153 implements the effects of the palm key point coordinate fusion and regression module in the network model corresponding to the method of the present invention; and the palm region of interest image extraction module 154 implements the effects of the palm region of interest image extraction module in the network model corresponding to the method of the present invention.

[0187] The present invention also provides a feature extraction and recognition device, characterized in that the feature extractor performs palm biometric detection on the palm region of interest image obtained by the robust palm region of interest localization method in natural scenes described above, and realizes user identity recognition and verification based on the detected palm biometric information.

[0188] Specifically, the palm ROI region contains rich palm print features, including: three main lines (heart line, head line, and life line), creases, fingerprint-like valley lines, ridge lines, details, skin texture, and subcutaneous vein patterns. A feature extraction and recognition device can extract information such as the main direction of the lines, the location of their intersections, the local gradient distribution of skin texture, and the spatial distribution of veins. This extracted information is then used to achieve high-precision user identification and verification.

[0189] The ROI region localization obtained by the method described in this invention can effectively reduce the interference caused by changes in palm posture and palm spatial position to subsequent feature recognition, so that the palmprint features collected by the same user in different scenarios have a high degree of consistency. As a prerequisite for palmprint recognition, ROI localization is a strong guarantee for the implementation of non-contact palmprint recognition system, thereby enabling the feature extraction and recognition device described in this invention to more accurately identify and verify the user's identity.

[0190] Based on the above embodiments, the present invention also provides a terminal device. The terminal device includes a memory, a processor, and a robust hand region of interest localization program for natural scenes, stored in the memory and executable on the processor. When the processor executes the robust hand region of interest localization program for natural scenes, it implements the steps of the robust hand region of interest localization method for natural scenes described above.

[0191] In summary, this invention provides a robust method, apparatus, feature extraction and recognition device for locating the region of interest (ROI) of a hand in natural scenes. The method includes acquiring an input hand image; detecting the hand region and hand contour region in the input image to obtain feature maps of the hand region and contour region; inputting the hand region and contour region feature maps into a trained hand ROI localization network model; regressing the hand region and contour region feature maps using the localization network of the hand ROI localization network model trained by a weakly supervised training strategy to obtain hand pose correction parameters; performing spatial transformation on the hand region feature maps based on the pose correction parameters; downsampling and finger root magnification operations on the image to generate an adjusted feature map; performing keypoint coordinate fusion regression on the adjusted feature map to obtain the first keypoint coordinates of the fused regression; and inversely transforming these coordinates back to the coordinate system of the original input image based on the aforementioned hand pose correction parameters to obtain the second keypoint coordinates of the hand ROI corresponding to the input hand image; and extracting the hand ROI image based on the obtained second keypoint coordinates. This invention enables the differentiation of palm region and palm contour region feature maps through a semantic segmentation network. The resulting image feature maps undergo pose adjustment and correction to normalize different palm poses, providing high-quality, pose-consistent input samples for subsequent network modules. Then, the pose-corrected image feature maps undergo coordinate-weighted regression based on global and local information, fusing global and local information to achieve robustness in coordinate regression. Finally, a projective transformation is performed on the coordinates to obtain the palm region of interest image, enabling the processing of palm images with complex backgrounds without the need for an infrared camera. Furthermore, during feature extraction, the network focuses on the finger root edge region, deriving the finger root edge distance loss for keypoint regression, providing high-quality feature input for subsequent coordinate regression and improving regression accuracy and robustness. In addition, this invention improves the training of palm region of interest localization. In the network modeling process, an iterative palm region of interest (ROI) localization algorithm is employed, which incorporates an anomaly detection module and utilizes both distance extreme point detection-based and line scan-based palm ROI localization methods. This approach not only eliminates erroneous palm ROI localizations but also improves the overall success rate of palm ROI localization through iterative search. After obtaining calibration information using the iterative palm ROI localization algorithm, this invention employs a natural scene palm dataset augmentation method. This involves synthesizing a background image from the original image and then augmenting the resulting image, thereby providing more realistic and richer training samples for the localization network and further enhancing its generalization ability. Simultaneously, this invention performs palm biometric detection on the palm ROI images obtained using the robust palm ROI localization method in natural scenes, enabling high-precision user identification and verification.

[0192] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0193] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0194] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications fall within the protection scope of the appended claims.

Claims

1. A robust method for locating the region of interest (ROI) of a hand in natural scenes, characterized in that, The method includes: The process involves acquiring an input image of a hand, detecting the hand region and the hand contour region in the input image, and obtaining feature maps of the hand region and the contour region. The palm region and contour region feature maps are input into a palm region of interest localization network model trained by a weakly supervised training strategy. The localization network of the trained palm region of interest localization network model regresses the palm region and contour region feature maps to obtain the palm pose correction parameters. Based on the pose correction parameters, the palm region feature map is spatially transformed, and the image is downsampled and the finger root is magnified to generate an adjusted feature map. The adjusted feature map is fused and regressed to obtain the first key point coordinates of the fused regression. These coordinates are then inversely transformed back to the coordinate system of the original input image based on the aforementioned palm pose correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image. Based on the obtained coordinates of the second key point, the region of interest image of the palm is extracted; The process of fusing and regressing the key point coordinates of the adjusted feature map specifically includes: The key point features are perceived and extracted by a multi-head attention network, and then the coordinates of the key points are globally regressed by a multilayer perceptron to obtain the key point coordinates predicted by the global regression. The adjusted feature map is processed by a Vision Transformer network for attention perception and feature extraction, then local key points are detected by a CNN convolutional network, and finally the key point response is mapped to the local regression key point coordinates by a differentiable space-to-numerical transformation. The keypoint coordinates obtained from global regression and local regression are fused using the fusion weights predicted from the current sample features to obtain the first keypoint coordinates of the fused regression. The finger root edge distance loss is calculated, and the finger root edge distance loss is defined as follows: , It corresponds to the first The root edge of the bar and the first Euclidean distance sequence of key points at the root of the finger. This refers to the loss due to the distance from the root edge.

2. The robust hand region of interest localization method in natural scenes according to claim 1, characterized in that, The training steps for the weakly supervised training strategy to train the palm region of interest localization network model include: The weak segmenter of the palm region uses a binary classifier trained on a set of palm images with a monochrome background to classify the pixels in the first input palm image, extract the palm region, and perform edge detection on the palm region to obtain the palm contour. The palm contour is processed by an iterative palm region of interest localization algorithm to obtain the first annotation information of the input first palm image; Based on the palm region of the first palm image, palm pixels are extracted, and the palm pixels are combined with natural background images from the natural image library to synthesize the background and generate a second palm image in a natural environment. The second palm image and the first annotation information obtained above are simultaneously subjected to the same random spatial transformation operation to obtain the third palm image and the second annotation information. Then, the image quality of the third palm image is randomly perturbed to obtain the fourth palm image. The resulting data augmented training samples contain the fourth palm image and the second annotation information. The obtained training samples are used to train the palm region of interest localization network model.

3. The robust method for locating the region of interest of a hand in natural scenes according to claim 2, characterized in that, The iterative palm region of interest localization algorithm specifically includes: The extracted palm region is subjected to key point detection, and the number of key points is verified. Based on the number of verified key points, when all 5 fingertip points and 4 finger valley points are detected, the palm region of interest localization method based on the distance extreme points is executed; If the verification does not obtain all 5 fingertip points and 4 finger valley points, then perform finger edge detection based on line scanning; if 4 fingertip points and 3 finger valley points are detected, then execute the palm region of interest localization method based on line scanning. If no four fingertips and three finger valleys are detected, move to the next scanning point and perform line-scan-based finger edge detection. The detection is then used to determine whether four fingertips and three finger valleys have been detected. If four fingertips and three finger valleys are detected, the palm region of interest localization method based on line scanning is executed. If no four fingertips and three finger valleys are detected, move to the next scanning position and perform line-scan-based finger edge detection again. The detection will exit when the scan triggers the stop condition.

4. The robust method for locating the region of interest of a hand in natural scenes according to claim 3, characterized in that, The iterative palm region of interest localization algorithm and the line-scan-based finger edge detection further include: The iterative palm region of interest localization algorithm obtains the palm region of interest through a localization method and then performs anomaly detection on the palm region of interest. The line-scan-based finger edge detection vertically detects the brightness value of the palm region image from top to bottom in the input palm region. When a complete change pattern is detected, the normal four-finger edge is detected. If no complete change pattern is detected, the vertical scan fails, and the device moves one step to the right to continue the next vertical scan. When the scan exceeds the preset scan area, the scan termination condition is triggered, and the scan detection is exited.

5. The robust method for locating the region of interest of a hand in natural scenes according to claim 1, characterized in that, The step of acquiring the input image and detecting the palm region and palm contour region in the input image to obtain the feature maps of the palm region and contour region specifically includes: Primary features are extracted from images using a backbone network with a multi-scale pyramid structure. Based on the acquired primary features, a semantic segmentation network is used to process the primary features to generate a region feature map containing three channels: background region map, palm region map, and palm contour region map. The semantic segmentation effect of the three channel outputs is supervised by focus loss. The generated three-channel region feature map is preprocessed to generate a feature map containing six channels.

6. The robust method for locating the region of interest of a hand in natural scenes according to claim 1, characterized in that, The step of extracting the region of interest image of the palm based on the obtained second key point coordinates specifically includes: The training samples are judged based on a pre-set threshold; When the sample size reaches the threshold, the coordinates of the corner points of the region of interest on the palm are output using the coordinates of the second keypoint. When the sample size does not reach the threshold, the coordinates of the center point of the palm region of interest and the key point of the finger root are used to establish a coordinate system and locate the square palm region of interest. Perform a projective transformation on the obtained corner coordinates of the region of interest of the palm to extract the image of the region of interest of the palm.

7. A robust palm region of interest localization device in natural scenes, used to implement the robust palm region of interest localization method in natural scenes as described in any one of claims 1-6, characterized in that, The device includes: The palm region and contour extraction module is used to acquire the input palm image, detect the palm region and palm contour region in the input image, and obtain the feature maps of the palm region and contour region. The hand pose adjustment module is used to input the feature maps of the hand region and the contour region into the hand region of interest localization network model trained by a weakly supervised training strategy. The localization network of the trained hand region of interest localization network model regresses the feature maps of the hand region and the contour region to obtain the pose correction parameters of the hand. Based on the pose correction parameters, the palm region feature map is spatially transformed, and the image is downsampled and the finger root is magnified to generate the adjusted feature map. The palm key point coordinate fusion and regression module is used to perform key point coordinate fusion and regression on the adjusted feature map to obtain the first key point coordinates of the fused regression, and then inversely transforms these coordinates back to the coordinate system of the original input image based on the above palm posture correction parameters to obtain the second key point coordinates of the palm region of interest corresponding to the input palm image. The palm region of interest image extraction module is used to extract the palm region of interest image based on the obtained second keypoint coordinates.

8. A feature extraction and recognition device, characterized in that, The feature extractor performs palm biometrics detection on the palm region of interest image obtained by the robust palm region of interest localization method in natural scenes according to any one of claims 1-6, and realizes user identity recognition and verification based on the detected palm biometrics.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a robust palm region of interest localization program for natural scenes stored in the memory and executable on the processor. When the processor executes the robust palm region of interest localization program for natural scenes, it implements the steps of the robust palm region of interest localization method for natural scenes as described in any one of claims 1-6.

Images