Palm key point detection method and device, equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a palm key point detection method and device, equipment and a storage medium, and the method comprises the steps: obtaining a palm image; performing feature extraction processing on the palm image through a pre-trained mixed state space model to obtain palm print features and finger joint features of the palm image; performing adaptive fusion processing on the palm print features and the finger joint features through a dynamic gating mechanism of a mixed state space model to obtain fusion features of the palm image; performing palm key point prediction processing on the palm image according to the fusion features to obtain predicted palm key point coordinates of the palm image; and through an adaptive residual likelihood estimation mode, carrying out optimization processing on the predicted palm key point coordinates to obtain palm key point coordinates of the palm image. The palm key point detection efficiency and precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a palm key point detection method, device, equipment and storage medium. BACKGROUND

[0002] Palm key point detection has wide application value. For example, in the medical rehabilitation field, palm key point detection helps to accurately track the movement of the palm, thereby helping doctors to assess the rehabilitation progress of patients and adjust the treatment plan, which is particularly important in the postoperative rehabilitation of the hand or the recovery process after nerve injury; in the field of human-computer interaction, palm key point detection can be used for gesture recognition, thereby realizing more natural touch control, gesture operation, virtual reality (VR) or augmented reality (AR) and other interactive modes, providing immersive and intuitive experience for users.

[0003] In related technologies, palm key point detection relies on edge detection methods of traditional image processing, such as extracting palm features through Canny operator and the like, but this method is easily affected by changes in light, thereby resulting in low accuracy of key point detection.

[0004] In related technologies, palm key point detection also relies on heat map regression methods based on convolutional neural network (CNN), such as generating heat maps using Hourglass and other networks, but this method has quantization errors, thereby affecting the accuracy of palm key point detection, and also has the defect of redundant calculation, which is not efficient.

[0005] In related technologies, palm key point detection also relies on end-to-end detection methods based on Transformer, although this method has global modeling capability, but its computational complexity increases quadratically with the increase of image resolution, which is not efficient.

[0006] Moreover, the above methods all have defects such as insufficient adaptability to skin elastic deformation and insufficient modeling of multi-joint linkage, resulting in large errors in palm key point detection, thereby affecting the accuracy of palm key point detection, especially in the medical rehabilitation field, it is difficult to meet the clinical evaluation standard.

[0007] Therefore, how to improve the efficiency and accuracy of key point detection has become a technical problem to be solved. SUMMARY

[0008] The present application provides a palm key point detection method, device, computer equipment and storage medium, aiming to improve the efficiency and accuracy of palm key point detection.

[0009] In a first aspect, a palm key point detection method is provided, comprising:

[0010] obtaining a palm image;

[0011] extracting features of the palm image by using a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image;

[0012] adaptively fusing the palm print features and the knuckle features by using a dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image;

[0013] predicting palm key points of the palm image according to the fused features to obtain predicted palm key point coordinates of the palm image;

[0014] optimizing the predicted palm key point coordinates by using an adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image.

[0015] In a second aspect, a palm key point detection device is provided, which includes:

[0016] an image acquisition module configured to acquire a palm image;

[0017] a feature extraction module configured to extract features of the palm image by using a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image;

[0018] a feature fusion module configured to adaptively fuse the palm print features and the knuckle features by using a dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image;

[0019] a key point prediction module configured to predict palm key points of the palm image according to the fused features to obtain predicted palm key point coordinates of the palm image;

[0020] a key point optimization module configured to optimize the predicted palm key point coordinates by using an adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image.

[0021] In a third aspect, a computer device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the palm key point detection method when executing the computer program.

[0022] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program, and the computer program implements the steps of the palm key point detection method when executed by a processor.

[0023] In the scheme implemented by the above-mentioned palm key point detection method, device, computer equipment and storage medium, a palm image is obtained; feature extraction processing is performed on the palm image through a pre-trained hybrid state space model to obtain palm print features and finger joint features of the palm image; the palm print features and finger joint features are adaptively fused through the dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image; palm key points of the palm image are predicted based on the fused features to obtain predicted palm key point coordinates of the palm image; the predicted palm key point coordinates are optimized through an adaptive residual likelihood estimation method to obtain the palm key point coordinates of the palm image. In the present invention, on the one hand, palm print features and finger joint features are quickly extracted from the palm image through a trained hybrid state space model, so that the palm print features and finger joint features complement each other, which can improve the comprehensiveness and accuracy of feature extraction; on the other hand, the palm print features and finger joint features are flexibly and adaptively fused through the dynamic gating mechanism of the trained hybrid state space model, which has stronger adaptability than the traditional fixed feature fusion method, and provides a more comprehensive, richer and more detailed feature representation for the palm image, which can better represent the comprehensive information of the palm image, so as to more accurately predict the predicted palm key point coordinates of the palm image based on the fused features; on the other hand, the predicted palm key point coordinates are optimized through the adaptive residual likelihood estimation method, which can significantly reduce the prediction error and ensure that the final palm key point coordinates are more accurate and conform to biological and anatomical laws, thereby significantly improving the key point detection performance under complex palm postures, thereby improving the efficiency and accuracy of palm key point detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0025] Figure 1 1 is a flow chart of a palm key point detection method according to an embodiment of the present invention;

[0026] Figure 2 yes Figure 1 A schematic flow chart of a specific implementation of step S20;

[0027] Figure 3 yes Figure 1 A schematic flow chart of a specific implementation of step S30;

[0028] Figure 4 yes Figure 1A specific embodiment flowchart of step S50 is shown in the figure;

[0029] Figure 5 A structure diagram of a palm key point detection device in an embodiment of the present application is shown in the figure.

[0030] Figure 6 A structure diagram of a computer device in an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0032] The palm key point detection method provided by the embodiments of the present application can be applied in a computer device. The computer device can acquire a palm image. The palm image is subjected to feature extraction processing through a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image. The palm print features and the knuckle features are subjected to adaptive fusion processing through a dynamic gating mechanism of the hybrid state space model to obtain fusion features of the palm image. The palm image is subjected to palm key point prediction processing according to the fusion features to obtain predicted palm key point coordinates of the palm image. The predicted palm key point coordinates are subjected to optimization processing through an adaptive residual likelihood estimation manner to obtain palm key point coordinates of the palm image. In the present application, on the one hand, the palm print features and the knuckle features are quickly extracted from the palm image through the trained hybrid state space model, so that the palm print features and the knuckle features complement each other, which can improve the comprehensiveness and accuracy of feature extraction. On the other hand, the palm print features and the knuckle features are subjected to flexible adaptive fusion processing through the dynamic gating mechanism of the trained hybrid state space model, which has stronger adaptability compared with a traditional fixed feature fusion manner, realizes providing more comprehensive, richer and more detailed feature representation for the palm image, and can better represent the comprehensive information of the palm image, so as to more accurately predict the predicted palm key point coordinates of the palm image based on the fusion features. On the other hand, the predicted palm key point coordinates are optimized through the adaptive residual likelihood estimation manner, which can significantly reduce prediction errors and ensure that the final palm key point coordinates are more accurate and comply with biological and anatomical laws. Therefore, the key point detection performance under a complex palm posture is significantly improved, so that the efficiency and accuracy of palm key point detection are improved.

[0033] The computer device can include a terminal device or a server, the terminal device can include a smartphone, a tablet computer, a notebook computer, a desktop computer, a personal digital assistant and a wearable device, etc., and the server can be a standalone server or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0034] Please refer to Figure 1 as shown, Figure 1 A flowchart of a palm key point detection method provided by an embodiment of the present application is shown in FIG. 1, which includes the following steps:

[0035] S10: Obtain a palm image.

[0036] The palm key point detection method provided by the present application mainly includes two processes: process one is to use a hybrid state space model to extract and fuse features of the palm image to obtain fused features of the palm image. Process two is to locate palm key points of the palm image based on the fused features, wherein process two includes two branches. Branch one is to predict palm key points of the palm image based on the fused features to obtain predicted palm key point coordinates. Branch two is to optimize the predicted palm key point coordinates by using an adaptive residual likelihood estimation method to obtain final palm key point coordinates. Thus, by combining the hybrid state space model and the adaptive residual likelihood estimation method, high-precision palm key points that conform to physiological structures are located, and the accuracy and robustness of palm key point detection are significantly improved.

[0037] For ease of understanding, the terms involved in the present application are first explained:

[0038] Artificial intelligence (AI): is a new technical science of studying, developing, simulating, extending and expanding human intelligence, and is a branch of computer science. Artificial intelligence aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0039] Mamba-Transformer hybrid architecture: a multi-modal state space fusion architecture that combines a bidirectional Mamba network and a Windowed Transformer network. This hybrid visual architecture can extract features from different scales of palm images. The bidirectional Mamba network can capture long-range dependencies of palm texture from a global scale and preserve the topological structure of the palm image, thereby more comprehensively understanding the direction of the palm print, and finally outputting global palm print features. The Windowed Transformer uses attention mechanisms to capture detailed features of the knuckle region from a local scale through sliding windows, increasing the feature expression of the knuckle region and other detailed regions, and finally outputting local knuckle features. The two complement each other and can improve the comprehensiveness and accuracy of feature extraction. The palm print features output by the bidirectional Mamba network and the knuckle features output by the Windowed Transformer network are adaptively fused through a dynamic gating mechanism to obtain fused features to better represent the comprehensive information of the palm image.

[0040] Dynamic gating mechanism: an optimization mechanism within the Mamba-Transformer hybrid architecture that dynamically adjusts the importance of the palm print features output by the bidirectional Mamba network and the knuckle features output by the Windowed Transformer network based on the characteristics of the palm print image.

[0041] Adaptive Residual Likelihood Estimation (ARLE): a method used in statistical modeling or machine learning, especially when dealing with uncertainty or estimation error. Its core idea is to optimize the likelihood estimation by dynamically adjusting the residual (i.e., the difference between the true value and the predicted value) to better adapt to complex patterns or noise in the data. The embodiments of the present invention use adaptive residual likelihood estimation to balance the accuracy of key point positioning and physiological reasonableness.

[0042] Based on this, the palm key point detection method provided by the present invention is described in detail below.

[0043] In step S10 of some embodiments, an original palm image can be obtained; the original palm image is dynamically enhanced to obtain a palm image.

[0044] In step S10, the original palm image can be captured by an infrared or visible light camera to ensure the quality of the original palm image. Then the original palm image is dynamically enhanced to obtain a palm image, further improving the quality, adaptability and usability of the palm image.

[0045] S20: The palm image is input into the pre-trained hybrid state space model, so that the pre-trained hybrid state space model performs feature extraction processing on the palm image to obtain palm print features and knuckle features of the palm image.

[0046] The palm image is input into the pre-trained hybrid state space model, so that the pre-trained hybrid state space model performs feature extraction processing on the palm image to obtain palm print features and knuckle features of the palm image.

[0047] For example, the resolution of the palm image can be 512x512x3, which can ensure that the palm image has clear details, balances the quality of the palm image, and does not occupy too much storage space or computing resources, which helps to improve the computational complexity and amount of palm key point detection.

[0048] In some embodiments, referring to Figure 2 , step S20 can include but is not limited to the following steps:

[0049] S21: The palm print features of the palm image are extracted by the first network of the hybrid state space model.

[0050] S22: The knuckle features of the palm image are extracted by the second network of the hybrid state space model.

[0051] For example, the pre-trained hybrid state space model includes a Mamba-Transformer hybrid architecture, the first network can be a bidirectional Mamba network, and the second network can be a Windowed Transformer network.

[0052] For S21-S22, the palm image is input into the Mamba-Transformer hybrid architecture. First, the palm image is convolved in the Stem layer of the Mamba-Transformer hybrid architecture to obtain a convolved palm image. Then, the convolved palm image is down-sampled by MaxPool to obtain a palm image with reduced size, for example, the size of the palm image can be reduced to 128x128, which facilitates subsequent feature extraction.

[0053] Then in the Stage1 layer of the Mamba-Transformer hybrid architecture, the palmprint feature of the reduced-size palm image is extracted by the bidirectional Mamba network, and the palmprint feature of the palm image is obtained. The palmprint feature is the long-range feature of the palmprint. Due to the uniqueness and complexity of the palmprint, the deformation of the palmprint in the palmprint image may occur due to the influence of the light change and the angle change during shooting. The bidirectional Mamba network can effectively cope with the deformation of the palmprint by capturing the long-range dependence relationship of the palm texture in the global, enhance the robustness to the deformation of the palmprint, and thus improve the accuracy of the palmprint feature.

[0054] Then in the Stage2 layer of the Mamba-Transformer hybrid architecture, the knuckle feature of the reduced-size palm image is extracted by the Windowed Transformer network, and the knuckle feature of the palm image is obtained. Due to the influence of the bending and stretching state of the finger during shooting, the knuckles in the palmprint image may also be deformed. The Windowed Transformer network can accurately capture the subtle changes of the knuckles by focusing on the local knuckle region, enhance the robustness to the deformation of the knuckles, and thus improve the accuracy of the knuckle feature.

[0055] S30: The palmprint feature and the knuckle feature are adaptively fused by a dynamic gating mechanism of a hybrid state space model to obtain a fusion feature of the palm image.

[0056] After the palmprint feature and the knuckle feature of the palm image are extracted, the palmprint feature and the knuckle feature are adaptively fused by a dynamic gating mechanism inside the Mamba-Transformer hybrid architecture to obtain a fusion feature of the palm image, which provides more comprehensive, richer and more detailed image representation for the palm image, and better represents the comprehensive information of the palm image, and provides accurate basis for subsequent palm key point positioning.

[0057] In some embodiments, referring to Figure 3 , step S30 can include but is not limited to the following steps:

[0058] S31: A dynamic gating coefficient is generated by the dynamic gating mechanism according to the palmprint feature and the knuckle feature.

[0059] S32: The palmprint feature and the knuckle feature are weighted and fused according to the gating coefficient to obtain the fusion feature.

[0060] For step S31, a dynamic gating mechanism is used to dynamically generate a gating coefficient based on the palmprint feature and the knuckle feature. The gating coefficient is a weight coefficient calculated by a Sigmoid activation function, which can be understood as a coefficient for controlling which feature (palmprint feature or knuckle feature) to obtain more information from, for controlling the weight of the palmprint feature and the knuckle feature in the feature fusion process.

[0061] In step S31 of some embodiments, the palmprint feature and the knuckle feature can be spliced to obtain a spliced feature; the spliced feature is passed to the dynamic gating mechanism to generate a gating coefficient using a Sigmoid activation function based on the spliced feature.

[0062] Specifically, the dynamic gating mechanism can calculate the gating coefficient using a gating coefficient calculation formula, as shown below:

[0063] G = σ(W g [F mamba , F transformer ] + b g )

[0064] Where G represents the gating coefficient;

[0065] σ represents the Sigmoid activation function;

[0066] W g represents the weight matrix;

[0067] F mamba represents the palmprint feature;

[0068] F transformer represents the knuckle feature;

[0069] represents the spliced feature obtained by splicing the palmprint feature and the knuckle feature;

[0070] b g represents a bias term.

[0071] That is, the palmprint feature F mamba and the knuckle feature F transformer are first spliced to obtain the spliced feature Then the spliced feature is passed to the dynamic gating mechanism, which uses the weight matrix W g to perform linear transformation on the spliced feature and the bias term b g to obtain a linear combination Finally, the Sigmoid activation function is used to normalize the linear combination to realize the normalization of the linear combination is converted into a numerical value between 0 and 1 as a gating coefficient G.

[0072] G represents the palmprint feature F mamba the weight coefficient in the feature fusion process, (1-G) represents the knuckle feature F transformer the weight coefficient in the feature fusion process.

[0073] For step S32, the dynamic gating mechanism can perform weighted fusion calculation on the palmprint feature F mamba and the knuckle feature F transformer based on G and (1-G) using the feature fusion calculation formula, and obtain the fused feature, and the feature fusion calculation formula is as follows:

[0074] F fusion = G O F mamba + (1-G) O F transformer

[0075] Wherein, F fusion represents the fused feature;

[0076] O represents element-wise multiplication (i.e. element-level multiplication).

[0077] The feature fusion calculation formula shows the adaptive fusion process of the palmprint feature F mamba and the knuckle feature F transformer When the value of the gating coefficient G is large, the fused feature F fusion pays more attention to the palmprint feature F mamba When the value of the gating coefficient G is small, the fused feature F fusion pays more attention to the knuckle feature F transformer .

[0078] In this way, the Mamba-Transformer hybrid architecture realizes flexible adjustment of the weight of the palmprint feature F mamba and the knuckle feature F transformer in the feature fusion process by introducing the dynamic gating mechanism, which helps to fully combine the advantages of the palmprint feature F mamba and the knuckle feature F transformer and improve the flexibility and rationality of feature fusion, thereby significantly improving the accuracy of the fused feature.

[0079] S40: Perform palm key point prediction processing on the palm image according to the fused feature to obtain the predicted palm key point coordinates of the palm image.

[0080] Then, the palm key point positioning network can be used to locate the palm key points of the palm image.

[0081] The palm key point positioning network comprises a palm key point prediction layer and an adaptive residual likelihood estimation layer.

[0082] The palm key point prediction layer is a fully connected layer, which is configured to regress the coordinates of each key point to obtain predicted palm key point coordinates of the palm image.

[0083] Specifically, the fusion features are input into the palm key point prediction layer, so that the palm key point prediction layer performs palm key point prediction processing on the palm image according to the fusion features to obtain predicted palm key point coordinates of the palm image, such as 21 key point coordinates in the palm image.

[0084] S50: The predicted palm key point coordinates are optimized by the adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image.

[0085] Finally, the predicted palm key point coordinates are optimized by the adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image, ensuring the rationality and accuracy of the final palm key point coordinates.

[0086] In S50 in some embodiments, please refer to Figure 4 The optimization of the predicted palm key point coordinates by the adaptive residual likelihood estimation method can include but is not limited to the following steps:

[0087] S51: Obtain the deformation residual of the predicted palm key point coordinates to model the distribution of the deformation residual by the maximum likelihood estimation method.

[0088] S52: Error constraint is performed on the predicted palm key point coordinates.

[0089] S53: The distribution of the deformation residual is subjected to normal distribution constraint.

[0090] S54: The predicted palm key point coordinates are subjected to anatomical constraint, wherein the weights of the error constraint, the normal distribution constraint and the anatomical constraint are adjusted by a dynamic temperature coefficient.

[0091] The adaptive residual likelihood estimation layer models the deformation residual of the predicted palm key point coordinates by the maximum likelihood estimation (MLE) method to perform normal distribution constraint on the residual distribution of the predicted palm key point coordinates, performs error constraint on the predicted palm key point coordinates, performs anatomical constraint on the predicted palm key point by a differentiable joint angle constraint, and balances the rationality of the residual distribution constraint, the error constraint and the anatomical constraint by a soft threshold algorithm adjusted by a dynamic temperature coefficient, so as to realize the optimization of the predicted palm key point coordinates and further improve the positioning accuracy of the palm key point.

[0092] Specifically, the predicted palm keypoint coordinates are input into the adaptive residual likelihood estimation layer, and the deformation residuals of the predicted palm keypoint coordinates are output by a multi-layer perceptron (MLP) first, for example, a deformation residual δ ∈ R^(21×3) represents the three-dimensional residuals of 21 keypoint coordinates, that is, δ is a 21×3 matrix, and each row represents the deformation residual of a keypoint.

[0093] The distribution of the deformation residual is modeled as a Gaussian distribution, and the specific formula is:

[0094]

[0095] wherein, δ i represents the deformation residual of the i-th predicted keypoint coordinate;

[0096] μ i represents the mean of the deformation residual δ i of the i-th predicted keypoint coordinate, which is output by the MLP;

[0097] σ i represents the standard deviation of the deformation residual δ i of the i-th predicted keypoint coordinate, which is output by the Softplus activation function of the MLP, and ensures that the standard deviation is positive.

[0098] The formula of the Gaussian distribution describes the residual distribution of the predicted palm keypoint coordinates (i.e., it indicates what kind of law the deformation residual of the predicted palm keypoint coordinates should follow).

[0099] On this basis, the predicted keypoint coordinates are optimized by various loss functions. The loss functions include Lwing loss function, Lres loss function, Langle loss function and their total loss function Ltotal.

[0100] ①The Lwing loss function is used to constrain the error of the predicted keypoint coordinates, and the formula is:

[0101]

[0102] wherein, x represents the error between the predicted keypoint coordinates and the real palm keypoint coordinates;

[0103] ω represents the wing length, which determines the position where the linear part starts;

[0104] ε represents an adjustment parameter, which is used to affect the curve shape of the nonlinear part;

[0105] C represents a constant term, which ensures that the formula of the Lwing loss function is continuous at the transition point;

[0106] The Lwing loss function can effectively control the error of the predicted key point coordinates.

[0107] The Lres loss function constrains the distribution of the deformation residual by KL divergence, so that it is close to the standard normal distribution, and its formula is:

[0108]

[0109] Where q(δ) represents the real residual distribution.

[0110] The Lres loss function promotes the residual distribution p(δ|θ) of the palm key point coordinates to be close to the standard normal distribution.

[0111] The Langle loss function constrains the angle between the predicted palm key points by anatomical constraints, so that it conforms to the reasonable anatomical range, and its formula is:

[0112]

[0113] Where p j , p k , p l represent the coordinates of the knuckles;

[0114] θ std represents the standard angle value from the medical atlas;

[0115] represents the tolerance.

[0116] The Langle loss function can constrain the posture of the predicted palm key points, ensuring that the angle between the predicted palm key points does not exceed the normal physiological range, thereby ensuring that the posture of the predicted palm key points is biologically reasonable.

[0117] The total loss function Ltotal is the weighted sum of the Lwing loss function, the Lres loss function and the Langle loss function, and the weight of each loss function is adjusted by the dynamic temperature coefficient, which can automatically adjust the importance of each loss function. The formula of the total loss function Ltotal is:

[0118]

[0119] Where L i represents the Lwing loss function, the Lres loss function and the Langle loss function;

[0120] ω i(t) represents the weights of the Lwing loss function, the Lres loss function and the Langle loss function, which means that when a certain loss function is smaller, its weight will increase, and when a certain loss function is larger, its weight will decrease.

[0121] T(t) represents a dynamic temperature coefficient.

[0122] Therefore, by balancing the error constraint of predicting palm key point coordinates, the normal distribution constraint of the residual distribution of predicting palm key point coordinates and the anatomical constraint of predicting palm key points through the total loss function Ltotal, the prediction of palm key point coordinates is adaptively optimized, which can significantly reduce the prediction error and ensure that the final palm key point coordinates are more accurate and consistent with biological and anatomical laws.

[0123] Before step S10 of some embodiments, the untrained hybrid state space model can also be trained using sample palm images to obtain a trained hybrid state space model.

[0124] Among them, the sample palm image can be preprocessed by illumination robustness enhancement and occlusion enhancement. Illumination robustness enhancement, for example, uses a physical rendering engine to add multiple lighting conditions to the sample palm image, thereby enhancing the adaptability of the hybrid state space model to different lighting environments; occlusion enhancement, for example, randomly erases the palm area in the sample palm image (erasure rate 15%-30%), which helps to improve the robustness of the hybrid state space model to occlusion.

[0125] The untrained hybrid state space model can be trained by a two-stage training method. For example, in the first stage, the parameters of the untrained bidirectional Mamba network are frozen, and only the Windowed Transformer network is trained to adjust the parameters of the Windowed Transformer network; then overall training is performed to adjust the parameters of the bidirectional Mamba network while further adjusting the parameters of the Windowed Transformer network, until the training times reach a threshold, to obtain a trained hybrid state space model.

[0126] The trained hybrid state space model can be deployed in a computer device by a dynamic quantization deployment method, for example, the bidirectional Mamba network part is quantized to FP16, and the Windowed Transformer network part is quantized to INT8, thereby reducing the memory occupancy of the hybrid state space model and improving the calculation speed.

[0127] Meanwhile, a dynamic memory allocator is adopted to dynamically allocate GPU memory according to the calculation requirements of the bidirectional Mamba network part and the Windowed Transformer network part, so that the use of GPU memory of the trained hybrid state space model during running is reasonable and effective, and GPU memory overflow or unnecessary resource waste is avoided.

[0128] It can be seen that, in the above scheme, the state space modeling and residual learning are innovatively combined to realize efficient feature extraction, feature adaptive fusion, accurate key point prediction and post-optimization, significantly improving the accuracy, robustness and efficiency of palm key point detection under complex palm postures, and being suitable for various practical application scenarios such as human-computer interaction gesture recognition, biometric recognition, rehabilitation medical assessment, etc.

[0129] The palm key point detection method provided by the embodiment of the application acquires a palm image; performs feature extraction processing on the palm image through a pre-trained hybrid state space model to obtain palm print features and finger joint features of the palm image; performs adaptive fusion processing on the palm print features and the finger joint features through a dynamic gating mechanism of the hybrid state space model to obtain fusion features of the palm image; performs palm key point prediction processing on the palm image according to the fusion features to obtain predicted palm key point coordinates of the palm image; and performs optimization processing on the predicted palm key point coordinates through an adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image. In the application, on the one hand, the palm print features and the finger joint features are quickly extracted from the palm image through the trained hybrid state space model, so that the palm print features and the finger joint features complement each other, and the comprehensiveness and precision of feature extraction are improved; on the other hand, the palm print features and the finger joint features are flexibly and adaptively fused through the dynamic gating mechanism of the trained hybrid state space model, which has stronger adaptability than a traditional fixed feature fusion method, provides more comprehensive, richer and more detailed feature representation for the palm image, and better represents the comprehensive information of the palm image, so that the predicted palm key point coordinates of the palm image are more accurately predicted based on the fusion features; and on the other hand, the predicted palm key point coordinates are optimized through the adaptive residual likelihood estimation method, which can significantly reduce prediction errors, ensure that the final palm key point coordinates are more accurate and comply with biological and anatomical laws, thereby significantly improving the key point detection performance under complex palm postures, and improving the efficiency and precision of palm key point detection.

[0130] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiment of the application.

[0131] It should be noted that the non-company software tools or components appearing in the embodiments of the present application are only illustrative and do not represent actual use.

[0132] In an embodiment, a palm key point detection device is provided, which corresponds to the palm key point detection method in the above embodiments. As shown in the figure, the palm key point detection device comprises an image acquisition module 101, a feature extraction module 102, a feature fusion module 103, a key point prediction module 104 and a key point optimization module 105. The detailed description of each functional module is as follows: Figure 5

[0133] The image acquisition module 101 is configured to acquire a palm image.

[0134] The feature extraction module 102 is configured to perform feature extraction processing on the palm image by using a pre-trained hybrid state space model, to obtain palm print features and knuckle features of the palm image.

[0135] The feature fusion module 103 is configured to perform adaptive fusion processing on the palm print features and the knuckle features by using a dynamic gating mechanism of the hybrid state space model, to obtain fusion features of the palm image.

[0136] The key point prediction module 104 is configured to perform palm key point prediction processing on the palm image according to the fusion features, to obtain predicted palm key point coordinates of the palm image.

[0137] The key point optimization module 105 is configured to perform optimization processing on the predicted palm key point coordinates by using an adaptive residual likelihood estimation method, to obtain palm key point coordinates of the palm image.

[0138] In an embodiment, the feature extraction module 102 is specifically configured to:

[0139] perform palm print feature extraction processing on the palm image by using a first network of the hybrid state space model, to obtain palm print features of the palm image.

[0140] perform knuckle feature extraction processing on the palm image by using a second network of the hybrid state space model, to obtain knuckle features of the palm image.

[0141] In an embodiment, the feature fusion module 103 is specifically configured to:

[0142] generate gating coefficients dynamically according to the palm print features and the knuckle features by using the dynamic gating mechanism.

[0143] ​According to the gating coefficient, the palmprint feature and the knuckle feature are weighted and fused to obtain the fused feature.

[0144] In an embodiment, the feature fusion module 103 is further configured to:

[0145] The palmprint feature and the knuckle feature are spliced to obtain a spliced feature.

[0146] The spliced feature is transmitted to the dynamic gating mechanism to generate the gating coefficient based on the spliced feature using a Sigmoid activation function.

[0147] In an embodiment, the key point optimization module 105 is specifically configured to:

[0148] Obtain a deformation residual of the predicted palm key point coordinates to model a distribution of the deformation residual by a maximum likelihood estimation method.

[0149] Error constraint is performed on the predicted palm key point coordinates.

[0150] The distribution of the deformation residual is subjected to a normal distribution constraint.

[0151] The predicted palm key point coordinates are subjected to an anatomical constraint, and the weights of the error constraint, the normal distribution constraint and the anatomical constraint are adjusted by a dynamic temperature coefficient.

[0152] In an embodiment, the hybrid state space model comprises a Mamba-Transformer hybrid architecture, the first network is a bidirectional Mamba network, and the second network is a Windowed Transformer network.

[0153] In an embodiment, the image acquisition module 101 is specifically configured to

[0154] Obtain an original palm image.

[0155] Perform dynamic enhancement processing on the original palm image to obtain the palm image.

[0156] The application provides a palm key point detection device, which can extract palm print features and knuckle features from a palm image quickly through a trained mixed state space model, so that the palm print features and the knuckle features complement each other, the comprehensiveness and accuracy of feature extraction are improved, the palm print features and the knuckle features are adaptively fused through a dynamic gating mechanism of the trained mixed state space model, the adaptability is higher than that of a traditional fixed feature fusion mode, more comprehensive, richer and more detailed feature representation is provided for the palm image, the comprehensive information of the palm image can be better represented, and the predicted palm key point coordinates of the palm image can be more accurately predicted based on the fused features; in addition, the predicted palm key point coordinates are optimized through an adaptive residual likelihood estimation mode, the prediction error can be significantly reduced, the final palm key point coordinates are more accurate and meet biological and anatomical laws, the key point detection performance under a complex palm posture is significantly improved, and the efficiency and accuracy of palm key point detection are improved.

[0157] The specific limitations of the palm key point detection device can be referred to the limitations of the palm key point detection method in the above, and will not be described here. Each module in the palm key point detection device can be realized by software, hardware and a combination thereof in whole or in part. The above modules can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above modules.

[0158] In one embodiment, a computer device is provided, and an internal structure diagram thereof can be as shown in Figure 6 The computer device includes a processor, a memory, a network interface and a database connected through a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client through a network connection. The computer program is executed by the processor to implement the functions or steps of the palm key point detection method on the computer device side.

[0159] In one embodiment, a computer device is provided, which includes a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the following steps when executing the computer program:

[0160] obtaining a palm image;

[0161] extracting features of the palm image by using a pre-trained hybrid state space model to obtain palmprint features and knuckle features of the palm image;

[0162] adaptively fusing the palmprint features and the knuckle features by using a dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image;

[0163] predicting palm key points of the palm image according to the fused features to obtain predicted palm key point coordinates of the palm image;

[0164] optimizing the predicted palm key point coordinates by using an adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image.

[0165] In one embodiment, a computer readable storage medium is provided, and the computer readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0166] obtaining a palm image;

[0167] extracting features of the palm image by using a pre-trained hybrid state space model to obtain palmprint features and knuckle features of the palm image;

[0168] adaptively fusing the palmprint features and the knuckle features by using a dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image;

[0169] predicting palm key points of the palm image according to the fused features to obtain predicted palm key point coordinates of the palm image;

[0170] optimizing the predicted palm key point coordinates by using an adaptive residual likelihood estimation method to obtain palm key point coordinates of the palm image.

[0171] It should be noted that the functions or steps that the computer readable storage medium or the computer device can implement correspond to the related descriptions of the computer device in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0172] Those skilled in the art will appreciate that all or part of the processes in the above-described embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the above-described embodiments. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct RAM bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM).

[0173] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0174] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A palm key point detection method, characterized in that: include: Get palm image; Performing feature extraction processing on the palm image using a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image; Adaptively fusing the palm print features and the finger joint features through the dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image; Performing palm key point prediction processing on the palm image according to the fusion features to obtain predicted palm key point coordinates of the palm image; The predicted palm key point coordinates are optimized by an adaptive residual likelihood estimation method to obtain the palm key point coordinates of the palm image.

2. The palm key point detection method according to claim 1, wherein: The method of performing feature extraction processing on the palm image by using a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image includes: performing palmprint feature extraction processing on the palm image through the first network of the hybrid state space model to obtain palmprint features of the palm image; The second network of the hybrid state space model is used to perform knuckle feature extraction processing on the palm image to obtain knuckle features of the palm image.

3. The palm key point detection method according to claim 1, wherein: The adaptive fusion processing of the palm print features and the finger joint features is performed through the dynamic gating mechanism of the hybrid state space model to obtain the fusion features of the palm image, including: Dynamically generating a gating coefficient according to the palm print features and the finger joint features through the dynamic gating mechanism; According to the gating coefficient, weighted fusion processing is performed on the palmprint feature and the finger joint feature to obtain the fused feature.

4. The palm key point detection method according to claim 3, wherein: The dynamic gating mechanism is used to dynamically generate a gating coefficient according to the palm print features and the finger joint features, including: performing splicing processing on the palm print feature and the finger joint feature to obtain a spliced ​​feature; The spliced ​​features are passed to the dynamic gating mechanism to generate the gating coefficients based on the spliced ​​features using a Sigmoid activation function.

5. The palm key point detection method according to claim 1, wherein: The optimization process of the predicted palm key point coordinates by adaptive residual likelihood estimation includes: Obtaining deformation residuals of the predicted palm key point coordinates to model the distribution of the deformation residuals by maximum likelihood estimation method; Performing error constraints on the predicted palm key point coordinates; Performing a normal distribution constraint on the distribution of the deformation residual; Anatomical constraints are applied to the predicted palm key point coordinates, wherein the weights of the error constraint, the normal distribution constraint, and the anatomical constraint are adjusted by a dynamic temperature coefficient.

6. The palm key point detection method according to any one of claims 1 to 5, characterized in that: The hybrid state space model includes a Mamba-Transformer hybrid architecture, the first network is a bidirectional Mamba network, and the second network is a Windowed Transformer network.

7. The palm key point detection method according to any one of claims 1 to 5, characterized in that: The obtaining of the palm image comprises: Get the original palm image; Dynamic enhancement processing is performed on the original palm image to obtain the palm image.

8. A palm key point detection device, characterized in that: include: An image acquisition module, used for acquiring a palm image; A feature extraction module is used to perform feature extraction processing on the palm image using a pre-trained hybrid state space model to obtain palm print features and knuckle features of the palm image; a feature fusion module, configured to adaptively fuse the palm print features and the finger joint features through the dynamic gating mechanism of the hybrid state space model to obtain fused features of the palm image; A key point prediction module, configured to perform palm key point prediction processing on the palm image according to the fusion features to obtain predicted palm key point coordinates of the palm image; The key point optimization module is used to optimize the predicted palm key point coordinates through an adaptive residual likelihood estimation method to obtain the palm key point coordinates of the palm image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the palm key point detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the palm key point detection method according to any one of claims 1 to 7 are implemented.

Images