Nail key point detection method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a nail key point detection method and device, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to the fields of deep learning, image processing and computer vision. The method comprises: performing feature extraction on point cloud points in a finger point cloud image to obtain point cloud features; determining initial position information of a target key point and key point features according to the point cloud features and average position information of the key points on the finger nail, wherein the average position information is generated according to the labeled position information of the key points in the labeled sample; determining local neighborhood features of the target key point according to the point cloud features, the initial position information and the key point features; and determining target position information of the target key point according to the dependence relationship between the target key point and other key points and the local neighborhood features. The present disclosure improves the precision and robustness of nail key point detection.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of deep learning, image processing and computer vision technology, specifically to a method, apparatus, electronic device and storage medium for nail key point detection. Background Technology

[0002] With the widespread application of 3D vision in fields such as human-computer interaction, virtual reality, and medical testing, the precise localization of key hand parts based on point clouds is becoming increasingly important. Among these, the detection of key points on the fingers and nails, as a crucial step in hand pose estimation and fine interaction, demands high accuracy in localization and structural rationality. For example, in the nail industry, by detecting key points on a client's fingers and nails, customized 3D (3D) wearable nail tips that fit the individual's nail shape can be quickly generated and printed, achieving a "one-of-a-kind" wearing effect. Summary of the Invention

[0003] This disclosure presents a method, apparatus, electronic device, and storage medium for detecting key points on a fingernail.

[0004] According to a first aspect of this disclosure, a fingernail keypoint detection method is provided, comprising: extracting features from point cloud points in a finger point cloud image to obtain point cloud features; determining initial position information and keypoint features of a target keypoint based on the point cloud features and average position information of keypoints on the fingernail, wherein the average position information is generated based on the labeled position information of the keypoint in a labeled sample; determining local neighborhood features of the target keypoint based on the point cloud features, initial position information, and keypoint features; and determining target position information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features.

[0005] According to a second aspect of this disclosure, a fingernail keypoint detection model is provided, comprising: a basic backbone network module for extracting features from point cloud points in a finger point cloud image to obtain point cloud features; a keypoint initial prediction module for determining the initial position information and keypoint features of a target keypoint based on the point cloud features and the average position information of keypoints on the fingernail, wherein the average position information is generated based on the labeled position information of the keypoint in the labeled sample; a local feature fusion module for determining the local neighborhood features of the target keypoint based on the point cloud features, the initial position information, and the keypoint features; and a keypoint graph neural network module for determining the target position information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features; wherein the basic backbone network module, the keypoint initial prediction module, the local feature fusion module, and the keypoint graph neural network module are sequentially connected.

[0006] According to a third aspect of this disclosure, a fingernail keypoint detection apparatus is provided, comprising: a feature extraction unit configured to extract features from point cloud points in a fingernail point cloud image to obtain point cloud features; an initial position determination unit configured to determine initial position information and keypoint features of a target keypoint based on the point cloud features and average position information of keypoints on the fingernail, wherein the average position information is generated based on the labeled position information of the keypoint in a labeled sample; a neighborhood feature determination unit configured to determine local neighborhood features of the target keypoint based on the point cloud features, the initial position information, and the keypoint features; and a target position determination unit configured to determine target position information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features.

[0007] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method as described in any implementation of the first aspect.

[0008] According to a fifth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing a computer to perform a method as described in any implementation of the first aspect.

[0009] According to a sixth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method as described in any implementation of the first aspect.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein: Figure 1 This is an exemplary system architecture diagram to which this disclosure can be applied; Figure 2 This is a flowchart of one embodiment of the nail key point detection method according to the present disclosure; Figure 3 This is a diagram showing the location of key points on the nail; Figure 4 This is a flowchart of another embodiment of the nail key point detection method according to the present disclosure; Figure 5 yes Figure 4A flowchart of an embodiment of step 403; Figure 6 This is a flowchart of yet another embodiment of the nail key point detection method according to the present disclosure; Figure 7 This is a flowchart of yet another embodiment of the nail key point detection method according to the present disclosure; Figure 7-1 This is a diagram illustrating the dependencies between key points; Figure 7-2 This is a schematic diagram illustrating the computation process of graph attention convolution; Figure 8 This is a schematic diagram of the structure of an embodiment of the nail key point detection model; Figure 8-1 This is a schematic diagram of the workflow of the key point initial prediction module; Figure 8-2 This is a schematic diagram of the structure of three multilayer perceptrons; Figure 8-3 This is a schematic diagram of the workflow of the local feature fusion module; Figure 8-4 This is a schematic diagram of the structure of a local multilayer perceptron module; Figure 8-5 This is a schematic diagram of the workflow of the key point graph neural network module; Figure 8-6 This is a schematic diagram of the final prediction multilayer perceptron module; Figure 8-7 This is a flowchart illustrating the workflow of the self-attention head module; Figure 8-8 This is a schematic diagram of the structure of a self-attention feedforward neural network module; Figure 8-9 This is a schematic diagram of the computation process of a multi-head attention network; Figure 9 This is a schematic diagram of a structure of one embodiment of the nail key point detection device according to the present disclosure; Figure 10 This is a block diagram of an electronic device used to implement the nail key point detection method according to the embodiments of this disclosure. Detailed Implementation

[0012] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0013] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0014] Figure 1 An exemplary system frame 100 is shown, to which embodiments of the nail key point detection method or nail key point detection device of the present disclosure can be applied.

[0015] like Figure 1 As shown, system architecture 100 may include terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0016] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications for enabling information communication between the terminal devices 101, 102, and 103 and server 105 can be installed. These applications include cloud storage applications and instant messaging applications.

[0017] Terminal devices 101, 102, and 103 and server 105 can be either hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices, and can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server. When server 105 is software, it can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitation is made here.

[0018] Server 105 can provide various services through built-in applications. For example, users can operate through applications on terminal devices 101, 102, and 103 and send fingernail keypoint detection requests to server 105. Server 105 can receive and process these fingernail keypoint detection requests, performing the following processing: extracting features from the point cloud points in the fingernail point cloud image to obtain point cloud features; determining the initial position information and keypoint features of the target keypoint based on the point cloud features and the average position information of the keypoints on the fingernail, wherein the average position information is generated based on the labeled position information of the keypoint in the labeled samples; determining the local neighborhood features of the target keypoint based on the point cloud features, the initial position information, and the keypoint features; and determining the target position information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features.

[0019] It should be noted that the nail key point detection method provided in this embodiment is generally executed by server 105, and correspondingly, the nail key point detection device is generally set in server 105.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a nail key point detection method according to the present disclosure. The nail key point detection method includes the following steps: Step 201: Extract features from the point cloud points in the finger point cloud image to obtain point cloud features.

[0022] In this embodiment, the execution subject of the nail key point detection method (e.g. Figure 1 The server 105 shown first acquires a finger point cloud image and extracts features from the point cloud points in the image to obtain point cloud features. Here, a point cloud image of the user's finger is first acquired using a depth camera or a structured light 3D acquisition device. A point cloud image (commonly referred to as point cloud data) is a collection of numerous discrete points in three-dimensional space. Each point not only contains positional information but can also carry various additional attributes, such as color information and normal vectors.

[0023] After acquiring the finger point cloud image, the aforementioned execution entity performs coordinate normalization and feature initialization on the 3D coordinates of each point cloud point, thereby mapping the spatial coordinates of each point cloud point to initial point features. Subsequently, the execution entity uses a multi-layer feature extraction unit to perform neighborhood sampling and geometric feature encoding on each point cloud point and its local neighborhood point cloud, extracting fine-grained geometric features such as local curvature, normal vector, and neighborhood distribution density of the point cloud layer by layer, and gradually aggregating them to form multi-scale structural features; at the same time, it also performs channel dimension splicing and nonlinear mapping on the features output from different levels, finally outputting a global point cloud feature with high representational ability. This point cloud feature integrates the local geometric information of the point cloud and the overall spatial structure information of the finger.

[0024] Step 202: Determine the initial position information and key point features of the target key points based on the point cloud features and the average position information of key points on the fingernail.

[0025] In this embodiment, the execution entity determines the initial position information and key point features of the target key point based on the point cloud features and the average position information of the key points on the fingernail. The average position information is generated based on the annotation position information of the key point in the annotation sample.

[0026] There are multiple key points on a fingernail. Based on experience, the number of key points on the fingernail is set to 9 in this embodiment. Of course, the number of key points can be set to other values ​​according to the actual situation. This embodiment does not make a specific limitation on this. The above-mentioned execution entity will first statistically analyze the labeled position coordinates of each key point in a large number of labeled samples to obtain the average position information. The average position information is the average value of the labeled coordinates of the key point in the labeled samples.

[0027] Then, the aforementioned execution entity takes the point cloud features, the average position information of key points, and the original 3D coordinates of the point cloud points in the finger point cloud image as input, uses a multi-head attention mechanism to determine the initial position coordinates of the target key points, and extracts the features of the target key points from the point cloud features to obtain key point features. Each key point on the fingernail is a target key point, meaning that the aforementioned execution entity will determine the initial position information of 9 key points and extract the features of 9 key points.

[0028] Step 203: Determine the local neighborhood features of the target key points based on the point cloud features, initial position information, and key point features.

[0029] In this embodiment, for each target key point, the above-mentioned execution entity will take its initial coordinate in the point cloud image as the center, delimit a local neighborhood range in the point cloud space, and screen the point cloud points falling within this range and their corresponding point cloud features. Then, the above-mentioned execution entity will perform weighted aggregation on the point cloud features in the neighborhood of the target key point and the key point features of the target key point, and at the same time perform spatial constraint encoding on the neighborhood features in combination with the initial position information of the target key point, so as to fully integrate the self-features of the target key point and the geometric structure information of the surrounding local point cloud, and finally obtain the local neighborhood features of each target key point.

[0030] Step 204: Determine the target position information of the target key point according to the dependency relationship and local neighborhood features between the target key point and other key points.

[0031] In this embodiment, multiple key points do not exist independently but are interdependent. Since 9 key points on the fingernail are detected in this embodiment, 9 key points will be taken as an example for illustration. Refer to Figure 3 , Figure 3 shows the positions of 9 key points on the fingernail. As can be seen from Figure 3 , all 9 key points are located on the surface of the fingernail and present a "dry" shape. The positioning of the 9 key points is based on both the edge position of the fingernail contour and their mutual relationship (the positions are not isolated from each other but are interrelated).

[0032] Among them, point 0 is located at the upper right edge (corner) of the fingernail contour, point 2 is located at the upper left edge (corner) of the fingernail contour, and point 1 is located in the middle of the curved surface line formed by point 0 and point 2. Point 3 is located at the lower right edge (corner) of the fingernail contour, point 4 is located in the middle of the rear edge of the fingernail, point 5 is located at the lower left edge (corner) of the fingernail contour, point 6 is on the right edge of the fingernail and is located in the middle between point 0 and point 3. Point 7 is at the intersection of the 1-4 line (the line formed by point 1 and point 4) and the 6-8 line (the line formed by point 6 and point 8). Point 8 is on the left edge of the fingernail and is located in the middle between point 2 and point 5. According to the positional relationships of the above multiple key points, the positional dependency relationships between different key points can be determined.

[0033] The aforementioned execution entity first determines the dependencies between the target keypoint and other keypoints. For example, if the target keypoint is point 8, then points 2 and 5 are dependent on it. Therefore, the execution entity determines the dependency relationship between the target keypoint and points 2 and 5. Then, based on the dependencies between the target keypoint and other keypoints, and combined with the local neighborhood features of the target keypoint, the execution entity uses a graph attention convolutional network to perform position constraint optimization and precise localization calculations on the target keypoint, ultimately obtaining the target keypoint location information that conforms to the physiological structure of the nail and has a precise spatial location.

[0034] The fingernail keypoint detection method disclosed herein first extracts features from point cloud images of the finger to obtain point cloud features. Then, based on the point cloud features and the average position information of keypoints on the fingernail, the initial position information and keypoint features of the target keypoint are determined. Next, based on the point cloud features, initial position information, and keypoint features, the local neighborhood features of the target keypoint are determined. Finally, based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features, the target position information of the target keypoint is determined. This method first achieves initial keypoint localization based on point cloud features and prior average positions, then integrates local neighborhood features to enhance geometric detail representation, and finally combines spatial dependencies between keypoints to complete constraint refinement. This suppresses positional shifts and structural distortions, making the prediction results more consistent with the inherent topological relationships and physiological structure of the fingernail, thus improving the accuracy and robustness of keypoint localization.

[0035] Furthermore, the collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, involved in the technical solutions disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0036] Continue to refer to Figure 4 , Figure 4 A flow 400 of another embodiment of the nail key point detection method according to this disclosure is shown. The nail key point detection method includes the following steps: Step 401: The feature extraction backbone network of the detection model is used to extract the coordinate and attribute information of the point cloud points in the finger point cloud image to obtain fixed-dimensional point cloud features.

[0037] In this embodiment, the execution subject of the nail key point detection method (e.g. Figure 1 The server 105 shown will extract the coordinate and attribute information of the point cloud points in the finger point cloud image through the feature extraction backbone network of the detection model to obtain fixed-dimensional point cloud features. The feature extraction backbone network is a three-dimensional point cloud segmentation network with the last linear layer and the normalized exponential function classification layer removed.

[0038] The detection model is a pre-trained model for detecting key points on a nail. The detection model can include multiple networks, such as a feature extraction backbone network. It can use general 3D point cloud segmentation neural networks such as PTv3 (Point Transformer V3, third-generation point cloud Transformer network) or PointNet (point cloud feature extraction network) as the base model, and remove the last linear layer and the last softmax (normalized exponential function) classifier layer in the base model to obtain the feature extraction backbone network.

[0039] The aforementioned execution entity inputs a finger point cloud image into the feature extraction backbone network. This image can include the 3D coordinates of the point cloud points, their RGB color values, and their normal vectors. When training the neural network to calculate the loss, manually labeled points are also required, with values ​​from 0 to 9. 0 represents background points, 1 represents keypoint number 0, and so on. The feature extraction backbone network extracts features based on the coordinate and attribute information of the point cloud points, outputting fixed-dimensional point cloud features. The dimensions are represented as [batch_size, N, D], where batch_size is the batch size (the number of samples used in one forward propagation and backward update during neural network training), N is the number of point cloud points, and D is the feature vector length (e.g., 1232 for PTv3 output).

[0040] By employing a 3D point cloud segmentation network with the classification correlation layer removed as the backbone, high-dimensional feature extraction is directly performed on the point cloud coordinates and attributes. This preserves the geometric and structural information of the point cloud while avoiding redundant classification interference, thereby improving feature effectiveness and computational efficiency.

[0041] Step 402: Input the pre-created query features of key points into the multi-head attention network and feedforward neural network of the detection model, and output the association features between the target key points and other key points.

[0042] In this embodiment, the query descriptor features associated with the nine key points (i.e., query features) are created during the initialization of the detection model. They are used to gradually learn the interrelationship features of the nine key points during model training. The feature length of the query features is D (which is consistent with the length of the point cloud features).

[0043] The detection model also includes a multi-head attention network and a feedforward neural network. The multi-head attention network is a module that uses a standard multi-head attention mechanism, such as an 8-head attention network. The aforementioned execution entity inputs the query features of the nine key points into the multi-head attention network and feedforward neural network of the detection model, and outputs the association features between the target key points and other key points.

[0044] In some optional implementations of this embodiment, step 402 includes: inputting the query features of the pre-created key points into a multi-head attention network, outputting the association relationship between the target key point and other key points; optimizing the association relationship using a feedforward neural network, and outputting the association features between the target key point and other key points.

[0045] In this implementation, the execution entity inputs the query features of key points into multiple self-attention heads of a multi-head attention network. By calculating the attention scores between the target key point and the other key points in parallel, it adaptively learns and highlights strong spatial dependencies, outputting attention association results that characterize the degree of association between key points. Then, the execution entity uses a feedforward neural network to perform nonlinear mapping, feature selection, and dimensionality regularization on the association results, suppressing redundant association information, strengthening effective topological constraints, and outputting key point association features that combine spatial relevance and structural rationality.

[0046] By modeling the spatial dependencies between key points through a multi-head attention network and then optimizing the features through a feedforward neural network, the topological constraints are strengthened, noise interference is suppressed, and the localization is more in line with physiological laws.

[0047] Step 403: Determine the initial position information and key point features of the target key point based on the point cloud features, average position information, and the correlation features between the target key point and other key points.

[0048] In this embodiment, the execution entity takes the point cloud features, the average position information of key points, the correlation features between the target key point and other key points, and the original three-dimensional coordinates of the point cloud points in the finger point cloud image as input, uses a multi-head attention mechanism to determine the initial position coordinates of the target key point, and extracts the features of the target key point from the point cloud features to obtain the key point features.

[0049] Step 404: Determine the local neighborhood features of the target key points based on the point cloud features, initial position information, and key point features.

[0050] Step 405: Determine the target location information of the target key point based on the dependency relationship between the target key point and other key points and the local neighborhood features.

[0051] Steps 404-405 are basically the same as steps 203-204 in the aforementioned embodiment. For specific implementation methods, please refer to the aforementioned description of steps 203-204, which will not be repeated here.

[0052] In some optional implementations of this embodiment, the loss value during the training process of the detection model is calculated in the following ways: the initial prediction loss is calculated based on the labeled location information and initial location information of the target key points; the target prediction loss is calculated based on the labeled location information and target location information; and the loss value is calculated based on the initial prediction loss, the first weight corresponding to the initial prediction loss, the target prediction loss, and the second weight corresponding to the target prediction loss.

[0053] During model training, a loss function is typically used to calculate the loss value, thereby quantifying the model's prediction performance and updating the model's parameters in reverse using the calculated loss value.

[0054] In this implementation, to simultaneously optimize the initial prediction and the refinement process of the graph neural module, the total loss of the detection model can consist of two parts: the initial prediction loss and the target prediction loss. It can be calculated using the following formula: ; in, It is the initial position information of the i-th key point predicted by the detection model, and it is the normalized three-dimensional coordinate value; It represents the annotation location information of the i-th keypoint, which is manually annotated, and its normalized 3D coordinate value. The calculation method is as follows: =coord_normal i =(coord i - coord_mean i ) / coord_std i ; Among them, coord i The coord_mean refers to the original 3D coordinates of the i-th keypoint. i The coord_std refers to the average position information of the i-th keypoint (i.e., the average point cloud coordinates of the i-th keypoint obtained by preprocessing all samples involved in model training); i The standard deviation of the coordinates of the i-th key point (i.e., the standard deviation of the point cloud coordinates of the i-th key point obtained by preprocessing all samples involved in training).

[0055] It should be noted that the mean and standard deviation of the coordinates of the 9 key points, coord_mean and coord_std, are calculated separately for each control point based on the control point number of all samples involved in the model training. coord_normal is calculated separately for each control point.

[0056] Target Predicted Loss It can be calculated using the following formula: ; in, It is the target location information of the i-th key point, and it is the normalized three-dimensional coordinate value.

[0057] Total loss It can be calculated using the following formula: ; in, It is the first weight. It is the second weight. and These are training hyperparameters, with values ​​set in the range [0,1]. At the beginning of training... It can be automatically set to a larger value (e.g.: =0.5、 =0.5), after training for a period of time The parameters can be adjusted to normal (e.g.: =0.2、 =0.8), which helps the algorithm converge faster.

[0058] By calculating the losses of the initial prediction and the final position separately, and then combining them with different weights for weighted fusion, multi-level supervision and constraints are achieved for the key point localization process, which effectively guides the model to optimize learning and improves the stability and accuracy of key point localization.

[0059] from Figure 4 It can be seen from this that, with Figure 2 Compared to the corresponding embodiments, the fingernail key point detection method in this embodiment emphasizes the steps of extracting point cloud features and determining the initial position information of the target key points. By introducing the correlation features between key points and combining point cloud features with prior average positions for collaborative localization, the method makes full use of spatial topological constraints, thereby improving the accuracy and structural rationality of key point localization.

[0060] Continue to refer to Figure 5 , Figure 5 It shows Figure 4 The process 500 of the embodiment of step 403 includes: Step 501: Normalize the initial coordinate information of the point cloud points, and convert the normalized coordinates into coordinate embedding features through the multilayer perceptron of the detection model.

[0061] The aforementioned execution entity first normalizes the initial 3D coordinates of the input point cloud points, mapping the coordinate values ​​to a preset numerical range to eliminate interference caused by differences in coordinate scale. The initial coordinate information can be normalized using the following formula: coord_normal i =(coord i - coord_mean i ) / coord_std i ; Among them, coord i The coord_mean refers to the original 3D coordinates of the i-th keypoint. i (referring to the average position information of the i-th key point); coord_std i This refers to the standard deviation of the coordinates of the i-th key point.

[0062] Then, the aforementioned execution entity will input the normalized coordinates into the multilayer perceptron in the detection model. After multilayer linear transformation and nonlinear activation operation, the low-dimensional coordinate information is mapped into a high-dimensional feature space to obtain coordinate embedding features with representational capabilities.

[0063] Step 502: Fuse the coordinate embedding features with the point cloud features to obtain the fused point cloud features.

[0064] The aforementioned execution entity will stitch together and fuse coordinate embedding features and point cloud features along the feature dimension. Through feature weighting and channel alignment operations, coordinate spatial information and semantic feature information complement each other, enhancing the feature's ability to express spatial location and local geometric structure, ultimately resulting in a fused point cloud feature that combines coordinate information and contextual semantic information.

[0065] Step 503: Using the correlation features between the target key point and other key points as the query vector, perform cross-attention calculation on the fused point cloud features to generate the correlation feature information between the target key point and other key points.

[0066] The aforementioned execution entity uses the correlation features between key points as the query vector and the fused point cloud features as the key and value vectors to perform cross-attention calculations. It then adaptively learns attention weights to mine the matching correlations between key points and local point cloud features, thereby filtering out effective geometric and semantic information and generating relevant feature information that combines spatial correlation and local features.

[0067] Step 504: Input the relevant feature information into the feedforward neural network and output the initial position information of the target key points and the corresponding key point features.

[0068] The aforementioned execution entity inputs the relevant feature information obtained through cross-attention calculation into the feedforward neural network. Through multi-layer linear transformation and non-linear activation function, it performs feature extraction and abstract expression, filters, enhances and regularizes the dimensions of the features, eliminates redundant information, and finally outputs the initial position information of the target key points, as well as the corresponding highly representative key point features.

[0069] In this embodiment, coordinate normalization and embedding transformation are combined with point cloud feature fusion to enhance spatial representation. Then, cross-attention is used to mine related information, and finally, the localization result is output through a feedforward network, which improves the feature representation capability and spatial matching accuracy, making the initial key point localization more accurate and the structure more reasonable.

[0070] Continue to refer to Figure 6 , Figure 6 A flowchart 600 of another embodiment of the nail key point detection method according to the present disclosure is shown. The nail key point detection method includes the following steps: Step 601: Extract features from the point cloud points in the finger point cloud image to obtain point cloud features.

[0071] Step 602: Determine the initial position information and key point features of the target key points based on the point cloud features and the average position information of key points on the fingernail.

[0072] Steps 601-602 are basically the same as steps 201-202 in the aforementioned embodiments. For specific implementation methods, please refer to the aforementioned description of steps 201-202, which will not be repeated here.

[0073] Step 603: Normalize the initial coordinate information of the point cloud points to obtain normalized coordinates.

[0074] In this embodiment, the execution subject of the nail key point detection method (e.g. Figure 1 The server 105 shown will first normalize the initial 3D coordinates of the input point cloud points, that is, map the coordinate values ​​to a preset numerical range to eliminate interference caused by differences in coordinate scale. The initial coordinate information can be normalized using the following formula: coord_normal i =(coord i - coord_mean i ) / coord_std i ; Among them, coord i The coord_mean refers to the original 3D coordinates of the i-th keypoint. i The average position information of the i-th key point; coord_std i This refers to the standard deviation of the coordinates of the i-th key point.

[0075] Step 604: Input the point cloud features, initial position information, key point features and normalized coordinates into two multilayer perceptrons with shared parameters, and output the aggregated key point features of the target key points and the neighborhood features of the target key points respectively.

[0076] In this embodiment, the aforementioned execution entity concatenates the point cloud features, the initial position information of the target key points, the corresponding key point features, and the normalized point cloud coordinates to form a unified input feature. This concatenated feature is then fed into two sets of weight-shared, structurally identical multilayer perceptrons. Feature extraction, compression, and aggregation are performed through multilayer linear transformations and nonlinear activation functions, respectively. Finally, one set outputs the globally aggregated target key point features, while the other set outputs the target key point neighborhood features representing local structural information.

[0077] Step 605: The aggregated key point features and neighborhood features are spliced ​​and fused to obtain the local neighborhood features of the target key point.

[0078] In this embodiment, the aforementioned execution entity directly concatenates and fuses the aggregated keypoint features and neighborhood features output by the shared-parameter multilayer perceptron along the feature channel dimension, enabling the semantic information of the keypoint itself and the spatial structure information of the local neighborhood to complement each other. Thus, through feature dimension alignment and information enhancement, a local neighborhood feature of the target keypoint is formed that combines global representation and local structure.

[0079] Step 606: Determine the target location information of the target key point based on the dependency relationship between the target key point and other key points and the local neighborhood features.

[0080] Step 606 is basically the same as step 204 in the aforementioned embodiment. For the specific implementation method, please refer to the aforementioned description of step 204, which will not be repeated here.

[0081] from Figure 6 It can be seen from this that, with Figure 2 Compared to the corresponding embodiments, the fingernail key point detection method in this embodiment emphasizes the step of determining the local neighborhood features of the target key point. It eliminates scale differences through coordinate normalization, and uses a shared parameter multilayer perceptron to aggregate the key point's own features and neighborhood features respectively, and then performs splicing and fusion. This method can fully explore local spatial structure information, enhance feature expression capabilities, and improve the stability and accuracy of subsequent key point localization.

[0082] Continue to refer to Figure 7 , Figure 7 A flow 700 of another embodiment of the nail key point detection method according to the present disclosure is shown. The nail key point detection method includes the following steps: Step 701: Extract features from the point cloud points in the finger point cloud image to obtain point cloud features.

[0083] Step 702: Determine the initial position information and key point features of the target key points based on the point cloud features and the average position information of key points on the fingernail.

[0084] Step 703: Determine the local neighborhood features of the target key point according to the point cloud features, initial position information, and key point features.

[0085] Steps 701 - 703 are basically the same as steps 201 - 203 in the foregoing embodiment. The specific implementation manners can refer to the description of steps 201 - 203 above and will not be elaborated herein.

[0086] Step 704: Input the local neighborhood features and the dependency graph into the graph attention convolutional network of the detection model, and output the spatial dependency weights between the target key point and other key points.

[0087] In this embodiment, the execution subject of the fingernail key point detection method (such as Figure 1 the server 105 shown) will input the local neighborhood features and the dependency graph into the graph attention convolutional network of the detection model, and output the spatial dependency weights between the target key point and other key points. Among them, the dependency graph is constructed according to the dependency relationships between key points.

[0088] All 9 key points in this embodiment are located on the surface of the fingernail and present a "dry" shape. The positioning of the 9 key points is based on both the fingernail contour edge positions and their mutual correlations (the positions are not isolated from each other but are mutually related). Among them, point 0 is located at the upper right edge (corner) of the fingernail contour, point 2 is located at the upper left edge (corner) of the fingernail contour, and point 1 is located exactly in the middle of the curved surface line formed by point 0 and point 2. Point 3 is located at the lower right edge (corner) of the fingernail contour, point 4 is located exactly in the middle of the posterior edge of the fingernail, point 5 is located at the lower left edge (corner) of the fingernail contour, point 6 is on the right edge of the fingernail and is located exactly in the middle between point 0 and point 3. Point 7 is at the intersection of the 1 - 4 line (i.e., the line formed by point 1 and point 4) and the 6 - 8 line (i.e., the line formed by point 6 and point 8). Point 8 is on the left edge of the fingernail and is located exactly in the middle between point 2 and point 5.

[0089] The above execution subject can construct a dependency graph according to the dependency relationships between the above 9 key points. Refer to Figure 7-1 , Figure 7-1 is the dependency graph of key points. It can be seen from Figure 7-1 that for the 9 key points, each key point corresponds to 1 node in the graph neural network. The node attribute is the local neighborhood feature. The dependency relationships between nodes are connected by directed edges, and the weight coefficient of the directed edge represents the degree of dependence on the adjacent edge. For example, Figure 7-1 in: Node 1, its dependent inputs are point 0, point 2, and itself, and the influence strength weight coefficients are 0.25, 0.25, and 0.5 respectively. Its influence strength weight coefficient on point 7 is 0.2.

[0090] Furthermore, the graph attention convolutional network GATConv is a multi-head graph attention convolution, and its specific implementation details are as follows: Let any node v in the graph i The feature vector corresponding to the l-th layer is , ∈R d(l) d(l) represents the characteristic length of a node, assuming the center node is v. i Let's say the neighbor node v j to v i weighting coefficient e ij for: ; in, These are the weight parameters for the feature transformation of nodes in this layer, and `a()` is a function that calculates the correlation between two nodes. In principle, it can be used to calculate the correlation between any node in the graph, restricting it to its first-order neighborhood, with a weighting coefficient `a`. ij for: ; The complete formula for calculating the weighting coefficient is as follows: ; Where α is the weight coefficient, the above formula ensures that the sum of the weight coefficients of all neighbors is 1. The neighboring node messages of node i are then concatenated and fused for calculation: ; The above formula represents the multi-head attention mechanism, where K is the total number of attention heads.

[0091] Continue to refer to Figure 7-2 , Figure 7-2 This is a schematic diagram illustrating the computation process of graph attention convolution. Figure 7-2 The left image illustrates the attention weight calculation process, specifically: first, the node features... , W is obtained by performing linear transformations on the weight matrix W respectively. W The transformed features are concatenated and then input into the attention calculation layer, where they are processed by softma. Normalization results in the final output attention coefficient a between nodes i and j. ij This is used to quantify the importance of the associations between nodes.

[0092] Figure 7-2 The right-hand diagram illustrates the feature aggregation and update process, specifically: with the central node... For example, based on the calculated self-attention coefficient and neighboring nodes Attention coefficient The neighborhood features are weighted and summed, and then feature fusion is completed through concat or average operations, finally outputting the updated node features. This enables adaptive enhancement of node features and topological relationship modeling.

[0093] By employing two GATConv modules, the receptive field of the second-order neighborhood can be obtained, enabling the interrelationships of the nine key points to be learned through message exchange.

[0094] Step 705: The position information of the target key point and the key points adjacent to the target key point are fused to obtain the fused position information.

[0095] In this embodiment, the execution entity will perform feature splicing on the location information of the target key point and the location information of the adjacent key points in its spatial domain, and perform weighted fusion on the adjacent node information in combination with weight allocation, so as to fully integrate the spatial position association between the central key point and the neighboring key points, and finally obtain the fused location information containing local spatial structure constraints.

[0096] Step 706: Based on the dependency relationship, spatial dependency weight, and fused position information, geometric constraints and position refinement are applied to the initial position information to obtain the normalized position information of the target key points.

[0097] In this embodiment, the aforementioned execution entity will apply weighted constraints to the fused location information based on the dependencies between key points and the spatial dependency weights, and introduce topological structure and geometric relationship regularization guidance to correct deviations and optimize space of the initial location information, eliminate abnormal offsets, achieve location refinement, and finally output standardized target key point location information that conforms to structural logic.

[0098] Step 707: Perform inverse normalization calculation on the normalized location information to obtain the target location information.

[0099] In this embodiment, the aforementioned execution entity performs a reverse mapping operation on the refined normalized position information based on the scaling factor and offset used in the previous coordinate normalization, restoring the standardized coordinates in the feature space to the original point cloud coordinate space, eliminating the influence of scale normalization, and finally obtaining the true position information of the target key points that conforms to the original spatial scale.

[0100] In some optional implementations of this embodiment, step 707 includes: calculating the coordinate mean and coordinate standard deviation based on the initial coordinate information of all point cloud points; performing inverse normalization calculation based on the normalized position information, coordinate mean, and coordinate standard deviation to obtain the target position information.

[0101] In this implementation, the aforementioned execution entity first calculates the coordinate mean and standard deviation based on the initial coordinate information of all point cloud points, and then performs inverse normalization calculation using the following formula: coord_normal i =(coord i - coord_mean i ) / coord_std i ; Among them, coord i The coord_mean refers to the original 3D coordinates of the i-th keypoint. i The average position information of the i-th key point; coord_std i This refers to the standard deviation of the coordinates of the i-th key point.

[0102] Therefore, by calculating the mean and standard deviation based on the overall point cloud coordinates and performing inverse normalization, the refined normalized position can be restored to the original coordinate space, ensuring that the position values ​​conform to the true scale, making the final positioning results accurate and reliable, while maintaining the consistency of the geometric structure, thus improving the practicality and authenticity of the detection results.

[0103] Step 708: Determine the target key points from the point cloud points of the finger point cloud image based on the nearest neighbor algorithm and the target location information.

[0104] In this embodiment, due to the existence of errors, the predicted coordinates of the nine key points generally will not fall exactly on the point set of the finger point cloud image. Therefore, the aforementioned execution entity will use the nearest neighbor algorithm to find the nearest point to each of the nine predicted points as the final predicted point, thereby accurately locating each target key point in the finger point cloud.

[0105] By employing the nearest neighbor algorithm combined with refined target location information to match point cloud points, the predicted location can be quickly and accurately mapped onto the real point cloud data, ensuring accurate key point positioning that closely matches the actual point cloud distribution, thus improving the practicality and stability of the detection results.

[0106] from Figure 7 It can be seen from this that, with Figure 2 Compared to the corresponding embodiments, the fingernail key point detection method in this embodiment emphasizes the step of determining the target location information of the target key point. By modeling spatial dependency weights through graph attention network, combining neighborhood location fusion and geometric constraint to refine the position, and then restoring the real coordinates through inverse normalization, the positioning deviation can be effectively optimized, the rationality of the topology structure can be enhanced, and the final key point position can be more accurate and closer to the real spatial distribution.

[0107] Continue to refer to Figure 8 , Figure 8This is a schematic diagram of the nail keypoint detection model structure. The nail keypoint detection model 800 includes: a basebackbone module, a keypointspredictInit module, a local feature fusion module, and a keypoints graph neural network module. The basebackbone module, keypointspredictInit module, local feature fusion module, and keypoints graph neural network module are connected in sequence. That is, these modules are connected in series in order. The finger point cloud image captured by the structured light camera is used as the input of the nail keypoint detection model, and the model predicts and outputs the three-dimensional coordinate values ​​(XYZ coordinates) of 9 key points of the nail.

[0108] The baseback network module is used to extract features from the finger point cloud image, obtaining point cloud features. Here, a general 3D point cloud neural network such as PTv3 or PointNet can be used as the base model, and the final linear layer and softmax classifier layer of the base model are removed to obtain the BaseBackbone module. The input to this module is the finger point cloud image, which includes the 3D coordinates, RGB color values, and normal vectors of the point clouds. The output is the extracted point cloud features, with dimensions [batch_size, N, D], where batch_size is the batch size, i.e., the number of samples used in one forward propagation and backward update during neural network training, N is the number of point clouds, and D is the feature vector length.

[0109] The parameters of the BaseBackbone module are initialized using the weight parameters of the general pre-trained model downloaded from the official website (such as PointNet, PointTransformerV3, etc.). A tiered learning rate is used (a smaller learning rate is used for the BaseBackbone module, and 10 times that learning rate is used for other modules). The number of training epochs is set to 1000. OneCycleLR (with warmup and cosine annealing) is used to control the learning rate during the training process.

[0110] The neural network model was trained using a manually labeled control point dataset of 4405 samples, which was randomly divided into three sets: a training set of 3105 (70%), a test set of 665 (15%), and a validation set of 665 (15%). During training, the training loss decreased with the learning rate (indicating good convergence), and both the distance deviation (DPD) and loss on the validation set decreased. Furthermore, the distance deviation (DPD) of all nine control points on the validation set also decreased.

[0111] It should be noted that DPD (Deviation of Point Distance) measures the overall average distance deviation between the predicted control points and manually labeled points on the test data sample. It is defined as follows: the labeling of the nine control points is highly arbitrary; the nine points labeled by different people (or by the same person at different times) will often not be exactly the same. This means that control point prediction cannot have a precise solution, and the predicted points will inevitably have some deviation. To measure this deviation, we use a custom distance deviation calculation method, DPD: DPD= ; Where N represents the total number of samples in the test set, and 9 is the number of control points to be predicted. This indicates that the algorithm predicts the XYZ coordinates of the j-th control point for the i-th sample. This represents the XYZ coordinates of the j-th control point of the i-th manually labeled sample.

[0112] The keypoint initial prediction module is used to determine the initial position information and keypoint features of the target keypoints based on the point cloud features and the average position information of keypoints on the fingernail. The average position information is generated based on the labeled position information of the keypoints in the labeled samples.

[0113] The workflow of the KeyPointsPredictInit module is as follows: Figure 8-1 As shown, the input to this module is the initial 3D coordinates (coords) of the finger point cloud. The finger point cloud coordinates are normalized to eliminate scale differences. Then, a multilayer perceptron (MLP_coord) is used to map the normalized coordinates into a high-dimensional coordinate embedding feature (coord_emb). The coord_emb is then fused with the base feature (base_feat) of the point cloud output from the backbone network to obtain a fused feature that combines spatial location and semantic information.

[0114] Then, taking the fused features and the target key point association features Q_prime_target as input, the query vector Q, key vector K, and value vector V are generated through three sets of linear layers, respectively. These are then input into a multi-head cross-attention computing network to mine the association information between key points and point cloud features. After being normalized by the linear layers, the attention feature attn_output is output and added to the residual of the input features.

[0115] Attention features are normalized by LayerNorm, connected to the residual by the feedforward neural network FFN_cross, and normalized again by a second layer to output keypoint features keypoints_feat. keypoints_feat is then input into the multilayer perceptron MLP_predInit to predict the keypoint position offset delta. Finally, the offset is added to the keypoint base coordinate keypoints_base_coord to output the initial predicted positions of the 9 keypoints predict_init_keypoints9, and keypoints_feat is also output for subsequent refinement of the network.

[0116] Continue to refer to Figure 8-2 It illustrates the structure of three multilayer perceptrons: MLP_coord, FFN_cross, and MLP_predInit, specifically: The coordinate embedding multilayer perceptron module MLP_coord takes 3-dimensional coordinate features as input, and then passes them through a linear layer Linear(3,d) to increase the dimensionality. Next, it passes through a LayerNorm normalization layer and a Gaussian error linear unit GELU to complete feature normalization and non-linear activation. Then, it uses Dropout to randomly deactivate the features to improve robustness. Finally, it passes through a linear layer Linear(d,d) to preserve the dimensionality and outputs the high-dimensional coordinate embedding feature coord_emb.

[0117] The cross-feedforward neural network (FFN_cross) takes attention features of dimension d as input and first processes them through a linear layer (Linear(d,d)). 2) Dimensional expansion of features; feature enhancement using LayerNorm and GELU; overfitting suppression via Dropout; finally, a linear layer (Linear(d)) is used. 2,d) Reduce the dimensionality back to the original dimension and output the features optimized by cross-attention.

[0118] The initial prediction multilayer perceptron module MLP_predInit takes keypoint features of dimension d as input and passes them through a linear layer Linear(d, d...) 2) Perform feature extraction and expansion; sequentially execute LayerNorm normalization and GELU nonlinear transformation; after randomly discarding some neurons via Dropout; and then perform linear layer (d... 2,d) Map the keypoint position offset delta and keypoint feature keypoints_feat to achieve initial keypoint position prediction.

[0119] The local feature fusion module is used to determine the local neighborhood features of the target key points based on point cloud features, original point cloud coordinates, initial predicted location information and key point features.

[0120] The workflow of the LocalFeatureFusion module is as follows: Figure 8-3 As shown, it includes the following steps: 1. Neighborhood point retrieval and feature acquisition: Taking the initial predicted keypoint position predict_init_keypoints9, the point cloud basic feature base_feat, and the original point cloud coordinates coords as input, the K nearest neighbor algorithm is used to retrieve the K nearest neighbor points of each initial keypoint, and the neighbor coordinates X_neigh and the neighborhood feature F_neigh are concatenated.

[0121] 2. Relative position encoding and feature fusion: Normalize X_neigh and calculate the relative position encoding (RPE) based on the initial predicted position; concatenate the RPE with F_neigh to obtain E_neigh, and simultaneously pad the keypoints_feat feature with zeros to obtain E_core.

[0122] 3. Feature Encoding and Aggregation: E_neigh and E_core are encoded by MLP_local respectively to output neighborhood feature Z_neigh and core feature Z_core; after concatenating the two, global max pooling is performed through max_pool to extract global context information F_max.

[0123] 4. Coordinate Transformation and Feature Output: The initial predicted position is converted into global absolute coordinates key_points_abs, concatenated with F_max, and then processed by a linear layer and layer normalization to finally output the local neighborhood features F_local of the target key points, providing input for the subsequent graph attention refinement network.

[0124] Continue to refer to Figure 8-4 The diagram illustrates the structure of the Local Multilayer Perceptron (MLP_local) block, which is used for encoding and enhancing local neighborhood features of keypoints in finger point clouds. The overall process is as follows: The input features are first normalized by LayerNorm to improve training stability; then, the input dimension is mapped to a high-dimensional feature d through a linear layer (Linear(d+3,d)); next, the GELU activation function is applied to introduce a non-linear transformation to enhance feature expressiveness; then, a Dropout layer is used to randomly deactivate some neurons to suppress overfitting and improve model robustness; finally, the feature dimension is regularized and information is refined through a linear layer (Linear(d,d)) to output the final encoded local features.

[0125] The keypoint graph neural network module is used to determine the target location information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and local neighborhood features.

[0126] The workflow of the KeyPointsGNN module is as follows: Figure 8-5 As shown, it includes the following steps: The input contains a graph adjacency matrix (dependency graph) and local neighborhood features F_local; the network adopts a dual graph attention convolution (GATConv) structure, and after each round of GATConv, layer normalization (LayerNorm) and GELU activation function are sequentially applied to enhance feature representation and improve training stability; the output features of the two GATConv and the input features are concatenated (cat) and linearly transformed (Linear), and then the residuals are added with the original F_local. Finally, the final predicted coordinates of 9 keypoints are output through a multilayer perceptron (MLP_predFinal), thereby achieving accurate correction of the initial keypoint positions.

[0127] Continue to refer to Figure 8-6 The diagram illustrates the structure of the final prediction multilayer perceptron module, MLP_predFinal. This module, serving as the core perceptron network for refining the final keypoint positions, consists of multiple layers of linear transformation, normalization, activation, and regularization units connected in series. The specific process is as follows: Input features are first standardized using LayerNorm (layer normalization) to improve training stability; then, a linear layer (Linear(d, d / 2)) is used for initial dimensionality reduction; subsequently, the GELU activation function is introduced to introduce nonlinear transformation, enhancing feature representation; then, a Dropout layer randomly deactivates some neurons to suppress overfitting; next, a linear layer (Linear(d / 2, d / 4)) is used to further compress the feature dimensions and extract key information; again, GELU activation and Dropout regularization are applied; finally, a linear layer (Linear(d / 4, 3)) is used to map the feature dimensions to 3D, outputting the final predicted coordinate features of the keypoints, achieving accurate correction of the initial positions.

[0128] In some optional implementations of this embodiment, the aforementioned nail keypoint detection model further includes a self-attention head module, located between the basic backbone network module and the keypoint initial prediction module. The self-attention head module includes a multi-head attention network and a feedforward neural network. The multi-head attention network is used to generate the association relationship between the target keypoint and other keypoints based on the query features of pre-created keypoints; the feedforward neural network is used to optimize the association relationship and generate association features between the target keypoint and other keypoints.

[0129] The workflow of the SelfAttentionHead module is as follows: Figure 8-7 As shown, it includes the following steps: 1. Input and preprocessing: The input is the association query descriptor features Q_target(9,D) of 9 key points. First, the feature distribution is standardized by layer normalization (LayerNorm) to improve training stability.

[0130] 2. Multi-head attention computation: Normalized features are input into the multi-head attention network, and query vector Q, key vector K, and value vector V are generated through three sets of linear layers. These are then input into the multi-head attention computation unit of the multi-head attention network to model the correlation and dependency relationships between key points and mine global semantic association information.

[0131] 3. Feature Enhancement and Output: The attention calculation result is added to the input feature residual. After layer normalization (LayerNorm), the feedforward neural network FFN_self is connected to the secondary residual, and the enhanced key point query feature Q_prime_target is output, which provides accurate query feature input for the subsequent cross-attention initial detection network.

[0132] Continue to refer to Figure 8-8 It shows the structure of the self-attention feedforward neural network module FFN_self, which is a feedforward neural network submodule in the self-attention network, used for deep enhancement of keypoint association query features. The specific process is as follows: The input features first pass through a linear layer Linear(d,d) 2) The linear layer performs dimensionality increase, expanding the feature space; then LayerNorm (layer normalization) is applied to standardize the feature distribution and improve training stability; a non-linear transformation is introduced through the GELU activation function to enhance feature representation ability; then a Dropout layer randomly deactivates some neurons to suppress overfitting and improve model robustness; finally, the linear layer (d... 2,d) The linear layer completes dimensionality reduction and information refinement, outputting enhanced features to provide high-quality input for subsequent attention calculations.

[0133] Continue to refer to Figure 8-9This diagram illustrates the computational process of a multi-head attention network. It takes a query vector Q, a key vector K, and a value vector V as input, and each is mapped to a feature dimension through an independent linear layer, preparing for multi-head attention computation. Then, the linearly transformed Q, K, and V are split into H independent sub-feature heads through a Split operation. These sub-feature heads are then fed in parallel into a Scaled Dot-Product Attention module to calculate the attention weight and weighted features for each head, achieving multi-dimensional and multi-view association modeling. The outputs of the H attention heads are then merged into a complete feature through a Concat operation. Finally, a linear layer performs dimension normalization, outputting the final multi-head attention computation result.

[0134] The formula for calculating Attention(Q, K, V) is as follows: ; Where, d k Indicates the length of the feature.

[0135] Further reference Figure 9 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a nail key point detection device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0136] like Figure 9 As shown, the fingernail keypoint detection device 900 of this embodiment includes: a feature extraction unit 901, an initial position determination unit 902, a neighborhood feature determination unit 903, and a target position determination unit 904. The feature extraction unit 901 is configured to extract features from point cloud points in a finger point cloud image to obtain point cloud features; the initial position determination unit 902 is configured to determine the initial position information and keypoint features of a target keypoint based on the point cloud features and the average position information of keypoints on the fingernail, wherein the average position information is generated based on the labeled position information of the keypoint in the labeled samples; the neighborhood feature determination unit 903 is configured to determine the local neighborhood features of the target keypoint based on the point cloud features, the initial position information, and the keypoint features; the target position determination unit 904 is configured to determine the target position information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features.

[0137] In this embodiment, the specific processing of the feature extraction unit 901, the initial position determination unit 902, the neighborhood feature determination unit 903, and the target position determination unit 904 in the nail key point detection device 900, and the resulting technical effects, can be found in the following references: Figure 2The relevant descriptions of steps 201-204 in the corresponding embodiments will not be repeated here.

[0138] In some optional implementations of this embodiment, the feature extraction unit 901 is further configured to: extract the coordinate information and attribute information of the point cloud points in the finger point cloud image through the feature extraction backbone network of the detection model to obtain fixed-dimensional point cloud features; wherein, the feature extraction backbone network is a three-dimensional point cloud segmentation network with the last linear layer and the normalized exponential function classification layer removed.

[0139] In some optional implementations of this embodiment, the nail key point detection device 900 further includes: an association feature determination unit, configured to input the query features of pre-created key points into the multi-head attention network and feedforward neural network of the detection model, and output the association features between the target key point and other key points; and an initial position determination unit 902 including: an initial position determination subunit, configured to determine the initial position information and key point features of the target key point based on point cloud features, average position information and the association features between the target key point and other key points.

[0140] In some optional implementations of this embodiment, the association feature determination unit is further configured to: input the query features of pre-created key points into a multi-head attention network, output the association relationship between the target key point and other key points; optimize the association relationship using a feedforward neural network, and output the association features between the target key point and other key points.

[0141] In some optional implementations of this embodiment, the initial position determination subunit is further configured to: normalize the initial coordinate information of the point cloud points, and convert the normalized coordinates into coordinate embedding features through the multilayer perceptron of the detection model; fuse the coordinate embedding features with the point cloud features to obtain fused point cloud features; use the correlation features between the target key point and other key points as a query vector, perform cross-attention calculation on the fused point cloud features to generate relevant feature information between the target key point and other key points; input the relevant feature information into the feedforward neural network, and output the initial position information of the target key point and the corresponding key point features.

[0142] In some optional implementations of this embodiment, the neighborhood feature determination unit 903 is further configured to: normalize the initial coordinate information of the point cloud points to obtain normalized coordinates; input the point cloud features, initial position information, key point features and normalized coordinates into two multilayer perceptrons with shared parameters, and output the aggregated key point features and neighborhood features of the target key point respectively; and splice and fuse the aggregated key point features and neighborhood features to obtain the local neighborhood features of the target key point.

[0143] In some optional implementations of this embodiment, the target location determination unit 904 includes: a weight determination subunit, configured to input local neighborhood features and a dependency graph into the graph attention convolutional network of the detection model, and output spatial dependency weights between the target keypoint and other keypoints, wherein the dependency graph is constructed based on the dependency relationships between keypoints; a fusion subunit, configured to fuse the location information of the target keypoint and the keypoints adjacent to the target keypoint to obtain fused location information; a normalization subunit, configured to perform geometric constraints and location refinement on the initial location information based on the dependency relationships, spatial dependency weights, and fused location information to obtain normalized location information of the target keypoint; and an inverse normalization subunit, configured to perform inverse normalization calculation on the normalized location information to obtain the target location information.

[0144] In some optional implementations of this embodiment, the inverse normalization subunit is further configured to: calculate the coordinate mean and coordinate standard deviation based on the initial coordinate information of all point cloud points; and perform inverse normalization calculation based on the normalized position information, coordinate mean, and coordinate standard deviation to obtain the target position information.

[0145] In some optional implementations of this embodiment, the nail key point detection device 900 further includes: a loss value calculation unit, configured to calculate an initial prediction loss based on the annotation position information and initial position information of the target key point; calculate a target prediction loss based on the annotation position information and target position information; and calculate a loss value based on the initial prediction loss, a first weight corresponding to the initial prediction loss, the target prediction loss, and a second weight corresponding to the target prediction loss.

[0146] In some optional implementations of this embodiment, the fingernail key point detection device 900 further includes a key point determination unit, configured to determine target key points from the point cloud points of the finger point cloud image based on the nearest neighbor algorithm and target position information.

[0147] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0148] Figure 10A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0150] like Figure 10 As shown, device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1002 or a computer program loaded into random access memory (RAM) 1003 from storage unit 1008. The RAM 1003 may also store various programs and data required for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via bus 1004. Input / output (I / O) interface 1005 is also connected to bus 1004.

[0151] Multiple components in device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of monitors, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] The computing unit 1001 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the nail keypoint detection method. For example, in some embodiments, the nail keypoint detection method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the nail keypoint detection method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform a nail keypoint detection method by any other suitable means (e.g., by means of firmware).

[0153] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0154] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0155] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0156] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0157] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0158] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for detecting key points on a nail, comprising: Feature extraction is performed on the point cloud points in the finger point cloud image to obtain point cloud features; Based on the point cloud features and the average position information of key points on the fingernail, the initial position information and key point features of the target key points are determined, wherein the average position information is generated based on the annotation position information of the key points in the annotation samples; Based on the point cloud features, the initial position information, and the key point features, determine the local neighborhood features of the target key point; Based on the dependency relationship between the target key point and other key points and the local neighborhood features, the target location information of the target key point is determined.

2. The method according to claim 1, wherein, The step of extracting features from the point cloud image of a finger to obtain point cloud features includes: The feature extraction backbone network of the detection model extracts the coordinate and attribute information of the point cloud points in the finger point cloud image to obtain fixed-dimensional point cloud features. The feature extraction backbone network is a 3D point cloud segmentation network with the last linear layer and the normalized exponential function classification layer removed.

3. The method according to claim 1, wherein, The method further includes: The pre-created query features of the key points are input into the multi-head attention network and feedforward neural network of the detection model, and the correlation features between the target key points and other key points are output; and The step of determining the initial position information and key point features of the target key points based on the point cloud features and the average position information of key points on the fingernail includes: Based on the point cloud features, the average position information, and the correlation features between the target key point and other key points, the initial position information and key point features of the target key point are determined.

4. The method according to claim 3, wherein, The process involves inputting the pre-created query features of the key points into the multi-head attention network and feedforward neural network of the detection model, and outputting the association features between the target key points and other key points, including: The query features of the pre-created key points are input into the multi-head attention network, and the relationship between the target key points and other key points is output. The feedforward neural network is used to optimize the association relationship and output the association features between the target key point and other key points.

5. The method according to claim 3, wherein, The step of determining the initial position information and key point features of the target key point based on the point cloud features, the average position information, and the correlation features between the target key point and other key points includes: The initial coordinate information of the point cloud points is normalized, and the normalized coordinates are converted into coordinate embedding features by the multilayer perceptron of the detection model. The coordinate embedding features are fused with the point cloud features to obtain fused point cloud features; Using the correlation features between the target key point and other key points as a query vector, cross-attention calculation is performed on the fused point cloud features to generate the correlation feature information between the target key point and other key points. The relevant feature information is input into the feedforward neural network, which outputs the initial position information of the target key point and the corresponding key point features.

6. The method according to claim 1, wherein, The step of determining the local neighborhood features of the target key point based on the point cloud features, the initial position information, and the key point features includes: The initial coordinate information of the point cloud points is normalized to obtain normalized coordinates; The point cloud features, the initial position information, the key point features, and the normalized coordinates are respectively input into two multilayer perceptrons with shared parameters, and the aggregated key point features of the target key point and the neighborhood features of the target key point are respectively output. The aggregated key point features and neighborhood features are spliced ​​and fused to obtain the local neighborhood features of the target key point.

7. The method according to claim 1, wherein, The step of determining the target location information of the target key point based on the dependency relationship between the target key point and other key points and the local neighborhood features includes: The local neighborhood features and dependency graph are input into the graph attention convolutional network of the detection model, and the spatial dependency weights between the target key point and other key points are output. The dependency graph is constructed based on the dependency relationships between the key points. The location information of the target key point and the key points adjacent to the target key point are fused to obtain the fused location information. Based on the dependency relationship, the spatial dependency weight, and the fused position information, geometric constraints and position refinement are applied to the initial position information to obtain the normalized position information of the target key point; The normalized location information is denormalized to obtain the target location information.

8. The method according to claim 7, wherein, The step of performing inverse normalization calculation on the normalized location information to obtain the target location information includes: Calculate the coordinate mean and standard deviation based on the initial coordinate information of all point cloud points; The target location information is obtained by performing inverse normalization calculation based on the normalized location information, the mean of the coordinates, and the standard deviation of the coordinates.

9. The method according to claim 2, 3 or 7, wherein, The loss value during the training process of the detection model is calculated in the following way: Calculate the initial prediction loss based on the labeled location information of the target key points and the initial location information; Calculate the target prediction loss based on the labeled location information and the target location information; The loss value is calculated based on the initial prediction loss, the first weight corresponding to the initial prediction loss, the target prediction loss, and the second weight corresponding to the target prediction loss.

10. The method according to claim 1, wherein, The method further includes: Based on the nearest neighbor algorithm and the target location information, the target key points are determined from the point cloud points of the finger point cloud image.

11. A nail keypoint detection model, comprising: The basic backbone network module is used to extract features from the point cloud in the finger point cloud image to obtain point cloud features; The key point initial prediction module is used to determine the initial position information and key point features of the target key point based on the point cloud features and the average position information of key points on the fingernail, wherein the average position information is generated based on the annotation position information of the key point in the annotation sample; The local feature fusion module is used to determine the local neighborhood features of the target key point based on the point cloud features, the initial position information, and the key point features; The keypoint graph neural network module is used to determine the target location information of the target keypoint based on the dependency relationship between the target keypoint and other keypoints and the local neighborhood features; The basic backbone network module, the key point initial prediction module, the local feature fusion module, and the key point graph neural network module are connected in sequence.

12. The model according to claim 11, wherein, The model further includes a self-attention head module, which is located between the basic backbone network module and the keypoint initial prediction module. The self-attention head module includes: A multi-head attention network is used to generate the association between the target key point and other key points based on the query features of the pre-created key points. A feedforward neural network is used to optimize the correlation and generate correlation features between the target key point and other key points.

13. A nail key point detection device, comprising: The feature extraction unit is configured to extract features from point cloud points in a finger point cloud image to obtain point cloud features. The initial position determination unit is configured to determine the initial position information and key point features of the target key point based on the point cloud features and the average position information of key points on the fingernail, wherein the average position information is generated based on the annotation position information of the key point in the annotation sample; The neighborhood feature determination unit is configured to determine the local neighborhood features of the target key point based on the point cloud features, the initial position information, and the key point features. The target location determination unit is configured to determine the target location information of the target key point based on the dependency relationship between the target key point and other key points and the local neighborhood features.

14. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

15. A non-transitory computer-readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-10.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.

Images