Hand surface segmentation method, system, medium and equipment

Abstract

The invention discloses a hand surface segmentation method, system, medium and device, and relates to the rehabilitation field and the robot vision field, and the method comprises the steps: obtaining the point cloud data of a hand surface, and determining the three-dimensional coordinate data of the point cloud; maximum pooling and average pooling are combined, joint pooling is obtained, disordering processing is carried out on point cloud data of the hand surface, and global features of the hand surface are extracted; acquiring a plurality of local features of the point cloud data of the hand surface, performing weighted aggregation on the plurality of local features to obtain an aggregated feature vector, and processing the aggregated feature vector by adopting a multi-layer perceptron to obtain an abstract feature vector; obtaining enhanced information by combining the normal vector with the three-dimensional coordinate data of the point cloud; and inputting the global feature, the abstract feature vector and the enhanced information into a segmentation model based on a neural network, and segmenting the hand surface point cloud through the trained segmentation model to obtain a segmentation result of the hand surface attitude.

Description

Technical Field

[0001] The present invention relates to the fields of rehabilitation and robot vision, and in particular to a hand surface segmentation method, system, medium and device. Background Art

[0002] In the field of hand rehabilitation, with the increasing aging of the population, the demand for efficient and precise rehabilitation methods is growing. Traditional rehabilitation methods often rely on manual guidance or simple mechanical equipment, making it difficult to provide a personalized and immersive training environment. In recent years, the rise of metaverse technology has brought new possibilities to hand rehabilitation. As a digital world integrating multiple advanced technologies such as virtual reality, augmented reality, and artificial intelligence, the metaverse provides a new and immersive training environment for hand rehabilitation. However, the key to effectively applying metaverse technology in hand rehabilitation lies in accurately analyzing the surface posture of the hand.

[0003] Laser scanning, as a high-precision, non-contact method for acquiring three-dimensional data, has been widely used to collect hand surface point clouds. These hand surface laser point clouds contain precise positional information for various hand parts, as well as rich surface texture and geometric features, providing a valuable data foundation for hand function rehabilitation. Based on this, some research has attempted to segment hand surface point clouds using deep learning techniques, aiming to achieve more accurate hand posture analysis.

[0004] Although some studies have attempted to use deep learning technology to segment the hand surface point cloud, the actual collected hand surface point cloud often has the problem of unclear geometric features. This is mainly because the point cloud data cannot completely and densely cover the hand surface. This makes it difficult for the model to accurately capture key information of the hand surface when extracting features, such as the degree of bending of the joints, which in turn affects the accurate segmentation of the hand surface point cloud. This feature brings challenges to the training of deep learning models, especially when dealing with complex hand postures and changing training environments. Existing methods often cannot meet actual needs, and the segmentation accuracy still needs to be improved. Summary of the Invention

[0005] The present invention provides a hand surface segmentation method, system, medium, and device to solve the above-mentioned problem existing in the prior art, namely, how to improve the accuracy of hand surface posture segmentation in the prior art. The present invention provides a hand surface segmentation method, which includes:

[0006] Obtaining point cloud data of the hand surface and determining the three-dimensional coordinate data of the point cloud;

[0007] By combining maximum pooling and average pooling to obtain joint pooling, the point cloud data of the hand surface is disordered and the global features of the hand surface are extracted;

[0008] Obtain multiple local features of the point cloud data of the hand surface, perform weighted aggregation on the multiple local features to obtain an aggregated feature vector, and use a multi-layer perceptron (MLP) to process the aggregated feature vector to obtain an abstract feature vector.

[0009] By combining the normal vector with the 3D coordinate data of the point cloud, enhanced information is obtained;

[0010] The extracted global features, abstract feature vectors and enhanced information of the hand surface are input into a segmentation model based on a neural network. The hand surface point cloud is segmented by the trained segmentation model to obtain the segmentation results of the hand surface posture.

[0011] Optionally, the step of combining the normal vector with the three-dimensional coordinate data of the point cloud to obtain enhanced information specifically includes:

[0012] Based on the hand point cloud data, the local neighborhood of each point cloud data is determined. Based on the local neighborhood, the normal vector is obtained by using the least squares method, and the normal vector is normalized. The normalized normal vector is combined with the multi-density hand surface point cloud dataset to construct joint data and obtain enhanced information.

[0013] Optionally, the step of obtaining multiple local features of the point cloud data of the hand surface and performing weighted aggregation on the multiple local features to obtain an aggregated feature vector specifically includes:

[0014] The following formula is used to obtain the hand point cloud data set:

[0015] P={p1,p2,...p n};

[0016] Among them, P is the hand point cloud data set, p i ∈R 3 Represents the 3D coordinates of the i-th point in the point cloud, i = 1, 2...n, n represents the total number of points in the point cloud;

[0017] For each point p i , by defining a radius r to determine its local neighborhood N i , use the following formula to get the local neighborhood:

[0018] N i ={p i ∈p:||p i -p j ||2≤r}

[0019] Among them, ||.||2 represents the Euclidean distance, N i Contains p iAs the center, all points within the radius r, N i is a local neighborhood;

[0020] According to the local neighborhood N i Every point p in j , where j∈N i , obtain local features f i ∈R d , and the weight function is obtained using the following formula:

[0021]

[0022] According to the local features and combined with the weight function, the following formula is used to obtain the aggregated feature vector:

[0023]

[0024] Among them, W ij is the weight function, representing point p i For point p j The importance weight of x i is the feature vector after aggregation.

[0025] Optionally, the method combines maximum pooling and average pooling to obtain joint pooling to perform disorder processing on the point cloud data of the hand surface, specifically including:

[0026] Set the hand point cloud data to an N×D matrix X, where N represents the number of points in the hand point cloud and D represents the dimension of the feature vector of each point.

[0027] The following formula is used to obtain the matrix of hand point cloud data:

[0028]

[0029] Among them, x i =(x i1, x i2,..., x iD )T represents the D-dimensional feature vector of the i-th point, i = 1, 2, ... N;

[0030] Through maximum pooling, the maximum value of each feature dimension of matrix X is obtained to obtain the result of maximum pooling;

[0031] Through average pooling, the average value of each feature dimension of matrix X is obtained to obtain the result of average pooling;

[0032] By concatenating the results of maximum pooling and average pooling, a two-dimensional feature vector is obtained to perform disordering on the point cloud data of the hand surface.

[0033] Optionally, after obtaining the point cloud data of the hand surface, it is cleaned, denoised and formatted; after obtaining the segmentation result of the hand surface posture, rehabilitation training of the hand function in the metaverse environment is performed based on the segmentation result.

[0034] The present invention provides a hand surface segmentation system, comprising:

[0035] A data acquisition module is used to obtain point cloud data of the hand surface and determine the three-dimensional coordinate data of the point cloud;

[0036] The feature extraction module is used to combine maximum pooling and average pooling to obtain joint pooling, thereby performing disorder processing on the point cloud data of the hand surface and extracting the global features of the hand surface; obtaining multiple local features of the point cloud data of the hand surface, performing weighted aggregation on the multiple local features to obtain an aggregated feature vector, and processing the aggregated feature vector using a multi-layer perceptron (MLP) to obtain an abstract feature vector;

[0037] Normal vector feature enhancement module, used to obtain enhanced information by combining the normal vector with the three-dimensional coordinate data of the point cloud;

[0038] The segmentation module is used to input the extracted global features, abstract feature vectors and enhanced information of the hand surface into a segmentation model based on a neural network, and segment the hand surface point cloud through the trained segmentation model to obtain the segmentation results of the hand surface posture.

[0039] Also includes:

[0040] A data preprocessing module is used to clean, denoise and format the acquired point cloud data of the hand surface;

[0041] The rehabilitation application module is used to conduct rehabilitation training of hand functions in the metaverse environment based on the segmentation results.

[0042] The present invention provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the hand surface segmentation method is implemented.

[0043] The present invention provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned hand surface segmentation method when executing the program.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention provides a hand surface segmentation method, which abstracts and enhances the three-dimensional information of the hand point cloud through a multi-layer perceptron (MLP), extracts a higher-level abstract representation from locally aggregated features, and accurately segments the extracted features, thereby enhancing the model's ability to understand the complex structure of the hand point cloud; at the same time, by combining maximum pooling and average pooling to obtain joint pooling, the joint pooling operation retains richer information, thereby improving segmentation accuracy and stability; in addition, by introducing a hand surface feature enhancement method based on normal vectors, the present invention enhances the sensitivity of the 3D deep learning model to different geometric features of the point cloud, thereby effectively improving the segmentation accuracy of the hand surface point cloud. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0046] Figure 1 A flow chart of a hand surface segmentation method provided by an embodiment of the present invention;

[0047] Figure 2 A diagram showing the structure of the PointNet point cloud deep learning network provided by an embodiment of the present invention;

[0048] Figure 3 A network architecture diagram of a segmentation model provided by an embodiment of the present invention;

[0049] Figure 4 Schematic diagram of a computer device for the hand surface segmentation method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0050] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0051] The following describes in detail the technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0052] Figure 1 is a flow chart of a hand surface segmentation method provided by an embodiment of the present invention, such as Figure 1 As shown, this embodiment shows a hand surface segmentation method, including:

[0053] S1: Obtain point cloud data of the hand surface and determine the three-dimensional coordinate data of the point cloud.

[0054] For example, the hand point cloud data can be set as a set P = {p1, p2, ... p n}, where p i ∈R 3 Represents the 3D coordinates of the i-th point in the point cloud, where i = 1, 2, ..., n. Here n is the total number of points in the point cloud.

[0055] S2: By combining maximum pooling and average pooling to obtain joint pooling, the point cloud data of the hand surface is disordered and the global features of the hand surface are extracted.

[0056] Exemplarily, the disorder processing is mainly that the hand point cloud data consists of a series of points discretely distributed in 3D space, and the arrangement order of these points is arbitrary, that is, the hand point cloud has disorder. The deep neural network needs to meet the requirement that each point in the hand point cloud is invariant to permutation. In the deep neural network, the pooling operation is usually used to reduce the spatial size of the feature map and improve the translation invariance of the model. For hand point cloud data, due to its inherent disorder, if maximum pooling is used, the information of the remaining points except the point with the largest eigenvalue in the hand point cloud will be lost. Therefore, the present invention proposes a combination of maximum pooling and average pooling into joint pooling. Joint pooling combines the advantages of maximum pooling and average pooling, and retains richer information by simultaneously considering the maximum value and average value on each feature dimension. The specific steps are as follows:

[0057] (1) Data representation. Let the hand point cloud data be X, which is an N×D matrix, where N represents the number of points in the hand point cloud and D represents the dimension of the feature vector of each point. That is:

[0058]

[0059] here, is the D-dimensional feature vector of the i-th point, i = 1, 2, ...N.

[0060] (2) Maximum pooling. The maximum pooling operation is to take the maximum value of each column (i.e. each feature dimension) of the matrix X. Let the result of the maximum pooling be m, which is a D-dimensional vector whose j-th element m j The calculation is as follows:

[0061]

[0062] Represent the maximum pooling operation in vector form:

[0063] m=max(X)

[0064] Among them, max(.) represents the operation of taking the maximum value by column.

[0065] The purpose of max pooling is to retain the most significant features in each feature dimension.

[0066] (3) Average pooling. The average pooling operation is to calculate the average value of each column (i.e. each feature dimension) of the matrix X. Let the result of average pooling be a, which is a D-dimensional vector whose j-th element a j The calculation is as follows:

[0067]

[0068] Represent the average pooling operation in vector form:

[0069]

[0070] (4) Joint pooling. Joint pooling combines the results of maximum pooling and average pooling, and splices them together to obtain a 2D feature vector c. That is:

[0071]

[0072] Among them, the upper part is the D-dimensional vector m obtained by maximum pooling, and the lower part is the D-dimensional vector a obtained by average pooling.

[0073] S3: Obtain multiple local features of the point cloud data of the hand surface, obtain an aggregated feature vector by weighted aggregation of the multiple local features, and use a multi-layer perceptron (MLP) to process the aggregated feature vector to obtain an abstract feature vector.

[0074] For example, in order to obtain the feature vector for the input of the multi-layer perceptron (MLP), we need to perform weighted aggregation on the features in the local neighborhood. Define the weight function W ij , which represents the point p j p j Importance weight for points. Weight functions can be defined based on various factors, such as distance-based weight functions. Where is a small positive number to avoid the denominator being 0. The aggregated eigenvector x i The calculation is as follows:

[0075]

[0076] Here x i ∈R d It's point p i The feature vector after local neighborhood aggregation will serve as the input of the multi-layer perceptron.

[0077] For example, suppose the multilayer perceptron has L layers, and the weight matrix of the lth layer is The bias vector is The activation function is Where d0=d is the dimension of the input layer feature vector, d l is the output dimension of the lth layer. The input layer receives the aggregated feature vector x i ,Right now From layer l to layer L, calculate the output layer by layer: for l = 1, 2, ..., L, After L layers of calculation, the final output is Just click p i The corresponding abstract feature vector. This abstract feature vector It contains a high-dimensional abstract representation of the hand point cloud, which enhances the model's ability to understand the complex structure of the hand point cloud.

[0078] S4: Obtain enhanced information by combining the normal vector with the 3D coordinate data of the point cloud.

[0079] Optionally, by introducing normal vector feature enhancement, the normal vector is combined with the 3D coordinate data of the point cloud to obtain enhanced information, including:

[0080] Based on the hand point cloud data, the local neighborhood of each point cloud data point is determined. Based on this local neighborhood, the normal vector is obtained using the least squares method and normalized. This normalized normal vector is then combined with the multi-density hand surface point cloud dataset to construct joint data and obtain enhanced information.

[0081] For example, normal vector feature enhancement is a method that introduces normal vector information to enhance the model's sensitivity to the geometric features of point clouds in three-dimensional point cloud data processing. In the task of accurate segmentation of hand surface point clouds, since the point cloud data may not be able to completely and densely cover the hand surface, the geometric features are not obvious enough, which in turn affects the segmentation accuracy. Normal vector feature enhancement is proposed to solve this problem. The normal vector is an important attribute that describes the local geometric features of the point cloud surface. It represents the normal direction of the point cloud surface at that point. By combining the normal vector with the three-dimensional coordinate data of the point cloud, the feature dimension of the point cloud can be enriched, so that the model can more accurately capture the key information of the hand surface, such as the degree of bending of the joints, thereby improving the segmentation accuracy. The specific steps are as follows:

[0082] (1) Determine the local neighborhood. Suppose the hand point cloud data is a set P = {p1, p2, ... p n},in Represents the three-dimensional coordinates of the i-th point. For each point p i, by defining a radius to determine its local neighborhood N i , that is, N i ={p i ∈p:||p i -p j ||2≤r}, where ||.||2 represents the Euclidean distance. N i Contains p i All points within the radius r as the center.

[0083] (2) Calculate the normal vector. For point p i and its local neighborhood N i , we can use the least squares method to fit a plane to calculate the normal vector. Let N i The point in {p i1 ,p i2 ,...p im} (m is the number of points in the neighborhood), and the coordinates of these points are expressed as a matrix:

[0084]

[0085] Solve the coefficients of the plane equation z=ax+by+c by the least squares method So that Z≈Xa, that is, a=(X T X) -1 X TZ , the plane normal vector n i It can be expressed as:

[0086]

[0087] (3) Normalization of normal vector. In order to ensure the consistency and stability of the normal vector, the calculated normal vector n i Perform normalization. The normalized normal vector is n i for:

[0088]

[0089] (4) Construct joint data representation. i After obtaining the normalized normal vector of , it is combined with the multi-density hand surface point cloud dataset. Assume that the point p in the multi-density hand surface point cloud dataset is i The original eigenvector of F i (may contain information such as three-dimensional coordinates), the new joint eigenvector is:

[0090]

[0091] For example, for each point p i , by defining a radius r to determine its local neighborhood Ni That is, N i ={p i ∈p:||p i -p j ||2≤r}, where ||.||2 represents the Euclidean distance. N i Contains p i As the center, all points within the radius r. Then for the neighborhood N i Every point p in j (j∈N i ) Calculate the initial eigenvector f i ∈R d , where is the dimension of the feature vector. These initial features can be calculated based on the geometric properties of the points (such as coordinate difference, angle, etc.).

[0092] S5: The extracted global features, abstract feature vectors and enhanced information of the hand surface are input into a segmentation model based on a neural network. The trained segmentation model is used to segment the hand surface point cloud to obtain the segmentation results of the hand surface posture.

[0093] Qi et al. from Stanford University designed a deep neural network model, the PointNet model, which breaks through the bottleneck of point cloud research. PointNet does not need to convert point clouds into multi-view or voxel grids, but can directly process point clouds and use the max-pooling layer as a symmetric function to extract the global features of point clouds. Its internal structure is as follows: Figure 2 shown.

[0094] like Figure 3 As shown, the present invention uses PointNet as a foundation to construct a segmentation model, achieving intelligent segmentation of hand point clouds for intelligent hand function rehabilitation, thereby providing a crucial foundation for calculating hand joint positions and accurately determining gestures. Furthermore, the present invention can also segment hand surface point clouds based on neural network models, such as the dynamic graph convolutional neural network (DGCNN) model, to obtain segmentation results of hand surface posture, without limitation here.

[0095] The above is a hand surface segmentation method provided in one or more embodiments of this specification. Based on the same idea, this specification also provides a corresponding hand surface segmentation system, including:

[0096] A data acquisition module is used to obtain point cloud data of the hand surface and determine the three-dimensional coordinate data of the point cloud;

[0097] The feature extraction module is used to combine maximum pooling and average pooling to obtain joint pooling, thereby performing disorder processing on the point cloud data of the hand surface and extracting the global features of the hand surface; obtaining multiple local features of the point cloud data of the hand surface, performing weighted aggregation on the multiple local features to obtain an aggregated feature vector, and processing the aggregated feature vector using a multi-layer perceptron (MLP) to obtain an abstract feature vector;

[0098] Normal vector feature enhancement module, used to obtain enhanced information by combining the normal vector with the three-dimensional coordinate data of the point cloud;

[0099] The segmentation module is used to input the extracted global features, abstract feature vectors and enhanced information of the hand surface into a segmentation model based on a neural network, and segment the hand surface point cloud through the trained segmentation model to obtain the segmentation results of the hand surface posture.

[0100] Optionally, the system further includes: a data preprocessing module and a rehabilitation application module.

[0101] Optionally, the data preprocessing module is used to clean, denoise and format the acquired point cloud data of the hand surface.

[0102] Optionally, the rehabilitation application module is used to perform rehabilitation training of hand function in a metaverse environment based on the segmentation results.

[0103] For specific definitions of the hand surface segmentation system, please refer to the definitions of the hand surface segmentation method above and will not be repeated here. Each module in the above-mentioned hand surface segmentation system can be implemented in whole or in part through software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor of the computer device in hardware form, or can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each of the above modules.

[0104] The present invention also provides a computer-readable storage medium, which stores a computer program. The computer program can be used to execute the hand surface segmentation method provided above.

[0105] The present invention also provides Figure 4 The structural diagram of the computer equipment shown in FIG. Figure 4 As shown, at the hardware level, the computer device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile storage into the internal memory and then runs it to implement the hand surface segmentation method provided in the above embodiment.

[0106] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0107] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A hand surface segmentation method, characterized in that: include: Obtaining point cloud data of the hand surface and determining the three-dimensional coordinate data of the point cloud; By combining maximum pooling and average pooling to obtain joint pooling, the point cloud data of the hand surface is disordered and the global features of the hand surface are extracted; Obtain multiple local features of the point cloud data of the hand surface, perform weighted aggregation on the multiple local features to obtain an aggregated feature vector, and use a multi-layer perceptron (MLP) to process the aggregated feature vector to obtain an abstract feature vector. By combining the normal vector with the 3D coordinate data of the point cloud, enhanced information is obtained; The extracted global features, abstract feature vectors and enhanced information of the hand surface are input into a segmentation model based on a neural network. The hand surface point cloud is segmented by the trained segmentation model to obtain the segmentation results of the hand surface posture.

2. The hand surface segmentation method according to claim 1, wherein: The enhanced information is obtained by combining the normal vector with the three-dimensional coordinate data of the point cloud, specifically including: Based on the hand point cloud data, the local neighborhood of each point cloud data is determined. Based on the local neighborhood, the normal vector is obtained by using the least squares method, and the normal vector is normalized. The normalized normal vector is combined with the multi-density hand surface point cloud dataset to construct joint data and obtain enhanced information.

3. The hand surface segmentation method according to claim 1, wherein: The step of obtaining multiple local features of the point cloud data of the hand surface and performing weighted aggregation on the multiple local features to obtain an aggregated feature vector specifically includes: The following formula is used to obtain the hand point cloud data set: P={p1,p2,...p n }; Among them, P is the hand point cloud data set, p i ∈R 3 Represents the 3D coordinates of the i-th point in the point cloud, i = 1, 2...n, n represents the total number of points in the point cloud; For each point p i , by defining a radius r to determine its local neighborhood N i , use the following formula to get the local neighborhood: N i ={p i ∈p:||p i -p j ||2≤r} Among them, ||.||2 represents the Euclidean distance, N i Contains p i As the center, all points within the radius r, N i is a local neighborhood; According to the local neighborhood N i Every point p in j , where j∈N i , obtain local features f i ∈R d , and the weight function is obtained using the following formula: According to the local features and combined with the weight function, the following formula is used to obtain the aggregated feature vector: Among them, W ij is the weight function, representing point p i For point p j The importance weight of x i is the feature vector after aggregation.

4. The hand surface segmentation method according to claim 1, wherein: The maximum pooling and average pooling are combined to obtain joint pooling to perform disordered processing on the point cloud data of the hand surface, specifically including: Set the hand point cloud data to an N×D matrix X, where N represents the number of points in the hand point cloud and D represents the dimension of the feature vector of each point. The following formula is used to obtain the matrix of hand point cloud data: Among them, x i =(x i1 ,x i2 ,...,x iD )T represents the D-dimensional feature vector of the i-th point, i = 1, 2, ... N; Through maximum pooling, the maximum value of each feature dimension of matrix X is obtained to obtain the result of maximum pooling; Through average pooling, the average value of each feature dimension of matrix X is obtained to obtain the result of average pooling; By concatenating the results of maximum pooling and average pooling, a two-dimensional feature vector is obtained to perform disordering on the point cloud data of the hand surface.

5. The hand surface segmentation method according to claim 1, wherein: After obtaining the point cloud data of the hand surface, it is cleaned, denoised and formatted; after obtaining the segmentation result of the hand surface posture, rehabilitation training of the hand function in the metaverse environment is performed based on the segmentation result.

6. A hand surface segmentation system, characterized in that: include: A data acquisition module is used to obtain point cloud data of the hand surface and determine the three-dimensional coordinate data of the point cloud; The feature extraction module is used to combine maximum pooling and average pooling to obtain joint pooling, thereby performing disorder processing on the point cloud data of the hand surface and extracting the global features of the hand surface; obtaining multiple local features of the point cloud data of the hand surface, performing weighted aggregation on the multiple local features to obtain an aggregated feature vector, and processing the aggregated feature vector using a multi-layer perceptron (MLP) to obtain an abstract feature vector; Normal vector feature enhancement module, used to obtain enhanced information by combining the normal vector with the three-dimensional coordinate data of the point cloud; The segmentation module is used to input the extracted global features, abstract feature vectors and enhanced information of the hand surface into a segmentation model based on a neural network, and segment the hand surface point cloud through the trained segmentation model to obtain the segmentation results of the hand surface posture.

7. The hand surface segmentation system according to claim 6, wherein: Also includes: A data preprocessing module is used to clean, denoise and format the acquired point cloud data of the hand surface; The rehabilitation application module is used to conduct rehabilitation training of hand functions in the metaverse environment based on the segmentation results.

8. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the hand surface segmentation method according to any one of claims 1 to 5 is implemented.

9. A computer device, characterized in that: The hand surface segmentation method comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the program, the hand surface segmentation method according to any one of claims 1 to 5 is implemented.

Images