Knee joint acupoint recognition method based on depth camera

Abstract

The invention discloses a knee joint acupoint positioning identification method based on a depth camera, and belongs to the field of image processing. The method comprises the steps that a depth camera shoots a knee joint depth image and a plane RGB image, acupuncture point areas are marked, acupuncture points are converted into a data set, an instance segmentation model and a key point recognition model are trained, the depth image is subjected to instance segmentation to obtain an acupuncture point area instance segmentation image, and the depth image and the plane RGB image are processed through an image partition map. And carrying out key point identification on the obtained four-dimensional data to obtain acupuncture point coordinates.

Description

Technical Field

[0001] The present invention is directed to the field of human acupoint identification and positioning, and is particularly directed to the design of a method for positioning and identifying acupoints of a knee joint based on a depth camera. Background Art

[0002] Acupuncture, massage and other TCM treatments for knee joint acupoints can effectively treat knee osteoarthritis. However, due to the large differences in the appearance of the knee joints between different people, manual identification and positioning of acupoints are prone to large deviations. Therefore, the use of machines to identify and locate knee joint acupoints is not only a theoretical need for the modernization strategy of TCM, but also an actual technical need for the treatment of knee osteoarthritis.

[0003] At present, some methods for identifying and locating human acupuncture points have emerged. For example, the patent document with publication number CN112184705A proposes a human acupuncture point recognition, positioning and application system based on computer vision technology. The system proposes a method for identifying acupuncture points throughout the human body, using a CNN convolutional network. Although the network can be used to identify and locate acupuncture points, it requires too much attention to background information, resulting in a very large network size, not easy to deploy, and a very slow recognition speed; the patent document with publication number CN115409840A proposes a human back acupuncture point intelligent positioning system and method. For the identification of human back acupuncture points, the semantic segmentation model U-Net symmetric network structure can only derive the acupuncture point area rather than the specific position; the patent document with publication number CN118557435A proposes an intelligent acupuncture positioning device based on a deep learning network to identify the whole human body. The acupuncture points of the body are first identified by using RTMDet and then the specific position of the acupuncture points is located by using the RTMpose model. Although this method can discard a large amount of background information, it still has a large amount of calculation using the RGB image frame, and only using RGB images is susceptible to variables such as environmental illumination in both training and prediction processes; the patent document with publication number CN118366002A proposes a method for determining acupuncture point reflection areas, which aims to identify acupuncture points of the whole body of the human body, and uses a multi-layer attention mechanism to learn high and low resolution features. This method divides the image into sequences and assembles them into sequences, and can learn the relationship between the various parts of the image and apply them, but the relationship between the various parts in a picture is significantly lower than the relationship between the words in a sentence, and only focusing on the relationship between the various parts in the picture is quite inefficient, and it is easy to ignore the features in the picture unit block, and can only identify the acupuncture point reflection area. The above method is for the recognition of acupuncture points of the whole body or back of the human body, but the features at the knee joint are quite different from the features of the whole body or back of the human body, and the above methods only use planar RGB images for training, without considering other picture forms that can highlight the features of the acupuncture points.

[0004] Therefore, TCM treatment for knee osteoarthritis requires a method for identifying and locating acupoints at the knee joint. This method can utilize more characteristic information at the knee joint, can accurately determine the coordinates of the acupoints, and can be combined with existing manual methods of identifying and locating acupoints. At the same time, it can more efficiently and accurately identify and locate knee joint acupoints. Summary of the invention

[0005] Based on the above needs, the present invention proposes a method for identifying acupoints of human knee joint based on a depth camera, which includes taking a depth image and a planar RGB image of the knee joint with a depth camera, marking the acupoint area and the acupoint points and converting them into a data set, training an instance segmentation model and a key point recognition model, segmenting the depth image by instance to obtain an instance segmentation image of the acupoint area, processing the depth image and the planar RGB image with an image partition map, obtaining four-dimensional data for key point recognition to obtain acupoint coordinates.

[0006] The technical solution of the system is: a method for locating and identifying knee joint acupoints based on a depth camera, the method comprising:

[0007] Step 1: Use a depth camera to capture knee joint depth images and planar RGB images: Use a depth camera to capture knee joint depth images and planar RGB images, and collect knee joint image samples of different ages, genders, and health conditions;

[0008] Step 2: Mark the acupoint areas and acupoint points and convert them into data sets;

[0009] A dataset generator that converts the original depth image and the original RGB color image into the corresponding dataset, wherein the dataset generator includes three links: annotation, classifier, and image processor processing. The original image is annotated to obtain an image set, the image set is obtained by the classifier to obtain a training image set and a verification image set, and the training image set and the verification image set are respectively obtained by the image processor to obtain a training dataset and a verification dataset;

[0010] Step 3: Train an instance segmentation model for depth images:

[0011] Step 3.1: Establish a segmentation model for quickly segmenting the acupoint area at the knee joint according to the following method;

[0012] Step 3.1.1: Use a 4x downsampling layer to treat four pixels as a prediction partition and extract low-dimensional feature map information to obtain abstract features;

[0013] Step 3.1.2: Use ConvNeXt Block convolutional layers to extract features from the depth image at multiple levels to obtain low-resolution feature maps and high-resolution feature maps respectively;

[0014] Step 3.1.3: Use the self-attention pooling layer to obtain the relationship between the parts in the low-resolution feature map, and use the least information to obtain the global overall information of the depth image;

[0015] Step 3.1.4: Use the multi-scale feature fusion module to fuse the feature sequence compressed by the high-resolution feature map and the global overall information feature sequence obtained by the self-attention pooling layer to form a feature sequence called a multi-scale feature fusion sequence;

[0016] Step 3.1.5: Use the attention pooling layer to use the annotation information to allocate and refine the multi-scale feature fusion sequence information, so that the model gradually increases the attention to the information that needs more attention in instance segmentation during the training process, and increases the corresponding weight;

[0017] Step 3.1.6: Use transposed convolution to upsample and restore the details of the whole image to obtain the instance segmentation image;

[0018] Step 3.2: Train the segmentation model:

[0019] Step 3.3: The iterator used in the training process is the Adamw optimizer with a learning rate scheduler:

[0020] Step 3.4: Get the area map of each acupoint, the pixel value of the acupoint area is True, and the pixel value of other positions is False;

[0021] Step 3.5: Use the training data set obtained by the data set generator to input the loss generator, and put the network structure into the loss generator;

[0022] Step 4: Use the trained instance segmentation model to segment the depth image and obtain the acupoint area;

[0023] Step 5: Use the acupuncture point area image to cut the depth image and the plane RGB image:

[0024] Step 5.1: Use the torch.nonzero(tensor) function to get the coordinates of the acupoint area:

[0025] Local=torch.nonzero(tensor); At this time, Local.shape() is (n,2), where n is the number of pixels in the acupoint area, and the first and second columns store the ordinate y and abscissa x of each pixel in the acupoint area respectively;

[0026] Step 5.2: Get the edge frame information of the acupoint area [(x_min, y_min), (x_max, y_max)], (x_min, y_min) represents the minimum coordinates of the edge frame of the acupoint area, and (x_max, y_max) represents the maximum coordinates of the edge frame of the acupoint area:

[0027] Step 5.3: Use the edge frame information to crop the original depth tensor image and the plane tensor RGB image to obtain the depth tensor edge frame image and the plane RGB tensor edge frame image of the acupuncture point area: Use

[0028] torchvision.transforms.functional.crop function crops image new_image = F.crop(last_image, y, x, h, w);

[0029] F.crop() represents the torchvision.transforms.functional.crop function;

[0030] Where last_image is the original tensor image before cropping;

[0031] Where y is y_min, x is x_min, h is (y_max-y_min), and w is (x_max-x_min);

[0032] Step 5.4: Fill the image in the edge frame of the acupuncture point area:

[0033] Step 6: Use the DWpose framework to obtain the key point recognition model:

[0034] Step 6.1: Use the built-in loss function, network structure, and iterator of the DWpose framework, where the number of input channels of the DWpose backbone network is changed to 4, corresponding to the R\G\B channels of the depth image and the plane image respectively;

[0035] Step 6.2: The instance segmentation model is obtained by training using the Ebbinghaus training strategy.

[0036] Furthermore, the specific method of step 2 is:

[0037] Step 2.1: Labeling: Use labelme labeling software to label the depth image and the plane RGB image respectively and convert them into COCO format, wherein the plane RGB image is labeled with acupuncture point information and acupuncture point region information, and the coordinate information of the acupuncture point and acupuncture point region labeled in the plane RGB image is equally mapped to the depth image to obtain the acupuncture point and acupuncture point region information of the labeled depth image, and then the acupuncture point region information of the plane RGB image is deleted;

[0038] Step 2.2: Classifier: According to the Ebbinghaus clustering training strategy, cluster training is first performed on the image set to obtain different types of images. Different types of images are defined as batch blocks. The order of images in the batch block is randomly disrupted at the training node. The images with the first 10% of the sequence numbers in the batch block are taken as the validation set, and the others are taken as the verification set. This operation is repeated for each training node.

[0039] Step 2.3: The image processor includes an image scaler, an image category determiner, a data enhancer, a tensor quantizer, and a pixel normalizer.

[0040] Furthermore, the specific method of step 2.3 is:

[0041] Step 2.3.1: Image resizer: Unify the image size to a standard size. For the depth image fed into the instance segmentation model, the unified size is 1024*1024;

[0042] Step 2.3.2: Image category judger: judge whether it is a training set according to the sequence number of the image in the batch block. If it is a training set, the image is imported into the data enhancer, otherwise it is input into the tensor quantizer;

[0043] Step 2.3.3: Data enhancer: Flip and rotate the image to simulate shooting at different angles of the camera, change the brightness and saturation to simulate shooting in different environments, add noise to simulate shooting with different cameras, increase data diversity through data enhancer, and improve model generalization;

[0044] Step 2.3.4: Tensor quantizer: convert image data into tensor format;

[0045] Step 2.3.5: Pixel Normalizer: Map the tensor data of each pixel of the image to the range of [0,1], so that the data can better cooperate with activation functions such as Sigmoid, which is conducive to model training and convergence; the mapping relationship is: X new Represents the pixel value after mapping, x last Represents the pixel value before mapping, x min Indicates the minimum pixel value of the image before mapping, x max Indicates the maximum pixel value of the image before mapping.

[0046] Furthermore, the specific method of step 3.2 is:

[0047] Step 3.2.1: Perform clustering training to divide the data set into different categories;

[0048] Step 3.2.2: Train one category of data in a batch, and verify with the next category of data to visualize the changes in the model's generalization ability; perform a gradient descent and model update for each training batch; using one category of data to update can make greater use of image information, prevent features from canceling each other out and causing slow gradient descent, and prevent the model from updating to the nearest local optimum while ignoring the farther but better local minimum point. After training a batch, verify with the next data to compare the model performance at each EbHe_epoch vertically and visualize the model's generalization ability;

[0049] Step 3.2.3: The training process includes repeated EbHe_epochs. One EbHe_epoch contains two training circuits. The first training circuit is the initial training circuit, and the second training circuit is the recurrent training circuit. The training data of the two circuits are the same but the time is different. The initial training circuit trains two categories of data each time, then enters the recurrent training circuit to train one category of data, and then enters the original position of the initial training circuit to continue training two categories of data, and so on until the recurrent training circuit has trained all the data of an Ebbinghaus cycle; through this training process, the model can gradually master and become familiar with various features, and update to a better local optimal point;

[0050] Step 3.2.4: Alternate the training order every EbHe_epoch to improve the generalization ability of the model; by changing the training order, the data diversity can be increased, which can improve the generalization ability of the model.

[0051] Furthermore, the specific method of step 5.2 is:

[0052] Step 5.2.1: Use x_max = torch.max(nonzero_indices[:,1]) to take out the second column of all rows and find the maximum value;

[0053] Step 5.2.2: Use x_min = torch.min(nonzero_indices[:, 1]) to take out the second column of all rows and find the minimum value;

[0054] Step 5.2.3: Use y_max = torch.max(nonzero_indices[:,0]) to take out the first column of all rows and find the maximum value;

[0055] Step 5.2.4: Use y_min = torch.min(nonzero_indices[:,0]) to take out the first column of all rows and find the minimum value.

[0056] Furthermore, the specific method of step 5.4 is:

[0057] Step 5.4.1: Traverse each pixel point of the acupoint area image;

[0058] Step 5.4.2: If the pixel point is within the acupoint area coordinate group, no change is made; otherwise, the tensor value of the pixel point is set to 0.

[0059] The present invention adopts a depth camera to capture a depth image and a plane RGB image of the knee joint, annotates the acupuncture point area and the acupuncture point and converts them into a data set, trains an instance segmentation model and a key point recognition model, and performs appropriate data enhancement on the training set to improve data diversity and enhance the generalization of the model; designs an image normalizer to linearly map the tensor data of each pixel of the image to the range of [0,1] to improve the training speed, accelerate model convergence and training stability; instance segmentation of the depth image obtains an instance segmentation image of the acupuncture point area, and the image partition map processes the depth image and the plane RGB image to obtain four-dimensional data for key point recognition to obtain acupuncture point coordinates. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A schematic diagram of the process of the knee joint acupoint recognition method based on a depth camera proposed in the present invention;

[0061] Figure 2 This is a schematic diagram of the framework of the identification method proposed in the present invention;

[0062] Figure 3 The present invention is aimed at Figure 1 Flowchart of the dataset generator designed by S103;

[0063] Figure 4 For the present invention Figure 3 A flow chart of an embodiment of the marking module;

[0064] Figure 5 For the present invention Figure 3 A flowchart of an embodiment of an image processor;

[0065] Figure 6 The present invention proposes Figure 1 A schematic diagram of an embodiment of the Ebbinghaus clustering training strategy in S103;

[0066] Figure 7 The present invention proposes Figure 1 A flow chart of an embodiment of model training in S103;

[0067] Figure 8 The present invention proposes Figure 1 A network structure diagram of a segmentation model for a depth image in S103;

[0068] Fig. 9 The present invention proposes Figure 1Flowchart of processing the depth image and the planar RGB image using the acupoint area map in S105; DETAILED DESCRIPTION

[0069] Figure 1 This is a flow chart of the knee joint acupoint recognition method based on a depth camera proposed in the present invention, which includes:

[0070] S101: Use a depth camera to capture a depth image and a plane RGB image of the knee joint. The depth camera can directly obtain the depth image and the plane RGB image.

[0071] S102: Labeling acupoint regions and acupoints and converting them into data sets. The original image can be converted into a data set through three steps: labeling, classifier and image processor.

[0072] Specifically, this embodiment designs a data set generator such as Figure 3 As shown, the original image is annotated to obtain an image set with annotation information, and then divided into a training image set and a verification image set through a classifier. The training image set and the verification image set are processed by an image processor to obtain a training data set and a verification data set respectively.

[0073] Among them, the labeling is preferentially performed using labelme annotation software to annotate the depth image and the plane RGB image and convert them into COCO format. Using the COCO format dataset can make DataLoader construction easier. Since the bone measurement positioning method and the surface anatomical landmark positioning method of traditional Chinese medicine are not easy to annotate on the depth image, but the depth image needs to be annotated with acupuncture points and acupuncture area information, considering that the pixels of the depth image and the plane RGB image taken by the depth camera are one-to-one corresponding, the annotation method is adopted as follows Figure 4 As shown:

[0074] S301: Marking acupuncture points and acupuncture point areas on the planar RGB image.

[0075] S302: Mapping the coordinates of the acupuncture points and acupuncture area annotations in the planar RGB image to the depth image

[0076] S302: Since the planar RGB image does not need to be labeled with acupoint area information, it is necessary to delete the redundant acupoint area information.

[0077] The image processor of the present invention is implemented as follows: Figure 4As shown, it includes an image scaler, an image category determiner, a data enhancer, a tensor quantizer, and a pixel normalizer. The input data for the model forward calculation needs to be set to a uniform size according to the model. In this embodiment, the depth image fed into the instance segmentation model has a uniform size of 1024*1024. The image scaler is used to unify the image size to 1024*1024. In the embodiment of the present invention, for images smaller than the standard size, the image size is increased by bilinear interpolation, and for images larger than the standard size, the image size is reduced by downsampling.

[0078] Obviously, appropriate data enhancement for the training set can improve data diversity and enhance model generalization, but the validation set, as data for verifying model performance, should not be enhanced, so this embodiment designs an image category judge to determine whether the image belongs to the training set or the validation set. According to the Ebbinghaus clustering training strategy, the validation set is in the first part of each batch and the training set is in the second part, so the image data can be judged according to the index number of the image data to determine whether the image belongs to the training set or the validation set. If it is a training set, the image is imported into the data enhancer, otherwise it is directly input into the tensor quantizer. Considering that Pytorch uses tensor data, this embodiment designs a tensor quantizer to convert the image data into the number of tensors. In order to increase the training speed, accelerate model convergence and training stability, this embodiment designs an image normalizer to linearly map the tensor data of each pixel of the image to the range of [0,1]. The mapping relationship is shown in the content of the invention.

[0079] S103: training the instance segmentation model and the key point recognition model, wherein the training strategy, model network structure, iterator, loss generator and other links need to be considered, and the present invention provides preferred embodiments, and the training process is as follows Figure 7 .

[0080] In order to improve the accuracy and enhance the model's understanding of the internal features of the image data, the present invention designs the Ebbinghaus clustering training strategy as follows: Figure 6As shown, the internal features of the data are first refined and classified through cluster training to obtain data sets with different features. In order to make the model deepen its grasp of different features, the training process is divided into two training circuits, initial training and retraining, referring to the Ebbinghaus forgetting curve, collectively referred to as one EbHe_epoch. In this embodiment, the training data of the two circuits in one EbHe_epoch are the same but at different times. The initial training circuit enters the retraining circuit to train one category of data for each training of two categories of data, and then enters the original position of the initial training circuit to continue training two categories of data, and so on until the retraining circuit has trained all the data of an Ebbinghaus cycle, the purpose is to use the retraining circuit to improve the grasp of features. One category of data per epoch, and verified with the data of the next category, so that training several EbHe_epochs can vertically compare the generalization ability of the model and visualize the changes in the generalization ability of the model.

[0081] Among them, the present invention designs an instance segmentation network structure for deep images such as Figure 8 As shown in the figure, the depth of each acupoint area in the depth map of the knee joint has obvious changes, which can be seen through the depth Figure 1 The overall idea is to extract the overall features of the image to obtain feature maps of two scales, namely high-resolution and low-resolution feature maps, which contain detail features and abstract features respectively. The low-resolution feature map containing abstract features is used through a self-attention pooling layer to obtain a feature sequence that mines the connection between the various parts of the image. The feature sequence is fused with the high-resolution feature map to obtain a two-scale fusion sequence. The two-scale fusion map is searched through the attention mechanism with the annotation information as the query information to obtain the instance segmentation result of the acupoint area, and then the image is restored through transposed convolution. Finally, each acupoint area is output separately in the form of a binary image.

[0082] The present invention uses ConvNeXt Block to build a Backbone feature extractor to optimize the feature extraction effect. First, the image resolution is reduced to 1 / 4 of the original through a 4-fold downsampling layer. It is worth noting that the ConvNeXtBlock in this embodiment replaces the depth separable convolution layer with an ordinary CNN layer with the same number of output channels but 1 input channel. Because the input data has only one channel of depth image data, there is no need to use the function of the depth separable convolution layer to fuse multi-channel data, and the other parts of the ConvNeXt Block, including the normalization layer, the activation function layer, and the residual connection structure, remain unchanged. At this time, when the depth image data passes through a ConvNeXt Block, a feature map with a receptive field of four pixels can be obtained, that is, a 1 / 4 resolution feature map, which contains the detailed features of the depth image. Through the same 4-fold downsampling layer and ConvNeXt Block, a feature map with a receptive field of sixteen pixels, that is, a 1 / 16 resolution feature map, can be obtained. This feature map contains the abstract features of the depth image.

[0083] The internal relationship of the whole image can be obtained by using the low-resolution feature map through the self-attention pooling layer, which greatly reduces the amount of data calculation. Therefore, this example uses the low-resolution feature map through the self-attention pooling layer to obtain the internal feature connection sequence of the entire depth image, and then uses the multi-scale feature fusion module to compress the high-resolution feature map into a sequence, and splice it with the internal feature connection sequence of the depth image into a sequence, which is the two-scale feature fusion sequence. At this time, the detail features, abstract features, and internal feature connections of the entire depth image are all in the two-scale feature fusion sequence, but not all features are what we need to pay attention to, so this example uses the attention mechanism to train attention to various information contained in the two-scale feature fusion sequence, and inputs the two-scale feature fusion sequence into the attention pooling layer, using the labeled information as a query. As the training continues, the weight of the features that need to be paid attention to will gradually increase, while the weight of the features that do not need to be paid attention to will gradually decrease, and the model will use the features of the depth image more and more efficiently.

[0084] The instance segmentation results on the feature sequence cannot be used to process depth images and planar RGB images, so the sequence information needs to be restored to the original image size. At this time, the detail information lost in Backbone can be restored through the transposed convolution layer to obtain the instance segmentation graph of the original image size.

[0085] S104: Instance segmentation of the depth image to obtain an image partition map, and use the trained instance segmentation model to perform forward calculation on the depth image to obtain a predicted acupuncture point area map of the depth image.

[0086] S105: The image partition map processes the depth image and the planar RGB image. The depth image and the planar RGB image of the acupuncture point area need to be extracted to reduce the amount of data processed by the key point recognition model.

[0087] Considering that the amount of data for key point recognition of normal depth images and plane RGB images is very large and easily affected by background interference, the present invention first uses instance segmentation to obtain image partitions to reduce the amount of training and calculation of the key point recognition model, and to highlight the edge information of the acupuncture point area. Therefore, a very important step of the present invention is to use the acupuncture point area results of instance segmentation to cut and fill the depth image and plane RGB image of the input key point model. Fig. 9 As shown, the functions of the Pytorch library are used as listed in the invention content to obtain the input image data required for key point recognition through image binarization, edge box search, cutting, and non-mask assignment to 0.

[0088] S106: Obtain four-dimensional data for key point recognition to obtain acupoint coordinates. The processed acupoint area depth image and plane RGB image, a total of 4-dimensional data, enter the key point recognition model network with 4 input channels for forward calculation to obtain the acupoint coordinates.

[0089] It is worth noting that it is natural to use planar RGB images for training, but adding another dimension of depth data can increase the number of features that can be used to identify acupoints. Moreover, the depth image after cutting and filling using the acupoint area map is already very small, and the increased amount of calculation is negligible relative to the improvement in model effect.

Claims

1. A method for locating and identifying knee joint acupoints based on a depth camera, the method comprising: Step 1: Use a depth camera to capture knee joint depth images and planar RGB images: Use a depth camera to capture knee joint depth images and planar RGB images, and collect knee joint image samples of different ages, genders, and health conditions; Step 2: Mark the acupoint areas and convert the acupoint points into data sets; A dataset generator that converts the original depth image and the original RGB color image into the corresponding dataset, wherein the dataset generator includes three links: annotation, classifier, and image processor processing. The original image is annotated to obtain an image set, the image set is obtained by the classifier to obtain a training image set and a verification image set, and the training image set and the verification image set are respectively obtained by the image processor to obtain a training dataset and a verification dataset; Step 3: Train an instance segmentation model for depth images: Step 3.1: Establish a segmentation model for quickly segmenting the acupoint area at the knee joint according to the following method; Step 3.1.1: Use a 4x downsampling layer to treat four pixels as a prediction partition and extract low-dimensional feature map information to obtain abstract features; Step 3.1.2: Use ConvNeXt Block convolutional layers to extract features from the depth image at multiple levels to obtain low-resolution feature maps and high-resolution feature maps respectively; Step 3.1.3: Use the self-attention pooling layer to obtain the relationship between the parts in the low-resolution feature map, and use the least information to obtain the global overall information of the depth image; Step 3.1.4: Use the multi-scale feature fusion module to fuse the feature sequence compressed by the high-resolution feature map and the global overall information feature sequence obtained by the self-attention pooling layer to form a feature sequence called a multi-scale feature fusion sequence; Step 3.1.5: Use the attention pooling layer to use the annotation information to allocate and refine the multi-scale feature fusion sequence information, so that the model gradually increases the attention to the information that needs more attention in instance segmentation during the training process, and increases the corresponding weight; Step 3.1.6: Use transposed convolution to upsample and restore the details of the whole image to obtain the instance segmentation image; Step 3.2: Train the segmentation model: Step 3.3: The iterator used in the training process is the Adamw optimizer with a learning rate scheduler: Step 3.4: Get the area map of each acupoint, the pixel value of the acupoint area is True, and the pixel value of other positions is False; Step 3.5: Use the training data set obtained by the data set generator to input the loss generator, and put the network structure into the loss generator; Step 4: Use the trained instance segmentation model to segment the depth image and obtain the acupoint area; Step 5: Use the acupuncture point area image to cut the depth image and the plane RGB image: Step 5.1: Use the torch.nonzero(tensor) function to get the coordinates of the acupoint area: Local=torch.nonzero(tensor); At this time, Local.shape() is (n,2), where n is the number of pixels in the acupoint area, and the first and second columns store the ordinate y and abscissa x of each pixel in the acupoint area respectively; Step 5.2: Get the edge frame information of the acupoint area [(x_min, y_min), (x_max, y_max)], (x_min, y_min) represents the minimum coordinates of the edge frame of the acupoint area, and (x_max, y_max) represents the maximum coordinates of the edge frame of the acupoint area: Step 5.3: Use the edge frame information to crop the original depth tensor image and the plane tensor RGB image to obtain the depth tensor edge frame image and the plane RGB tensor edge frame image of the acupuncture point area: Use torchvision.transforms.functional.crop function crops image new_image = F.crop(last_image, y, x, h, w); F.crop() represents the torchvision.transforms.functional.crop function; Where last_image is the original tensor image before cropping; Where y is y_min, x is x_min, h is (y_max-y_min), and w is (x_max-x_min); Step 5.4: Fill the image inside the edge frame of the acupuncture point area: Step 6: Use the DWpose framework to obtain the key point recognition model: Step 6.1: Use the built-in loss function, network structure, and iterator of the DWpose framework, where the number of input channels of the DWpose backbone network is changed to 4, corresponding to the R\G\B channels of the depth image and the plane image respectively; Step 6.2: The instance segmentation model is obtained by training using the Ebbinghaus training strategy.

2. A method for locating and identifying knee joint acupoints based on a depth camera as claimed in claim 1, characterized in that: The specific method of step 2 is: Step 2.1: Labeling: Use labelme labeling software to label the depth image and the plane RGB image respectively and convert them into COCO format, wherein the plane RGB image is labeled with acupuncture point information and acupuncture point region information, and the coordinate information of the acupuncture point and acupuncture point region labeled in the plane RGB image is equally mapped to the depth image to obtain the acupuncture point and acupuncture point region information of the labeled depth image, and then the acupuncture point region information of the plane RGB image is deleted; Step 2.2: Classifier: According to the Ebbinghaus clustering training strategy, cluster training is first performed on the image set to obtain different types of images. Different types of images are defined as batch blocks. The order of images in the batch block is randomly disrupted at the training node. The images with the first 10% of the sequence numbers in the batch block are taken as the validation set, and the others are taken as the verification set. This operation is repeated for each training node. Step 2.3: The image processor includes an image scaler, an image category determiner, a data enhancer, a tensor quantizer, and a pixel normalizer.

3. The method for locating and identifying knee joint acupoints based on a depth camera as claimed in claim 1, characterized in that: The specific method of step 2.3 is: Step 2.3.1: Image resizer: Unify the image size to a standard size. For the depth image fed into the instance segmentation model, the unified size is 1024*1024; Step 2.3.2: Image category judger: judge whether it is a training set according to the sequence number of the image in the batch block. If it is a training set, the image is imported into the data enhancer, otherwise it is input into the tensor quantizer; Step 2.3.3: Data enhancer: Flip and rotate the image to simulate shooting at different angles of the camera, change the brightness and saturation to simulate shooting in different environments, add noise to simulate shooting with different cameras, increase data diversity through data enhancer, and improve model generalization; Step 2.3.4: Tensor quantizer: convert image data into tensor format; Step 2.3.5: Pixel Normalizer: Map the tensor data of each pixel of the image to the range of [0,1], so that the data can better cooperate with activation functions such as Sigmoid, which is conducive to model training and convergence; the mapping relationship is: x new Represents the pixel value after mapping, x last Represents the pixel value before mapping, x min Indicates the minimum pixel value of the image before mapping, x max Indicates the maximum pixel value of the image before mapping.

4. The method for locating and identifying knee joint acupoints based on a depth camera as claimed in claim 1, characterized in that: The specific method of step 3.2 is: Step 3.2.1: Perform clustering training to divide the data set into different categories; Step 3.2.2: Train one category of data in batches, and verify with the next category of data, and visualize the changes in the model's generalization ability; perform a gradient descent and model update once for each training batch; Step 3.2.3: The training process includes repeated EbHe_epochs. One EbHe_epoch includes two training circuits. The first training circuit is the initial training circuit, and the second training circuit is the recurrent training circuit. The training data of the two circuits are the same but the time is different. After the initial training circuit trains two categories of data, it enters the recurrent training circuit to train one category of data, and then enters the original position of the initial training circuit to continue training two categories of data, and so on until the recurrent training circuit has trained all the data of an Ebbinghaus cycle; Step 3.2.4: Alternate the training order every EbHe_epoch to improve the generalization ability of the model.

5. The method for locating and identifying knee joint acupoints based on a depth camera as claimed in claim 1, characterized in that: The specific method of step 5.2 is: Step 5.2.1: Use x_max = torch.max(nonzero_indices[:,1]) to take out the second column of all rows and find the maximum value; Step 5.2.2: Use x_min = torch.min(nonzero_indices[:, 1]) to take out the second column of all rows and find the minimum value; Step 5.2.3: Use y_max = torch.max(nonzero_indices[:,0]) to take out the first column of all rows and find the maximum value; Step 5.2.4: Use y_min = torch.min(nonzero_indices[:,0]) to take out the first column of all rows and find the minimum value.

6. The method for locating and identifying knee joint acupoints based on a depth camera as claimed in claim 1, characterized in that: The specific method of step 5.4 is: Step 5.4.1: Traverse each pixel point of the acupoint area image; Step 5.4.2: If the pixel point is within the acupoint area coordinate group, no change is made; otherwise, the tensor value of the pixel point is set to 0.

Images