Knee joint acupoint recognition method based on depth camera

Abstract

The application discloses a knee joint acupoint positioning and recognizing method based on a depth camera and belongs to the field of image processing. The method comprises the following steps: a depth camera shoots a knee joint depth image and a plane RGB image, labels an acupoint region and an acupoint point and converts them into a data set, trains an instance segmentation model and a key point recognizing model, obtains an acupoint region instance segmentation image through instance segmentation of the depth image, processes the depth image and the plane RGB image through an image partitioning image, obtains four-dimensional data, recognizes key points and obtains acupoint coordinates.

Description

TECHNICAL FIELD

[0001] The application relates to the field of human acupoint recognition and positioning, and particularly to a knee joint acupoint positioning and recognition method based on a depth camera. BACKGROUND

[0002] Knee joint acupoints are treated by acupuncture, massage and other traditional Chinese medicine treatment methods to treat knee osteoarthritis. However, due to the large appearance difference of knee joints between different people, manual recognition and positioning of acupoints is prone to large deviation, so machine recognition and positioning of knee joint acupoints are required by the theory of modernization of traditional Chinese medicine and the actual technical requirement of treatment of knee osteoarthritis.

[0003] Now, some methods for recognizing and positioning human acupoints have appeared, for example, a patent file with the publication number CN112184705A proposes a human acupoint recognition, positioning and application system based on computer vision technology, which proposes a method for recognizing human acupoints, adopts a CNN convolution network, the network can be used to recognize and position acupoints, but needs to pay too much attention to background information, resulting in a very large network volume, which is not easy to deploy, and the recognition speed is very slow; a patent file with the publication number CN115409840A proposes a human back acupoint intelligent positioning system and method, which uses a semantic segmentation model U-Net symmetrical network structure to obtain only the acupoint region but not the specific position for recognizing human back acupoints; a patent file with the publication number CN118557435A proposes an intelligent acupuncture positioning device based on a deep learning network, which is used to recognize human acupoints, uses RTMDet to recognize the acupoint region and uses RTMpose model to position the specific acupoint position, this method can discard a large amount of background information, but using an RGB image frame still has a large amount of calculation, and only using an RGB image is susceptible to environmental light and other variables in the training and prediction processes; a patent file with the publication number CN118366002A proposes an acupoint reflex zone determination method, which is used to recognize human acupoints, uses a multi-layer attention mechanism to learn high and low resolution features, the method divides an image into groups and assembles into a sequence, can learn the relationship between each part of the image and apply it, but the relationship between each part of an image is significantly lower than the relationship between words in a sentence, and only paying attention to the relationship between each part of the image is quite inefficient, and features in the picture unit block are easily ignored, and only the acupoint reflex zone can be recognized. The above methods are used for recognizing acupoints of the whole human body or back, but the features of the knee joint are quite different from those of the whole human body or back, and the above methods only use a plane RGB image for training, without considering other picture forms that can highlight the features of acupoints.

[0004] Therefore, for the treatment of knee osteoarthritis in traditional Chinese medicine, a method for identifying and positioning the acupoints at the knee joint is needed. This method can utilize more feature information at the knee joint, accurately locate the acupoint coordinates, and be combined with existing manual identification and positioning of acupoints, while being more efficient and accurate in identifying and positioning the acupoints of the knee joint. SUMMARY

[0005] Based on the above needs, the present application provides a human knee joint acupoint recognition method based on a depth camera, which includes depth camera shooting knee joint depth image and plane RGB image, labeling acupoint area and acupoint point conversion to data set, training instance segmentation model and key point recognition model, instance segmentation depth image to get acupoint area instance segmentation image, image partition image processing depth image and plane RGB image, getting four-dimensional data for key point recognition to get acupoint coordinates.

[0006] The technical scheme of the system is as follows: a knee joint acupoint positioning and recognition method based on a depth camera, which includes:

[0007] Step 1: Depth camera shooting knee joint depth image and plane RGB image: using depth camera to shoot knee joint depth image and plane RGB image, collecting knee joint image samples of different ages, genders and health states;

[0008] Step 2: Labeling acupoint area and acupoint point conversion to data set;

[0009] The original depth image and the original RGB color image are converted into the corresponding data set by a data set generator, wherein the data set generator includes three links of labeling, classifier and image processor. The original image is obtained by labeling to obtain an image set, the image set is obtained by a classifier to obtain a training image set and a verification image set, and the training image set and the verification image set are respectively obtained by an image processor to obtain a training data set and a verification data set;

[0010] Step 3: Training instance segmentation model for depth image:

[0011] Step 3.1: A segmentation model for quickly segmenting the acupoint area at the knee joint is established according to the following method;

[0012] Step 3.1.1: Use 4 times down-sampling layer to take four pixels as a prediction partition, and extract low-dimensional feature map information to obtain abstract features;

[0013] Step 3.1.2: Use ConvNeXt Block convolution layer to extract features from the depth image at multiple levels to obtain low-resolution feature maps and high-resolution feature maps respectively;

[0014] Step 3.1.3: Obtain the relationship between each part in the low-resolution feature map using the self-attention pooling layer, and use the least information to obtain the global overall information of the depth image;

[0015] Step 3.1.4: Use the multi-scale feature fusion module to fuse the feature sequence compressed from the high-resolution feature map and the global overall information feature sequence obtained from the self-attention pooling layer, and become a feature sequence as a multi-scale feature fusion sequence;

[0016] Step 3.1.5: Use the attention pooling layer to use the annotation information to perform attention allocation and refinement on the multi-scale feature fusion sequence information, so that the model gradually improves the attention to the information that needs to be focused on during the training process. Instance segmentation, improve the corresponding weight;

[0017] Step 3.1.6: Use transposed convolution for upsampling to restore the full graph detail information, and obtain the instance segmentation picture;

[0018] Step 3.2: Train the segmentation model:

[0019] Step 3.3: The iterator used in the training process is the Adamw optimizer with a learning rate scheduler:

[0020] Step 3.4: Obtain each acupoint region map, and the acupoint region value pixel is True, and the pixel value of other positions is False;

[0021] Step 3.5: Use the training dataset generated by the dataset generator to input the loss generator, and put the network structure into the loss generator;

[0022] Step 4: Use the trained instance segmentation model to segment the depth image to obtain the acupoint region;

[0023] Step 5: Cut the depth image and the plane RGB image using the acupoint region image:

[0024] Step 5.1: Use the torch.nonzero(tensor) function to obtain the acupoint region coordinates:

[0025] Local=torch.nonzero(tensor); At this time, Local.shape() is (n,2), where n is the number of pixels in the acupoint region, and the first column and the second column store the vertical coordinate y and horizontal coordinate x of each pixel in the acupoint region;

[0026] Step 5.2: Obtain the edge box information [(x_min, y_min), (x_max, y_max)] of the acupoint region, where (x_min, y_min) represents the minimum coordinates of the edge box of the acupoint region, and (x_max, y_max) represents the maximum coordinates of the edge box of the acupoint region:

[0027] Step 5.3: Crop the original depth tensor image and the plane tensor RGB image to obtain the depth tensor edge-in-box image and the plane RGB tensor edge-in-box image of the acupoint region using the edge box information: using

[0028] torchvision.transforms.functional.crop function to crop the image new_image = F.crop(last_image, y, x, h, w);

[0029] F.crop() represents the torchvision.transforms.functional.crop function;

[0030] where last_image is the original tensor image before cropping;

[0031] where y is y_min, x is x_min, h is (y_max-y_min), and w is (x_max-x_min);

[0032] Step 5.4: Fill the acupoint region edge-in-box image:

[0033] Step 6: Obtain the key point recognition model using the DWpose framework:

[0034] Step 6.1: Use the loss function, network structure, and iterator built-in the DWpose framework, where the input channel number of the DWpose backbone network is changed to 4, corresponding to the R\G\B channels of the depth image and the plane image respectively;

[0035] Step 6.2: As in the instance segmentation model, the instance segmentation model is obtained by training through the Ebbinghaus training strategy.

[0036] Further, the specific method of step 2 is:

[0037] Step 2.1: Labeling: Use the labelme labeling software to label the depth image and the plane RGB image and convert them into COCO format, where the acupoint point information and acupoint region information of the plane RGB image are labeled and mapped to the depth image to obtain the acupoint point and acupoint region information of the labeled depth image, and then the acupoint region information of the plane RGB image is deleted;

[0038] Step 2.2: Classifier: According to the Ebbinghaus clustering training strategy, first, the image set is clustered and trained to obtain different types of images. Different types of images are defined as batch blocks. The image order in the batch block is randomly shuffled at the training node. The first ten percent of images in the batch block are taken as the validation set, and the others are taken as the validation set. Each training node repeats this operation;

[0039] Step 2.3: Image processor includes image scaler, image category determiner, data enhancer, tensorizer, and pixel normalizer.

[0040] Further, the specific method of step 2.3 is:

[0041] Step 2.3.1: Image scaler: unify the image size to the standard size. For depth images input into the instance segmentation model, the size is unified to 1024*1024;

[0042] Step 2.3.2: Image category determiner: determine whether it is a training set according to the sequence number of the image in the batch block. If it is a training set, the picture is input into the data enhancer, otherwise it is input into the tensorizer;

[0043] Step 2.3.3: Data enhancer: flip and rotate the picture to simulate different angles of camera shooting, change brightness and saturation to simulate different environments, and add noise to simulate different camera shooting. Through data enhancement, data diversity is increased and model generalization is improved;

[0044] Step 2.3.4: Tensorizer: convert picture data to tensor format;

[0045] Step 2.3.5: Pixel normalizer: map the tensor data of each pixel of the picture to the range [0, 1], so that the data is better matched with the Sigmoid activation function, which is conducive to the training and convergence of the model. The mapping relationship is: X new represents the pixel value after mapping, x last represents the pixel value before mapping, x min represents the minimum pixel value of the picture before mapping, x max represents the maximum pixel value of the picture before mapping.

[0046] Further, the specific method of step 3.2 is:

[0047] Step 3.2.1: Cluster training to divide the data set into different categories;

[0048] Step 3.2.2: Train one batch of data for one class and validate with the next class of data to visualize the changes in model generalization ability; perform gradient descent and model update once per training batch; update the model to better utilize image information and prevent features from canceling each other out, leading to slow gradient descent, while also preventing the model from updating to the nearest local optimum and ignoring a more distant but better local minimum; after training a batch, validate with the next data to compare the performance of each EbHe_epoch and visualize the model's generalization ability;

[0049] Step 3.2.3: The training process includes repeated EbHe_epochs, each EbHe_epoch containing two training lines, the first being the initial training line and the second being the retraining line; the training data for both lines is the same but at different times; the initial training line trains two classes of data, then enters the retraining line to train one class of data, and then returns to the initial training line to continue training two classes of data, and so on until the retraining line has trained all the data in an EbHe_epoch; through this training process, the model can gradually master and familiarize itself with various features and update to a better local optimum;

[0050] Step 3.2.4: Alternate the training order for each EbHe_epoch to improve model generalization ability; changing the training order can improve data diversity and enhance model generalization ability.

[0051] Further, the specific method of step 5.2 is:

[0052] Step 5.2.1: Use x_max = torch.max(nonzero_indices[:,1]) to take the second column of all rows and find the maximum value;

[0053] Step 5.2.2: Use x_min = torch.min(nonzero_indices[:,1]) to take the second column of all rows and find the minimum value;

[0054] Step 5.2.3: Use y_max = torch.max(nonzero_indices[:,0]) to take the first column of all rows and find the maximum value;

[0055] Step 5.2.4: Use y_min = torch.min(nonzero_indices[:,0]) to take the first column of all rows and find the minimum value.

[0056] Further, the specific method of step 5.4 is:

[0057] Step 5.4.1: Traverse each pixel point of the acupoint region image;

[0058] Step 5.4.2: If the pixel point is located in the acupoint region coordinate group, no change is made, otherwise, the tensor value of the pixel point is set to 0.

[0059] The knee joint depth image and the plane RGB image are shot by the depth camera, the acupoint region and the acupoint point are labeled into a data set, the instance segmentation model and the key point recognition model are trained, the data augmentation is properly performed on the training set to improve the data diversity and the model generalization; the image normalizer is designed to linearly map the tensor data of each pixel of the picture to the range of [0, 1], improve the training speed, accelerate the model convergence and the training stability; the acupoint region instance segmentation image is obtained by instance segmentation of the depth image, the depth image and the plane RGB image are processed by image partitioning, four-dimensional data are obtained, and the acupoint coordinates are obtained by key point recognition. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The flowchart of the knee joint acupoint recognition method based on the depth camera is provided for the present application.

[0061] Figure 2 The framework diagram of the recognition method is provided for the present application.

[0062] Figure 3 The data set generator designed in S103 in the present application is provided for the present application. Figure 1

[0063] Figure 4 The data set generator designed in S103 in the present application is provided for the present application. Figure 3 The flowchart of one embodiment of the labeling module in the present application is provided for the present application.

[0064] Figure 5 The flowchart of one embodiment of the image processor in the present application is provided for the present application. Figure 3

[0065] The flowchart of one embodiment of the image processor in the present application is provided for the present application. Figure 6 Figure 1 The flowchart of one embodiment of the model training in S103 in the present application is provided for the present application.

[0066] Figure 7 Figure 1 The flowchart of one embodiment of the model training in S103 in the present application is provided for the present application.

[0067] Figure 8 The network structure diagram of one embodiment of the segmentation model for the depth image in S103 in the present application is provided for the present application. Figure 1

[0068] The network structure diagram of one embodiment of the segmentation model for the depth image in S103 in the present application is provided for the present application. Figure 9 Figure 1 ​​​​Flow chart of processing depth image and planar RGB image using acupoint region map in S105; DETAILED DESCRIPTION

[0069] Figure 1 Flow chart of knee acupoint recognition method based on depth camera proposed by the present application, which includes:

[0070] S101: Use a depth camera to capture a knee depth image and a planar RGB image. The depth camera can directly obtain a depth image and a planar RGB image.

[0071] S102: Label acupoint regions and acupoint points and convert them into a data set. The original image can be converted into a data set through three links of labeling, classifier and image processor.

[0072] Specifically, the data set generator is designed as shown in Figure 3 The original image is labeled to obtain an image set with labeling information, and then the training image set and the verification image set are divided by the classifier. The training image set and the verification image set are processed by the image processor to obtain the training data set and the verification data set, respectively.

[0073] The labeling software labelme is used to label the depth image and the planar RGB image and convert them into COCO format. The data set in COCO format can make the DataLoader easier to build. Since it is not easy to label the depth image using the bone degree and cun positioning method and the body surface anatomy landmark positioning method of traditional Chinese medicine, the depth image needs to be labeled with acupoint point and acupoint region information. Considering that the pixels of the depth image and the planar RGB image captured by the depth camera are one-to-one corresponding, the labeling method is as shown in Figure 4

[0074] S301: Label acupoint points and acupoint regions on the planar RGB image.

[0075] S302: Map the acupoint point and acupoint region labeling coordinates of the planar RGB image to the depth image

[0076] S302: Since the planar RGB image does not need to be labeled with acupoint region information, the redundant acupoint region information needs to be deleted.

[0077] The image processor of the present application is implemented as shown in Figure 4 ​As shown, including image scaler, image category determinator, data enhancer, tensorizer, pixel normalizer. The input data of the model forward calculation needs to be unified according to the model setting size, in this embodiment, the depth image sent into the instance segmentation model is unified to 1024*1024, the function of the image scaler is to unify the image size to 1024*1024, in the embodiment of the application, for the image with a size smaller than the standard size, the image size is increased by using a bilinear interpolation method, and for the image with a size larger than the standard size, the image size is reduced by using a downsampling method.

[0078] Obviously, appropriate data enhancement for the training set improves data diversity and improves model generalization, but the validation set as the data for verifying the performance of the model should not be data enhanced, so the embodiment designs an image category determinator to determine whether the picture belongs to the training set or the validation set. According to the Ebbinghaus clustering training strategy, the validation set is in the first part of each batch and the training set is in the latter part, so according to the index number of the picture data, it can be determined whether the picture belongs to the training set or the validation set, if it is the training set, the picture is imported into the data enhancer, otherwise it is directly input into the tensorizer. Considering that Pytorch uses tensor data, the embodiment designs a tensorizer to convert the picture data into a tensor number. In order to improve the training speed, accelerate the model convergence and training stability, the embodiment designs an image normalizer to linearly map the tensor data of each pixel of the picture to the range of [0, 1], and the mapping relationship is as shown in the invention content.

[0079] S103: training the instance segmentation model and the key point recognition model, wherein the training strategy, the model network structure, the iterator, the loss generator and the like need to be considered, the present application provides an embodiment priority selection, and the training process is as follows Figure 7 .

[0080] In order to improve the accuracy and improve the model's understanding of the internal feature relationship of the picture data, the application designs an Ebbinghaus clustering training strategy as follows Figure 6As shown, the data is first refined by clustering training to extract internal features and classified to obtain data sets with different features. In order to make the model deeper to master different features, the training process is divided into two training lines of initial training and retraining by referring to the Ebbinghaus forgetting curve, which is collectively called an EbHe_epoch. In this embodiment, the training data of the two lines in an EbHe_epoch is the same but the time is different. The initial training line trains two categories of data, enters the retraining line to train one category of data, and then enters the initial training line to continue training two categories of data. This is repeated until the retraining line trains all the data in an Ebbinghaus ring. The purpose is to improve the mastery of features by using the retraining line. Each epoch has one category of data, and the next category of data is used for verification. Training several EbHe_epochs can compare the generalization ability of the model vertically, and visualize the change of the generalization ability of the model.

[0081] In the embodiment, the instance segmentation network structure for depth images is designed as shown in the following figure. Figure 8 As shown, the depth changes of the acupoint regions of the knee joint in the depth map are obviously used, and the depth Figure 1 The data of the Zhang picture is quickly instance segmented in each acupoint region with such a small amount of data. The overall idea is to extract the features of the whole image to obtain two scale feature maps, which are high-resolution and low-resolution feature maps, respectively, containing detailed features and abstract features. The low-resolution feature map containing abstract features is obtained by self-attention pooling layer to obtain the feature sequence of the relationship between each part of the picture. The feature sequence is fused with the high-resolution feature map to obtain a two-scale fusion sequence. The attention mechanism is used to search the two-scale fusion map with the label information as the query information to obtain the acupoint region instance segmentation result. The picture is restored by transposed convolution. Finally, each acupoint region is output in the form of a binary map.

[0082] The application uses a ConvNeXt Block to build a Backbone feature extractor to optimize the feature extraction effect. First, the image resolution is reduced to 1 / 4 of the original through a 4-fold downsampling layer. It is worth noting that the ConvNeXt Block in this embodiment replaces the depth separable convolution layer with a normal CNN layer with the same number of output channels but an input channel number of 1, because the input data is only a depth image data channel, and does not need to use the function of fusing multi-channel data using a depth separable convolution layer, while the other parts of the ConvNeXt Block, including the normalization layer, the activation function layer and the residual connection structure, remain unchanged. At this time, when the depth image data passes through a ConvNeXt Block, a feature map with a receptive field of four pixels, i.e. a 1 / 4 resolution feature map, can be obtained, which contains the detailed features of the depth image. Then, through the same 4-fold downsampling layer and ConvNeXt Block, a feature map with a receptive field of sixteen pixels, i.e. a 1 / 16 resolution feature map, can be obtained, which contains the abstract features of the depth image.

[0083] The low-resolution feature map is used to obtain the internal relationship of the whole image through a self-attention pooling layer, which greatly reduces the data operation amount, so in this example, the low-resolution feature map is used to obtain the internal feature connection sequence of the whole depth image through a self-attention pooling layer, and then a multi-scale feature fusion module is used to compress the high-resolution feature map into a sequence, which is spliced with the internal feature connection sequence of the depth image into a sequence, i.e. a two-scale feature fusion sequence. At this time, the detailed features, abstract features and internal feature connections of the whole depth image are all in the two-scale feature fusion sequence, but not all features are what we need to pay attention to, so in this example, the attention mechanism is used to train the attention of various information contained in the two-scale feature fusion sequence. The two-scale feature fusion sequence is input into the attention pooling layer with the labeled information as the query. As the training continues, the weight of the features that need to be paid attention to will gradually increase, while the weight of the features that do not need to be paid attention to will gradually decrease, and the efficiency of the model in using the features of the depth image will be higher and higher.

[0084] The instance segmentation result on the obtained feature sequence cannot be used to process the depth image and the planar RGB image, so the sequence information needs to be restored to the original image size. At this time, the transpose convolution layer can be used to restore the lost detailed information in the Backbone, and the instance segmentation graph of the original image size can be obtained.

[0085] S104: instance segmentation of the depth image to obtain an image partition graph, using the trained instance segmentation model to perform forward calculation on the depth image to obtain a predicted acupoint region graph of the depth image.

[0086] S105: image partitioning processing depth image and plane RGB image, the depth image and the plane RGB image of the acupoint region need to be extracted to reduce the data amount processed by the key point recognition model.

[0087] The present application considers that the key point recognition data amount of normal depth image and plane RGB image is extremely large, and is easily disturbed by the background, so the image partitioning is obtained by using instance segmentation to reduce the training amount and calculation amount of the key point recognition model, and the edge information of the acupoint region is highlighted. Therefore, a very important step of the present application is to cut and fill the depth image and the plane RGB image to be input into the key point model using the acupoint region result of instance segmentation as shown in Figure 9 The function of the Pytorch library is used to obtain the input image data required for key point recognition through image binarization, edge frame search, cutting, and non-mask code assignment as shown in the summary of the invention.

[0088] S106: obtaining four-dimensional data for key point recognition to obtain acupoint coordinates, the processed acupoint region depth image and plane RGB image are four-dimensional data, which are input into the key point recognition model network with an input channel of 4 for forward calculation to obtain acupoint coordinates.

[0089] It is worth noting that it is a natural approach to use the plane RGB image for training, but if another dimension of depth data is added, the number of features that can be used to recognize acupoints can be increased, and the depth image after cutting and filling using the acupoint region graph is already very small, and the increase in calculation amount is negligible compared to the effect of the model.

Claims

1. A method for locating and identifying knee joint acupoints based on a depth camera, the method comprising: Step 1: Use a depth camera to capture knee joint depth images and planar RGB images: Use a depth camera to capture knee joint depth images and planar RGB images, and collect knee joint image samples of different ages, genders, and health conditions; Step 2: Mark the acupoint areas and acupoint points and convert them into data sets; A dataset generator that converts the original depth image and the original RGB color image into the corresponding dataset. The dataset generator includes three steps: annotation, classifier, and image processing. The original image is annotated to obtain an image set, and the image set is processed by the classifier to obtain a training image set and a verification image set. The training image set and the verification image set are processed by the image processor to obtain a training dataset and a verification dataset respectively. Step 3: Train instance segmentation model for depth images: Step 3.1: Establish a segmentation model for quickly segmenting the acupoint area at the knee joint according to the following method; Step 3.1.1: Use a 4x downsampling layer to treat four pixels as a prediction partition and extract low-dimensional feature map information to obtain abstract features; Step 3.1.2: Use ConvNeXt Block convolutional layers to extract features from the depth image at multiple levels to obtain low-resolution feature maps and high-resolution feature maps respectively; Step 3.1.3: Use the self-attention pooling layer to obtain the relationship between the parts in the low-resolution feature map, and use the least information to obtain the global overall information of the depth image; Step 3.1.4: Use the multi-scale feature fusion module to fuse the feature sequence compressed by the high-resolution feature map and the global overall information feature sequence obtained by the self-attention pooling layer to form a feature sequence called the multi-scale feature fusion sequence; Step 3.1.5: Use the attention pooling layer to allocate and refine the attention of the multi-scale feature fusion sequence information using the annotation information, so that during the training process the model gradually increases its attention to the information that is more important for instance segmentation and increases the corresponding weight; Step 3.1.6: Use transposed convolution to upsample and restore the details of the entire image to obtain the instance segmentation image; Step 3.2: Train the segmentation model: Step 3.3: The iterator used in the training process is the Adamw optimizer with a learning rate scheduler: Step 3.4: Get the map of each acupoint area, where the pixel value of the acupoint area is True and the pixel value of other positions is False; Step 3.5: Use the training dataset obtained by the dataset generator to input the loss generator, and put the network structure into the loss generator; Step 4: Use the trained instance segmentation model to segment the depth image and obtain the acupoint area; Step 5: Use the acupoint area image to cut the depth image and the planar RGB image: Step 5.1: Use the torch.nonzero(tensor) function to get the coordinates of the acupoint area: Local = torch.nonzero(tensor); Local.shape() is (n, 2), where n is the number of pixels in the acupoint area, and the first and second columns store the y-coordinate and x-coordinate of each pixel in the acupoint area, respectively. Step 5.2: Get the edge frame information of the acupoint area [(x_min, y_min), (x_max, y_max)], (x_min, y_min) represents the minimum coordinates of the edge frame of the acupoint area, and (x_max, y_max) represents the maximum coordinates of the edge frame of the acupoint area: Step 5.3: Use the edge frame information to crop the original depth tensor image and the plane tensor RGB image to obtain the depth tensor edge frame image and the plane RGB tensor edge frame image of the acupuncture point area: Use torchvision.transforms.functional.crop function crops the image new_image = F.crop(last_image,y,x,h,w); F.crop() represents the torchvision.transforms.functional.crop function; Where last_image is the original tensor image before cropping; Where y is y_min, x is x_min, h is (y_max - y_min), and w is (x_max - x_min); Step 5.4: Fill the image within the edge frame of the acupuncture point area: Step 6: Use the DWpose framework to obtain the key point recognition model: Step 6.1: Use the built-in loss function, network structure, and iterator of the DWpose framework. The number of input channels of the DWpose backbone network is changed to 4, corresponding to the R, G, and B channels of the depth image and the planar image respectively. Step 6.2: Obtain the instance segmentation model in the same way, and obtain the instance segmentation model through training using the Ebbinghaus training strategy.

2. The method for locating and identifying knee joint acupoints based on a depth camera according to claim 1, wherein: The specific method of step 2 is: Step 2.1: Labeling: Use labelme to label the depth image and the planar RGB image separately and convert them into COCO format. The acupoint information and acupoint region information are annotated on the planar RGB image. The coordinate information of the acupoints and acupoint regions annotated on the planar RGB image is mapped equally to the depth image to obtain the acupoints and acupoint region information of the annotated depth image. The acupoint region information of the planar RGB image is then deleted. Step 2.2: Classifier: First, perform cluster training on the image set according to the Ebbinghaus cluster training strategy to obtain different types of images. Define different types of images as batch blocks. Randomly shuffle the order of images in the batch blocks at the training node. Take the first 10% of the images in the batch blocks as the validation set, and the rest as the verification set. Repeat this operation for each training node. Step 2.3: The image processor includes an image scaler, an image category determiner, a data augmenter, a tensor quantizer, and a pixel normalizer.

3. The method for locating and identifying knee joint acupoints based on a depth camera according to claim 1, wherein: The specific method of step 2.3 is: Step 2.3.1: Image resizer: Unify the image size to a standard size. For the depth image fed into the instance segmentation model, the unified size is 1024*1024; Step 2.3.2: Image category judger: Determine whether the image is a training set based on its sequence number in the batch block. If it is a training set, the image is imported into the data enhancer, otherwise it is input into the tensor quantizer; Step 2.3.3: Data Augmenter: Flip and rotate the image to simulate shooting at different camera angles, change the brightness and saturation to simulate shooting in different environments, add noise to simulate shooting with different cameras, and use data augmentation to increase data diversity and improve model generalization. Step 2.3.4: Tensor quantizer: convert image data into tensor format; Step 2.3.5: Pixel Normalizer: Map the tensor data of each pixel of the image to the range of [0,1] to make the data better compatible with activation functions such as Sigmoid, which is conducive to model training and convergence; the mapping relationship is: x new Represents the pixel value after mapping, x last Represents the pixel value before mapping, x min Indicates the minimum pixel value of the image before mapping, x max Indicates the maximum pixel value of the image before mapping.

4. The method for locating and identifying knee joint acupoints based on a depth camera according to claim 1, wherein: The specific method of step 3.2 is: Step 3.2.1: Perform clustering training to divide the data set into different categories; Step 3.2.2: Train one class of data per batch and verify with the next class of data, visualizing the changes in the model's generalization ability; perform a gradient descent and model update once for each training batch; Step 3.2.3: The training process includes repeated EbHe_epochs. An EbHe_epoch consists of two training circuits. The first training circuit is the initial training circuit, and the second training circuit is the recurrent training circuit. The training data of the two circuits are the same but at different times. After the initial training circuit trains two categories of data, it enters the recurrent training circuit to train one category of data, and then enters the original position of the initial training circuit to continue training two categories of data. This process is repeated until the recurrent training circuit has trained all the data of an Ebbinghaus cycle. Step 3.2.4: Alternate the training order every EbHe_epoch to improve the generalization ability of the model.

5. The method for locating and identifying knee joint acupoints based on a depth camera according to claim 1, wherein: The specific method of step 5.2 is: Step 5.2.1: Use x_max = torch.max(nonzero_indices[:,1]) to retrieve the second column of all rows and find the maximum value. Step 5.2.2: Use x_min = torch.min(nonzero_indices[:,1]) to extract the second column of all rows and find the minimum value; Step 5.2.3: Use y_max = torch.max(nonzero_indices[:,0]) to extract the first column of all rows and find the maximum value; Step 5.2.4: Use y_min = torch.min(nonzero_indices[:,0]) to take the first column of all rows and find the minimum value.

6. The method for locating and identifying knee joint acupoints based on a depth camera according to claim 1, wherein: The specific method of step 5.4 is: Step 5.4.1: Traverse each pixel point of the acupoint area image; Step 5.4.2: If the pixel point is located within the acupoint region coordinate group, no change is made; otherwise, the tensor value of the pixel point is set to 0.

Images