A method for locating wrist pulse diagnosis points based on heatmaps and temporal attention priors

Abstract

This invention belongs to the field of computer image processing technology, specifically a method for locating wrist pulse points based on heatmaps and temporal attention priors. The invention includes: constructing two-dimensional coordinates of key points (cun, guan, chi) from an infrared video dataset as supervisory information; constructing a deep learning model, specifically using U-Net as the baseline architecture, combined with a temporal attention prior module; the U-Net encoder extracts features from the input data, the decoder generates a heatmap, and after each upsampling, the generated heatmap is fused with the corresponding level features to restore the original resolution, resulting in a keypoint heatmap; finally, keypoint coordinates are obtained through keypoint fitting. The temporal attention prior, through skip connections, fuses features with the low-level semantic features of the encoder and the high-level semantic features of the decoder, enabling the model to better utilize the temporal information of the infrared video and output a more accurate keypoint heatmap. Finally, a heatmap of the wrist pulse points is predicted, and the pulse point coordinates are obtained through keypoint fitting.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing technology, and specifically relates to a method for locating pulse diagnosis points on the wrist. Background Technology

[0002] Pulse diagnosis is a commonly used method in Traditional Chinese Medicine (TCM) for diagnosing illnesses. TCM practitioners observe the pulse by pressing their fingers on the cun, guan, and chi positions on the patient's wrist to assess the condition. However, traditional pulse diagnosis faces several problems: TCM practitioners often rely on subjective assumptions in their judgments, leading to different diagnoses for the same condition; the pressure applied by the fingers can affect the patient's pulse, resulting in inaccurate diagnoses; and the pulse data is not effectively stored, impacting future assessments. With the advent of the information age, the objectification of pulse diagnosis has become an emerging research direction in TCM, and the rapid development of artificial intelligence in recent years has opened up new avenues for its objectification.

[0003] With the improvement of hardware performance and the optimization of deep learning theory, more and more artificial intelligence-related applications are emerging, including high-precision key point localization, high-precision object detection, and high-precision image recognition. Many researchers have developed pulse diagnosis devices to simulate traditional Chinese medicine practitioners' pulse diagnosis, using various sensors to locate pulse points and acquire pulse information. In the process of simulating traditional Chinese medicine pulse diagnosis, accurately locating the wrist pulse point is a crucial step. Currently, the most accurate method is deep learning-based wrist pulse point localization, but it still faces the problem of insufficient accuracy. The purpose of this invention is to provide a wrist key point localization method based on heatmaps and temporal attention priors. Summary of the Invention

[0004] The purpose of this invention is to provide a method for locating wrist pulse points based on heatmaps and temporal attention priors, enabling rapid and accurate location of wrist pulse points with a small-scale infrared video dataset.

[0005] The wrist pulse diagnosis point localization method based on heatmap and temporal attention prior provided by the present invention has the following basic steps: First, based on the infrared video dataset, the two-dimensional coordinates of the key points of "cun, guan, chi" are marked as supervision information for the deep learning model.

[0006] Secondly, a deep learning model is constructed, using U-Net as the baseline architecture and incorporating a temporal attention prior. U-Net mainly consists of an encoder, decoder, skip connections, max pooling layers, and transposed convolutional layers. The encoder extracts features from the input data and downsampled data, while the decoder generates heatmaps. After each upsampling, the generated heatmap is fused with the corresponding layer's features to restore the original resolution and obtain the keypoint heatmap. The temporal attention prior fuses features from the encoder's low-level semantic features and the decoder's high-level semantic features through skip connections, enabling the model to better utilize the temporal information of the infrared video and output more accurate keypoint heatmaps. Max pooling layers reduce data dimensionality; transposed convolutional layers increase feature map size for feature map reconstruction; finally, the wrist video data to be tested is used as input to the trained deep learning model to predict the wrist pulse point heatmap, and the wrist pulse point coordinates are obtained through keypoint fitting.

[0007] The wrist pulse diagnosis point localization method based on heatmaps and temporal attention priors proposed in this invention has the following specific steps:

[0008] (1) Based on the infrared video dataset, construct the two-dimensional coordinates of the key points of “inch, gate, and ruler” as supervision information;

[0009] (2) Constructing deep learning networks;

[0010] The deep learning model uses U-Net as the baseline architecture and incorporates a temporal attention prior module.

[0011] In the temporal attention prior module, the average gray value of each spatial location in the video is first calculated. Then, the average gray value is subtracted from each frame in the video image set, and the absolute value is taken to represent the degree of change of gray value (i.e., the difference). Then, the differences of each spatial location are added together to obtain the spatial attention map. Finally, a threshold is set to suppress spatial locations with little change to obtain their temporal attention prior.

[0012] U-Net consists of a contraction path and an expansion path, comprising multiple encoders, decoders, skip connections, max pooling layers, and transposed convolutional layers. The encoders extract features from the input data; the decoders generate heatmaps; skip connections allow the network to combine high-resolution details with upsampled coarse information, helping to address the vanishing gradient problem in deep learning models; max pooling layers reduce feature map size, retaining important features, reducing computation, and minimizing overfitting; transposed convolutional layers increase feature map size for reconstruction. Each encoder or decoder consists of two 3×3 convolutional layers, followed by a ReLU activation layer and a batch normalization layer. The convolutional layers extract features from the input data; the batch normalization layer accelerates model convergence; and the ReLU activation layer enables non-linear fitting, addressing complex data fitting problems. Furthermore, building upon the skip connections of U-Net, a temporal attention prior is incorporated, fusing it with the low-level semantic features of the encoder and the high-level semantic features of the decoder. This allows the model to better utilize the temporal information of the infrared video and output more accurate keypoint heatmaps. Specifically, feature fusion is achieved by concatenating the features after convolution in the decoder.

[0013] (3) Train the deep learning network;

[0014] The collected infrared video dataset is divided into a training set, a validation set, and a test set. The training set is used to train the deep learning model to fit the data distribution. The validation set is used to verify the generalization ability of the deep learning model during the training process and record the model with the best generalization ability. The test set is used to perform performance tests on the trained deep learning model to measure its performance.

[0015] (4) Using the trained deep learning model, obtain the heat map of the key points “cun, guan, chi”.

[0016] (5) The key point heatmap of each channel is fitted with the mean value of the central region to obtain the coordinates of the wrist pulse diagnosis point.

[0017] Furthermore, in step (1), the two-dimensional coordinates of the key points “cun, guan, chi” are constructed based on the infrared video dataset as supervision information. Specifically, “cun, guan, chi” in the infrared video are labeled by a traditional Chinese medicine practitioner and used as supervision information for the deep learning model.

[0018] Furthermore, the infrared video dataset mentioned in step (3) is divided into a training set, a validation set, and a test set, specifically as follows:

[0019] The infrared video dataset described in step (1) is divided into three parts: 70% as the training set, 20% as the validation set, and 10% as the test set.

[0020] Further, the training of the deep learning model in step (3) includes:

[0021] First, set the model hyperparameters:

[0022] Batch size: Larger batch sizes can make the model converge faster, but the amount of video memory needs to be considered.

[0023] Learning rate: The learning rate affects the speed at which the model parameters are updated. A large learning rate may prevent the model from converging to the extreme value, while a small learning rate may make the model converge more slowly.

[0024] Number of iterations (epoch): The number of times the entire training set is fully trained;

[0025] The model is then trained, and the mean squared error (MSE) loss function is used as a measure of model performance.

[0026]

[0027] Where h represents the actual heat map. This represents the heatmap generated by the model. The physical meaning of this loss function is the average of the squared differences between the predicted and true values ​​for each pixel. The model uses backpropagation of gradients, and the optimizer updates the parameters based on these gradients, making the model's predicted coordinates closer to the true coordinates. During training, there are training losses and validation losses. If both decrease with the same trend, it indicates that the model's predictive ability is gradually improving. When both stop decreasing, it means the model has converged, and training ends.

[0028] Further, step (4) involves using the trained deep learning model to output a keypoint heatmap; specifically:

[0029] (1) First, the wrist is captured on video using an infrared camera;

[0030] (2) Use the wrist video as the model input to generate a key point heatmap;

[0031] Further, in step (5), the key point heatmap of each channel is fitted with the mean value of the circle region to obtain the coordinates of the wrist pulse diagnosis point; specifically, the expected value of the probability within the unit circle is taken as the center of the circle, with the point with the maximum probability value as the center.

[0032] Compared with existing deep learning-based methods for wrist pulse diagnosis point localization, the advantages of this invention are as follows:

[0033] With a small-scale infrared video dataset, the positions of the "cun, guan, chi" wrist pulse diagnosis points can be quickly and accurately located. Attached Figure Description

[0034] Figure 1 This is the overall flowchart of the present invention.

[0035] Figure 2 This serves as the temporal attention prior template for different samples.

[0036] Figure 3 For deep learning model architecture.

[0037] Figure 4 This is a visualization of the wrist pulse diagnosis point prediction using a trained deep learning model combined with key point fitting on test data. Detailed Implementation

[0038] The invention will be further illustrated by experimental examples below.

[0039] I. Parameter Settings

[0040] Set the batch size to 4 and the learning rate to 10. -5 The number of iterations is set to 100.

[0041] II. Experimental Results

[0042] like Figure 4 As shown, the red dots represent the actual coordinates, the blue dots represent the model-predicted coordinates, and the green circles represent the threshold range. After 100 iterations, the model-predicted coordinates are very close to the actual coordinates, indicating that the model can accurately predict the coordinates of the wrist pulse diagnosis point.

[0043] III. Comparison with Keypoint Detection Task Model

[0044] We evaluate the model's localization accuracy based on the Euclidean distance between the predicted and actual coordinates. Accuracy is the ratio of the number of test samples where the Euclidean distance between the predicted and actual coordinates is less than a threshold T to the total number of test samples.

[0045]

[0046] Where S is the test set, S i For the test sample set where the predicted coordinates are within the threshold, y i For the actual coordinates, The model predicts coordinates, and T is the threshold, set to 100, 80, 60, 40, and 20 respectively. The model proposed in this invention is compared with other commonly used keypoint detection architectures, as shown in Table 1. Table 1 shows that the localization accuracy of the architecture proposed in this invention is significantly higher than other model architectures at thresholds of 80 pixels, 60 pixels, 40 pixels, and 20 pixels. This indicates that the architecture based on heatmaps and temporal attention priors is indeed superior to other methods in its ability to capture spatiotemporal features, demonstrating that the architecture proposed in this invention can accurately locate the three wrist pulse diagnosis points: "cun," "guan," and "chi."

[0047] Table 1

[0048]

[0049]

[0050] References

[0051] [1]NEWELLA,YANGK,DENGJ.Stacked hourglass networks for human poseestimation[C] / / European conference on computer vision.2016:483-499.

[0052] [2]CHEN Y,WANG Z,PENG Y,et al.Cascaded pyramid network for multi-person poseestimation[C] / / Proceedings of the IEEE conference on computervision and patternrecognition.2018:7103-7112.

[0053] [3]ESLAMI M,NEUSCHAEFER-RUBE C,SERRURIER A.Automatic vocal tractlandmarklocalization from midsagittal mri data[J].Scientific reports,2020,10(1):1468.[4]CAI Y,WANGZ,LUOZ,et al.Learning delicate local representationsformulti-person pose estimation[C] / / European Conference on ComputerVision.Springer,2020:455-472.

[0054] [5]YANGB,MENGK,LUH,et al.Sfcn:Spoon fully convolutional networks forpulselocalization[C] / / 2022IEEE International Symposium on Circuits andSystems(ISCAS).IEEE,2022:2861-2865.

[0055] [6]RONNEBERGER O,FISCHER P,BROX T.U-net:Convolutional networks forbiomedicalimage segmentation[C] / / International Conference on Medical imagecomputing andcomputer-assisted intervention.Springer,2015:234-241。

Claims

1. A method for locating wrist pulse diagnosis points based on heatmaps and temporal attention priors, characterized in that, The specific steps are as follows: (1) Based on the infrared video dataset, construct the two-dimensional coordinates of the key points of "inch, gate, and ruler" as supervision information; (2) Constructing deep learning networks; The deep learning model uses U-Net as the baseline architecture, combined with a temporal attention prior module; wherein: In the temporal attention prior module, the average gray value of each spatial location in the video is first calculated. The average gray value is subtracted from each frame in the video image set and the absolute value is taken to represent the degree of change of gray value. Then, the difference of each spatial location is added up to obtain the spatial attention map. Finally, a threshold is set to suppress spatial locations with little change to obtain their temporal attention prior. U-Net is divided into contraction and expansion paths, consisting of multiple encoders, decoders, skip connections, max pooling layers, and transposed convolutional layers. The encoders extract features from the input data; the decoders generate heatmaps; skip connections allow the network to combine high-resolution detail with upsampled coarse information, helping to solve the vanishing gradient problem in deep learning models; max pooling layers reduce feature map size, retain important features, reduce computation, and minimize overfitting; transposed convolutional layers increase feature map size for reconstruction. Each encoder or decoder consists of two 3×3 convolutional layers, followed by a ReLU activation layer and a batch normalization layer. The convolutional layers extract features from the input data; the batch normalization layer accelerates model convergence; and the ReLU activation layer enables non-linear fitting, solving complex data fitting problems. Furthermore, based on U-Net's skip connections, temporal attention priors are fused with the encoder's low-level semantic features and the decoder's high-level semantic features, allowing the model to better utilize the temporal information of infrared video and output more accurate keypoint heatmaps. (3) Train the deep learning network; The collected infrared video dataset is divided into a training set, a validation set, and a test set. The training set is used to train the deep learning model to fit the data distribution. The validation set is used to verify the generalization ability of the deep learning model during the training process and record the model with the best generalization ability. The test set is used to perform performance tests on the trained deep learning model to measure its performance. (4) Using the trained deep learning model, obtain the heat map of the key points "cun, guan, chi"; (5) The key point heatmap of each channel is fitted with the mean value of the central region to obtain the coordinates of the wrist pulse diagnosis point.

2. The wrist pulse diagnosis point localization method according to claim 1, characterized in that, In step (1), the two-dimensional coordinates of the key points "cun, guan, chi" are constructed based on the infrared video dataset as supervision information. Specifically, the key points "cun, guan, chi" in the infrared video are labeled by TCM doctors and used as supervision information for the deep learning model.

3. The wrist pulse diagnosis point localization method according to claim 1, characterized in that, The infrared video dataset mentioned in step (3) is divided into a training set, a validation set, and a test set. Specifically, the infrared video dataset mentioned in step (1) is divided into three parts: 70% as the training set, 20% as the validation set, and 10% as the test set.

4. The wrist pulse diagnosis point localization method according to claim 1, characterized in that, Step (3) of training the deep learning model includes: First, set the model hyperparameters, including: batch size, learning rate, and number of iterations; The model is then trained, and the mean squared error (MSE) loss function is used as a measure of model performance. Where h represents the actual heat map. This represents the heatmap generated by the model; the physical meaning of this loss function is the average of the squares of the differences between the predicted and true values ​​of each pixel; the model updates its parameters based on the gradients through backpropagation, making the model's predicted coordinates closer to the true coordinates; during training, it includes training loss and validation loss. If both decrease in the same direction, it indicates that the model's predictive ability is gradually improving. When both stop decreasing, it indicates that the model has converged and training has ended.

5. The wrist pulse diagnosis point localization method according to claim 1, characterized in that, Step (4) involves using the trained deep learning model to output a keypoint heatmap; specifically: (1) First, the wrist is captured on video using an infrared camera; (2) Use the wrist video as the model input to generate a key point heatmap.

6. The wrist pulse diagnosis point localization method according to claim 1, characterized in that, Step (5) describes fitting key points to the heatmap of each channel using the mean value method of the center region to obtain the coordinates of the wrist pulse diagnosis point; specifically, the expected probability value within the unit circle is taken as the center of the circle, with the point with the maximum probability value as the center.

Images