A method for accurately locating the pulse point on the wrist based on near-infrared imaging and deep learning

Abstract

The present invention belongs to the field of computer image processing technology, specifically a method for accurately locating the position of the pulse of a person's wrist based on near-infrared photography and deep learning. The basic steps of the present invention are: first, a hand image of multiple people is sampled by a near-infrared camera, and a professional doctor marks the position of the pulse as training data for a deep learning pulse location network model; then a deep learning network model is constructed, the network model includes a feature extraction network and a regression network, the feature extraction network extracts features such as wrist contour texture color from the training set, and the extracted wrist features are used to predict the position of the pulse through the regression network to obtain a deep learning pulse location network model with strong generalization ability; finally, the captured hand image is used as the input of a pre-trained deep learning pulse location network model to predict the position of the wrist pulse on the image. The method of the present invention can find the pulse position of a person's wrist with high accuracy, and provide real-time visual positioning for robots to perform traditional Chinese medicine pulse diagnosis.

Description

Technical Field

[0001] The invention belongs to the technical field of computer image processing, and in particular relates to a method for accurately locating a pulse point on a human wrist. Background Art

[0002] Pulse diagnosis is the process of taking the pulse. During the pulse diagnosis process, the doctor gently presses the index finger, middle finger and ring finger on a section of the radial artery on the patient's left or right wrist, feeling the pulse change pattern, and analyzing the patient's health condition through clinical experience. Pulse diagnosis is an important basis for clinical diagnosis in traditional Chinese medicine. With the development of sensor technology and electronic information technology, pulse diagnosis digitization has become an important research field. In particular, the rapid rise of artificial intelligence in recent years has provided a new development direction for traditional Chinese medicine pulse diagnosis.

[0003] The robot can locate the pulse mouth through the camera and the pulse mouth positioning algorithm, and can obtain the corresponding pulse information through the mechanical arm and the pressure sensor. In addition, with the upgrading of hardware and the optimization of algorithms, the speed of object detection is getting faster and faster, the detection accuracy of objects is getting higher and higher, and the implementation of object detection algorithms has reached the real-time level. Therefore, the precise positioning of the pulse mouth can be completed by collecting data with an infrared camera and deep learning methods. Summary of the invention

[0004] The purpose of the present invention is to provide a method for accurately locating the position of the pulse point on a person's wrist.

[0005] The method for accurately locating the position of the pulse at the wrist of a person provided by the present invention is based on a near-infrared camera and deep learning technology; the basic steps are: first, hand images of multiple people are sampled by a near-infrared camera, and the pulse positions are marked by professional doctors as training data for a deep learning pulse positioning network model; then a deep learning network model is constructed, the network model includes a feature extraction network and a regression network, the feature extraction network extracts wrist features from the training set, and the pulse position is predicted by the regression network to obtain a deep learning pulse positioning network model with strong generalization ability; finally, the captured hand image is used as the input of a pre-trained deep learning pulse positioning network model to predict the position of the wrist pulse on the image.

[0006] The method for accurately locating the position of the pulse point on the wrist of a person based on deep learning proposed by the present invention has the following specific steps:

[0007] (1) Obtaining training data for deep learning;

[0008] (2) Construct a deep learning network model; segment the collected wrist images; and train the network model;

[0009] The network model structure consists of a feature extraction network and a regression network; wherein, the basic convolution blocks of the feature extraction network are of two types, the first convolution block includes a convolution layer, a batch normalization layer, an activation layer, a maximum pooling layer and a dropout layer; the second convolution block only includes a convolution layer, a batch normalization layer, an activation layer and a dropout layer, and there is no maximum pooling layer; the convolution layers are different in different convolution blocks, and they have different convolution kernels and step sizes; wherein, the convolution layer is used to extract wrist features; the batch normalization layer is used to speed up the convergence speed of the model, making it easier and more stable to train deep network models; the activation layer is used to increase the nonlinear expression ability of the network; the maximum pooling layer is used to reduce the dimension of features, compress the number of data and parameters, reduce overfitting, and improve the fault tolerance of the model; the dropout layer is used to eliminate and weaken the joint adaptability between neuron nodes and enhance the generalization ability; the regression network consists of a fully connected layer, and the fully connected layer is used to integrate the local features extracted by the previous convolution layer and pooling layer to locate the pulse mouth;

[0010] (3) Use the trained network model to predict the pulse location of the collected hand images.

[0011] Furthermore, the specific process of obtaining deep learning training data in step (1) is as follows:

[0012] (11) Sampling hand images of multiple persons using an infrared camera; the infrared camera is composed of an infrared filter added in front of a camera, and is capable of collecting infrared images; the collected infrared images are cropped and scaled according to the shape requirements of the input image by the deep learning model; the number of images sampled for each person is not less than 20; during collection, the shooting height is generally fixed at 20 cm, the wrist is changed to various different postures (i.e., the wrist is in different directions), and it is ensured that the entire hand and wrist are included in the image;

[0013] (12) The pulse location is effectively marked by a professional doctor;

[0014] Furthermore, the convolutional layers in step (2) are different in different convolutional blocks, and they have different convolutional kernels and step sizes. Specifically, the first convolutional block has three convolutional layers, and their convolutional kernels are 7*7, 5*5, and 3*3, respectively, and their step sizes are 3, 2, and 1, respectively; the second convolutional block has four convolutional layers, and their convolutional kernels are all 3*3, and their step sizes are all 1.

[0015] Furthermore, the step (2) of dividing the collected wrist images into a training set, a test set, and a validation set;

[0016] The data set (infrared image) collected in step (1) is divided, of which 20% is used as a test set; then 80% of the data is enhanced, specifically by data enhancement through translation transformation; first, these infrared images are shifted right by 50 pixels to generate new images that are shifted right by 50 pixels relative to the original images, and the generated new images are shifted down to obtain images that are shifted down by 50 pixels relative to the new images, and the above three data sets are combined to obtain images that are three times the size of the original data sets; then these data sets are divided into training sets and validation sets in a ratio of 8:2; the role of the training set is to fit the model, and the deep learning pulse location network model is trained by setting the hyperparameters of the deep learning pulse location network model; the role of the validation set is to use each model to predict the validation set data in order to find the best model after training multiple models through the training set, and record the model accuracy. Select the parameters corresponding to the best model. The role of the test set is to use the test set to predict the model after the training set and validation set have obtained the optimal model, so as to measure the performance and classification ability of the optimal model.

[0017] Furthermore, the training network model in step (2) includes:

[0018] First, set the model hyperparameters:

[0019] Batch size: A larger batch size makes the network easier to converge, but too large a batch size can easily cause insufficient memory. Generally, the batch size can be set to 15-25;

[0020] Learning rate: The learning rate determines the speed of weight update. If it is set too high, the result will exceed the optimal value, and if it is too low, the decline rate will be too slow. The learning rate can be set to 0.0001-0.0005;

[0021] Iterations (steps): The total number of training steps. Generally, the number of iterations can be set to 1000-2000.

[0022] Then the network model is trained. The present invention uses mean square error loss (MSE) as the standard for measuring the model prediction results:

[0023] (1)

[0024] in, is the true value of the pulse mouth position, is the value predicted by the deep learning pulse location network. The training loss and validation loss values ​​are calculated by the formula (1); during the training process, pay attention to the training loss and validation loss values. When both values ​​decrease at the same time, it means that the network prediction ability is gradually increasing. When both values ​​no longer decrease, it means that the network has converged and the training can be ended.

[0025] The above hyperparameters in the model were adjusted multiple times to obtain the model with the best generalization performance.

[0026] Furthermore, in step (3), the trained network model is used to predict the pulse position of the collected hand image. The specific process is as follows:

[0027] (31) First, the hand image is sampled using an infrared camera;

[0028] (32) The collected hand image is input into the trained network model, and the wrist features are extracted by the feature extraction network. Then the image is input into the regression network model to obtain the prediction result, which is the final pulse position.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] (1) The user only needs to provide a hand image to accurately find the location of the pulse;

[0031] (2) Provide a visual positioning method for robots to perform traditional Chinese medicine pulse diagnosis. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is the overall flow chart of the present invention.

[0033] Figure 2 It is a network structure used for precise positioning of the pulse mouth.

[0034] Figure 3 Data enhancement is performed by shifting for step (22). Where (a) is a sample of the collected hand image, (b) is shifting image a to the right, and (c) is shifting image b downward.

[0035] Figure 4 The changes of validation loss value and training loss value during training. Among them, (a) is the change of validation loss value, and (b) is the change of training loss value.

[0036] Figure 5 To extract features such as wrist contour, texture, color, etc. from wrist images, it is the feature map obtained in different convolutional layers of the deep learning pulse location network.

[0037] Figure 6As described in step (32), the hand image to be predicted is put into the pre-trained model for regression to obtain the predicted pulse position. The blue dot in the figure is the predicted pulse position. DETAILED DESCRIPTION

[0038] 1. Data Collection and Data Enhancement

[0039] Sample hand images of multiple people, with no less than 20 samples for each person, and a fixed shooting height of 20 cm. The wrists have different postures during sampling (that is, the wrists are in different directions), ensuring that the entire hand and wrist are included in the image during sampling. In the case of insufficient sample number, data enhancement can be performed by shifting the samples to enhance the generalization ability of the network. Figure 3 As shown in FIG. 8 , firstly, 873 images are shifted right by 50 pixels to generate new images that are shifted right by 50 pixels relative to the original images. The generated new images are then shifted down to obtain images that are shifted down by 50 pixels relative to the new images. The above three data sets are combined to obtain a total of 2619 images.

[0040] 2. Parameter settings

[0041] Adjust the parameters for different training processes to obtain the model with the best prediction effect. The batch size is set to 25, the number of iterations is set to 1000, and the learning rate is initialized to 10 -3 .

[0042] 3. Training Process

[0043] See also Figure 4 As shown in the figure, Figure (a) is the validation loss value, and Figure (b) is the training loss value, which are calculated by formula (1). The simultaneous decrease of both indicates that the network prediction ability is gradually increasing. When both no longer decrease, it means that the network has converged and the training can be ended. The vertical axis is the loss value, and the horizontal axis is the number of iterations. It can be seen that when the number of iterations exceeds 1000, the loss value almost no longer decreases.

[0044] See also Figure 5 As shown, the wrist features are learned layer by layer by the pulse location network.

[0045] See also Figure 6 As shown in the figure, we can see that after 1000 training iterations, the true value is the red dot, the predicted value is the blue dot, and the green circle is a circle with the true value position as the center and the corresponding threshold as the radius. It can be seen that the predicted values ​​are all within the green circle, indicating that the prediction results are close to the true value.

[0046] 4. Comparison with Classical Networks

[0047] We evaluate the accuracy of the pulse location results based on pixel distance. Accuracy is the distance between the predicted position and the actual position.

[0048] Less than the set threshold T The ratio of the number of images in the test set to the number of images in the test set:

[0049] ,

[0050] (2)

[0051] N is the test set, N in is a set of images with accurate predictions, y i Is the test set I i The real pulse position in the picture, It is through

[0052] The network predicts the location, T is the threshold, which is set to 50, 40, 30, 25 and 15 pixels respectively. We compare our proposed network model with some classic network architectures (i.e., Visual Geometry Group Network [1] (VGG16) and Residual Network [2] (ResNet-50), as shown in Table 1, it is found that the performance of the proposed network is close to Res-Net50 at large thresholds (i.e., 50, 40, and 30 pixels). However, the performance is better at small thresholds (i.e., 25 and 15 pixels). Compared with these two networks, our proposed network model has fewer parameters.

[0053] Table 1

[0054] .

[0055] References

[0056] 1. SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[C] / / International Conference on LearningRepresenta-tions. [Sl: sn], 2015.

[0057] 2. He K, Zhang X, Ren S, et al. Deep Residual Learning for ImageRecognition[C] / / 2016 IEEE Conference on Computer Vision and PatternRecognition (CVPR).[S.l.: s.n.], 2016: 770-778。

Claims

1. A method for accurately locating the pulse point of a person's wrist based on deep learning, characterized in that: First, hand images of multiple people are sampled through a near-infrared camera, and the pulse positions are marked by professional doctors as training data for a deep learning pulse position location network model; then a deep learning network model is constructed, which includes a feature extraction network and a regression network. The feature extraction network extracts features such as wrist contour, texture, and color from the training set, and the extracted wrist features are used to predict the pulse position through a regression network to obtain a deep learning pulse position location network model with strong generalization ability; finally, the captured hand image is used as the input of a pre-trained deep learning pulse position location network model to predict the position of the wrist pulse on the image; The specific steps are as follows: (1) Obtain training data for deep learning; (2) Construct a deep learning positioning network model; divide the collected wrist images; and train the network model; The network model structure consists of a feature extraction network and a regression network; wherein, the basic convolution blocks of the feature extraction network have two types, the first convolution block includes a convolution layer, a batch normalization layer, an activation layer, a maximum pooling layer and a dropout layer; the second convolution block only includes a convolution layer, a batch normalization layer, an activation layer and a dropout layer, and no maximum pooling layer; the convolution layers are different in different convolution blocks, and they have different convolution kernels and step sizes; wherein, the convolution kernel in the convolution layer can process each small pixel area on the image and extract image texture information; the edge information in the image is extracted through multiple convolution layers, and the higher layers are summarized. Structural information is used to identify the contour of the radius and the contour of the wrist, and other features to assist in locating the pulse position; the batch normalization layer is used to accelerate the convergence speed of the model, making the training of deep network models easier and more stable; the activation layer is used to increase the nonlinear expression ability of the network; the maximum pooling layer is used to reduce the dimension of features, compress the number of data and parameters, reduce overfitting, and improve the fault tolerance of the model; the dropout layer is used to eliminate the weakening of the joint adaptability between neuronal nodes and enhance the generalization ability; the regression network is composed of a fully connected layer, which is used to integrate the local features extracted by the previous convolutional layer and pooling layer to locate the pulse position; The collected wrist images are divided into a training set, a test set, and a validation set; The data set collected in step (1), i.e., the infrared images, is divided, wherein 20% is used as a test set; then, the data of the 80% portion is enhanced: first, these infrared images are shifted right by 50 pixels to generate new images that are shifted right by 50 pixels relative to the original images, and the generated new images are shifted downward to obtain images that are shifted downward by 50 pixels relative to the new images, and the above three data sets are combined to obtain images that are three times the size of the original data sets; then, these data sets are divided into a training set and a validation set in a ratio of 8:2; the training set is used to fit the model, and the deep learning pulse location network model is trained by setting the hyperparameters of the deep learning pulse location network model; the role of the validation set is to use each model to predict the validation set data in order to find the best model after multiple models are trained through the training set, and the model accuracy is recorded to select the parameters corresponding to the best model; the role of the test set is to use the test set to predict the model after the training set and the validation set have obtained the optimal model, so as to measure the performance and classification ability of the optimal model; (3) Use the trained network model to predict the pulse location of the collected hand images.

2. The method for accurately locating the position of the pulse point of a person's wrist based on deep learning according to claim 1 is characterized in that: The specific process of obtaining deep learning training data in step (1) is as follows: (11) Hand images of multiple people are sampled using an infrared camera; the collected infrared images are cropped and scaled according to the shape requirements of the deep learning pulse location network model for the input image; the number of images sampled for each person is not less than 20; during the collection, the shooting height is fixed at 20 cm, the wrist is transformed into various different postures, and the entire hand and wrist are ensured to be included in the image; (12) The pulse location is effectively marked by a professional doctor.

3. The method for accurately locating the position of the pulse point of a person's wrist based on deep learning according to claim 1 is characterized in that: The step (2) of training the deep learning pulse location network model includes: (1) First, set the model hyperparameters: Batch size: set to 15-25; Learning rate: set to 0.0001-0.0005; Iterations: set to 1000-2000; (2) Then the network model is trained, and the mean square error loss (MSE) is used as the standard for measuring the model prediction results: Among them, y i ∈R 2 is the true value of the pulse mouth position, is the position value predicted by the deep learning pulse location network model; the training loss and validation loss values ​​are calculated by formula (1); during the training process, pay attention to the training loss and validation loss values. If both values ​​decrease at the same time, it means that the network prediction ability is gradually increasing. When both values ​​no longer decrease, it means that the network has converged and the training is ended; (3) The above-mentioned hyperparameters in the model are adjusted multiple times to obtain the deep learning pulse location network model with the best generalization performance.

4. The method for accurately locating the position of the pulse point of a person's wrist based on deep learning according to claim 3 is characterized in that: In step (3), the trained network model is used to predict the pulse position of the collected hand image. The specific process is as follows: (31) First, the hand image is sampled using an infrared camera; (32) The collected hand image is input into the trained network model, and the wrist features are extracted by the feature extraction network. Then, the prediction result is obtained through the regression network model, which is the final pulse position.

5. The method for accurately locating the position of the pulse point on the wrist of a person based on deep learning according to any one of claims 1 to 4, characterized in that: In the deep learning network model, the convolution layer has different convolution kernels and step sizes in different convolution blocks, as follows: the first convolution block has three convolution layers, their convolution kernels are 7*7, 5*5, and 3*3, and their step sizes are 3, 2, and 1, respectively; the second convolution block has four convolution layers, their convolution kernels are all 3*3, and their step sizes are all 1.

Images