A hand acupoint automatic positioning method based on fuzzy inference system

Abstract

The application discloses a hand acupoint automatic positioning method based on a fuzzy reasoning system, and comprises the following specific steps: collecting image data of a hand, obtaining 21 hand navigation points through a hand key point extraction network of Mediapipe from a sample picture, and then obtaining a mask of the hand region; creating a local coordinate system for each hand navigation point; generating a special region of each hand navigation point by using a deep learning algorithm, and generating potential acupoint sampling points; converting the potential acupoint points into scaled coordinates as input for subsequent processing; calculating the confidence degree of the potential acupoint sampling points belonging to a certain type of acupoint points, and finally determining the acupoint points by selecting the highest confidence degree, thereby completing the automatic positioning of the hand acupoints. The application not only improves the accuracy and efficiency of hand acupoint positioning, but also has the advantages of strong category scalability, confidence degree output and the like.

Description

Technical Field

[0001] This invention relates to the technology and application of intelligent positioning of acupoints on the hand in Traditional Chinese Medicine, and more specifically, to an automatic positioning method for acupoints on the hand based on a fuzzy reasoning system. Background Technology

[0002] Currently, there are two main types of methods for extracting acupoints on the hand: neural network retraining method and empirical formula method.

[0003] Neural network retraining requires acquiring the original labeled data and then performing secondary labeling to expand new categories. However, this labeling is very expensive; it also requires acquiring the structure and parameters of the original network and then repeating the training, which places high demands on computer performance; furthermore, the retraining method does not take into account the experience and fuzzy characteristics of acupoint location in traditional Chinese medicine, and is entirely based on data-driven learning.

[0004] Another type is the empirical formula method, which converts the description of acupoints in traditional Chinese medicine into a calculation formula to obtain the location of acupoints. The parameters of these formulas are fixed, and the results cannot give the confidence level that the point belongs to an acupoint.

[0005] Chinese Patent: A method for precise acupoint location based on cascaded deep neural networks, application patent number: CN202310411627.5, applicant: Ningbo University. The acupoint location method disclosed in this patent document is obtained by neural network training.

[0006] Chinese Patent: An Automatic Method for Locating Hand Acupoints, Patent Application No.: CN202110734019.9, Applicant: Zhejiang University. The acupoint positioning method disclosed in this patent document is obtained by empirical formula method, and the confidence level is unstable.

[0007] This technical solution proposes a category-expandable automatic hand acupoint localization method based on fuzzy reasoning system, which fully considers the fuzziness and empirical nature of TCM acupoint diagnosis and uses fuzzy theory to describe and model acupoints. It aims to explore a new method for applying neural network and fuzzy control technology to the localization of hand acupoints in TCM. Summary of the Invention

[0008] This invention discloses an automatic hand acupoint location method based on a fuzzy inference system, the main purpose of which is to overcome the above-mentioned deficiencies and shortcomings of the prior art.

[0009] The technical solution adopted in this invention is as follows:

[0010] An automatic hand acupoint localization method based on a fuzzy inference system, the method comprising the following specific steps:

[0011] Step 1: Collect image data of the hand and label some acupoints to form the original sample;

[0012] Step 2: Use Mediapipe's hand keypoint extraction network to obtain 21 hand landmarks from the sample image. Then, input the hand landmarks and the hand sample image from Step 1 into the MobileSAM segmentation network to obtain a mask for the hand region.

[0013] Step 3: Based on TCM acupoint data, create a local coordinate system for each hand navigation point;

[0014] Step 4: Based on Euclidean distance, deep learning algorithms are used to generate a dedicated region for each hand beacon point in the hand mask space, generating potential acupoint sampling points;

[0015] Step 5: Obtain the scale information of key points from the maximum connectivity of each key point region on the hand, and convert potential acupoints into scaled coordinates as input for subsequent processing;

[0016] Step Six: Based on the TCM diagnosis and treatment guidelines, set the corresponding membership function and fuzzy inference table to calculate the confidence level of potential acupoint sampling points belonging to a certain type of acupoint. The final acupoint is determined by selecting the highest confidence level, thus completing the automatic location of empty areas on the hand.

[0017] Furthermore, in step one, the hand is in a relaxed state when the hand image data is collected.

[0018] Furthermore, in step two, the 21 hand navigation points include the joints of each finger and the joints of the wrist, and the hand navigation points are connected by lines.

[0019] Furthermore, in step two, the input is fed into the MobileSAM segmentation network, and the mask of the hand region is extracted using the knn nearest neighbor algorithm.

[0020] Furthermore, the creation of a local coordinate system for each hand-held beacon point in step three includes:

[0021] The line segment combination A of the thumb-sized navigation beacon points is: L1, L2, L3, L4;

[0022] Line segment combination B of the index finger segment navigation beacon point: L5, L6, L7, L8;

[0023] The line segment combination C of the middle finger section navigation beacon point is: L10, L11, L12;

[0024] The line segment combination D of the navigation beacon point is: L14, L15, L16;

[0025] The line segment combination E of the small finger segment navigation beacon point is: L18, L19, L20, L21;

[0026] The line segment combination F for the navigation beacon points in the palm section: L9, L13, L17;

[0027] Based on A to E, a local acupoint coordinate system of navigation points was constructed. The Vonoroi algorithm was used to divide the space into multiple regions according to the 21 navigation points, and the bounding box of the region where the navigation point is located was extracted as the setting anchor quantity of the scale to mitigate the influence of scale transformation and individual finger shape differences.

[0028] Taking the C8 coordinate system as an example, the origin of the coordinate system is set to p8, and the ordinate is set to the vector y = (x8-x7, y8-y7) between p7 and p8. The vector V between p5 and p9 is selected. 5-9 = (x9-x5, y9-y5) is used as a reference vector to determine the direction of the x-axis, which is x = (x8-x7, y8-y7)*sgn(V 5-9 ·y ⊥ ), that is, the x-axis of C-8 in this article points in the direction of the thumb, where y ⊥ =(-y8+y7, x8-x7), sgn(V 5-9 ·y ⊥ ) is a vector V 5-9 With y ⊥ The sign function of the dot product, where, Normalize vectors y and x to obtain v y8 and v x8 ,then,

[0029] For any point on the image, its coordinates in the C8 coordinate system are: ((x-x8, y-y8)·v x8 ,(x-x8,y-y8)·v y8 ).

[0030] Furthermore, based on the 21 extracted navigation points, the KNN algorithm is used to divide the gesture region, thus forming a unique region for each navigation point. The height h and width w of its bounding box are calculated. By calculating the maximum connectivity of each key point region, the scale information of the key points is obtained, and the potential acupoints are converted into scaled coordinates. On this basis, the C8 coordinates are further scaled to obtain the scale-C8 coordinates as: ((x-x8, y-y8)gv x8 / w,(x-x8,y-y8)gv y8 / h), this coordinate will serve as the precision input in the subsequent fuzzy inference system;

[0031] After constructing the local coordinate system of the beacon points, each point on the image can be represented in its local coordinate system. All the original pixels in the region of beacon point i can be represented by a set. To represent, n i This represents the number of pixels in the region, and the vertex coordinates are then mapped to a scaled local coordinate system.

[0032]

[0033] Furthermore, in step six, setting the corresponding membership function and fuzzy inference table according to the TCM diagnosis and treatment guidelines includes the following steps:

[0034] (1) Fuzzification module: The input precise information x-coordinate and y-coordinate are fuzzified and converted into a fuzzy set on a given universe of discourse;

[0035] (2) Membership functions of fuzzy input and output: (a) Membership function of input fuzzy quantity x; (b) Membership function of input fuzzy quantity y; (c) Membership function of output fuzzy quantity.

[0036] (3) Fuzzy rule base: Establish a database containing fuzzy rules related to acupoints;

[0037] (4) Fuzzy reasoning method: Based on the input fuzzy facts and activated fuzzy rules, select an appropriate fuzzy reasoning method to perform reasoning;

[0038] (5) Defuzzification module: Defuzzifies the fuzzy conclusions obtained from reasoning to obtain more specific and accurate output results.

[0039] As can be seen from the above description and explanation of the present invention, compared with the prior art, the advantages of the present invention are as follows:

[0040] 1. No need for secondary labeled data: This method does not require obtaining the original labeled data and performing secondary labeling to expand new categories, thus reducing labeling costs. Furthermore, this method does not require obtaining the original network training structure and parameters, avoiding the tedious process of repeated training.

[0041] 2. Considering the fuzzy characteristics of TCM acupoint location: This method utilizes a fuzzy inference system model to model the fuzzy characteristics of TCM acupoint location, which can better adapt to the actual needs of TCM acupoint location. Furthermore, this method incorporates experience in TCM acupoint location, improving the accuracy of acupoint location.

[0042] 3. Strong scalability of categories: This method uses a fuzzy inference system model for acupoint location, which has the characteristics of strong scalability of categories. When it is necessary to add a new acupoint category, only the corresponding fuzzy inference system needs to be added, without the need to retrain the entire model.

[0043] 4. Confidence Output: This method can output the predicted value of the acupoint location and the confidence level that the point belongs to the acupoint, providing patients with more comprehensive acupoint location information.

[0044] This technical solution proposes a class-expandable automatic hand acupoint localization method based on a fuzzy inference system. This method not only overcomes the shortcomings of existing technologies and improves the accuracy and efficiency of hand acupoint localization, but also possesses advantages such as strong class scalability and confidence output. The implementation of this method will bring new opportunities and challenges to the development of the field of traditional Chinese medicine acupoint therapy. Attached Figure Description

[0045] Figure 1 This is a flowchart of the overall process structure of the present invention.

[0046] Figure 2 This is a structural diagram of the hand acupoints collected and labeled by the present invention.

[0047] Figure 3 This is a schematic diagram of the structure for the extraction of navigational markers and the extraction of hand masks in this invention.

[0048] Figure 4 This is a schematic diagram of the structure of the navigation beacon and connecting line markings of the present invention.

[0049] Figure 5 This is a schematic diagram of the coordinate system of the navigation beacon point of the present invention.

[0050] Figure 6 This is a schematic diagram of the structure of the navigation mark area and its elliptical bounding frame in this invention.

[0051] Figure 7 This is a schematic diagram of the membership function for fuzzy input and output in this invention.

[0052] Figure 8 This is the fuzzy decision plane diagram of the present invention.

[0053] Figure 9 This is the detection diagram of the Ten Xuan acupoints in the fuzzy inference system of this invention. Detailed Implementation

[0054] The specific embodiments of the present invention will be further described and illustrated below with reference to the accompanying drawings.

[0055] like Figures 1 to 9 As shown, an automatic hand acupoint localization method based on a fuzzy inference system is described, the method comprising the following specific steps:

[0056] Step 1: Collect image data of the hand and label some acupoints to form the original sample;

[0057] Step 2: Use Mediapipe's hand keypoint extraction network to obtain 21 hand landmarks from the sample image. Then, input the hand landmarks and the hand sample image from Step 1 into the MobileSAM segmentation network to obtain a mask for the hand region.

[0058] Step 3: Based on TCM acupoint data, create a local coordinate system for each hand navigation point;

[0059] Step 4: Based on Euclidean distance, deep learning algorithms are used to generate a dedicated region for each hand beacon point in the hand mask space, generating potential acupoint sampling points;

[0060] Step 5: Obtain the scale information of key points from the maximum connectivity of each key point region on the hand, and convert potential acupoints into scaled coordinates as input for subsequent processing;

[0061] Step Six: Based on the TCM diagnosis and treatment guidelines, set the corresponding membership function and fuzzy inference table to calculate the confidence level of potential acupoint sampling points belonging to a certain type of acupoint. The final acupoint is determined by selecting the highest confidence level, thus completing the automatic location of empty areas on the hand.

[0062] In step one, the hand image data is collected when the hand is in a relaxed state.

[0063] like Figure 2 As shown, image data of the hand was collected and some acupoints were labeled to form the original sample. To better identify the acupoints, this invention required the volunteers' hands to be in a relaxed state, and images were collected based on this. To verify the accuracy and effectiveness of the method of this invention, some images were selected and acupoints were labeled by professional TCM doctors, thus constructing a dataset containing various gestures and acupoint annotations. We divided the dataset into training and testing sets to ensure the model's generalization ability on different datasets.

[0064] like Figure 3 As shown, we extracted hand landmarks and region masks. After obtaining the original samples, we used the MediaPipe hand keypoint extraction network to process the image, obtaining 21 landmarks for the hand. These landmarks contain key feature information of the hand, such as finger joints and wrists, and some landmarks were connected. Next, we input these landmarks and the hand image together into the MobileSAM segmentation network, and extracted the mask for the hand region using a deep learning algorithm. This mask will be used for subsequent acupoint region localization and recognition.

[0065] like Figure 4As shown, creating a local coordinate system and potential acupoint sampling points typically requires the subject to spread their fingers to better distinguish different finger areas. This process usually involves several variations: slight finger bending; changes in relative displacement between fingers; and individual differences in finger shape. Due to these variations, the relative positional relationships of the navigation points also change to varying degrees. However, we found that several sets of point combinations exhibit a certain degree of stability; these line segment combinations are as follows:

[0066] The line segment combination A of the thumb-sized navigation beacon points is: L1, L2, L3, L4;

[0067] Line segment combination B of the index finger segment navigation beacon point: L5, L6, L7, L8;

[0068] The line segment combination C of the middle finger section navigation beacon point is: L10, L11, L12;

[0069] The line segment combination D of the navigation beacon point is: L14, L15, L16;

[0070] The line segment combination E of the small finger segment navigation beacon point is: L18, L19, L20, L21;

[0071] The line segment combination F for the navigation beacon points in the palm section: L9, L13, L17;

[0072] Based on coordinates A through E, a local coordinate system for navigational aids is constructed. Within A through E, each navigational aid exhibits invariance in translational and rotational scales, and also invariance to slight curvature. F is invariant to all the aforementioned deformations. This invention constructs a local coordinate system based on this.

[0073] The key elements of a local coordinate system are the origin, coordinate axes, and scale. The construction of a local coordinate system needs to facilitate the extraction of target acupoints. Based on the distribution of some common acupoints on the palm, a local acupoint coordinate system for navigation marks was constructed.

[0074] like Figure 5 As shown, the Vonoroi algorithm is used to divide the space into multiple regions based on 21 navigation points, and the bounding box of the region where the navigation point is located is extracted as the setting anchor quantity of the scale to mitigate the influence of scale transformation and individual finger shape differences; where the green line is the vertical axis, the blue line is the horizontal axis, and the black dot is the origin of the local coordinate system.

[0075] Taking the C8 coordinate system as an example, the origin of the coordinate system is set to p8, and the ordinate is set to the vector y = (x8-x7, y8-y7) between p7 and p8. The vector V between p5 and p9 is selected. 5-9 = (x9-x5, y9-y5) is used as a reference vector to determine the direction of the x-axis, which is x = (x8-x7, y8-y7)*sgn(V 5-9·y ⊥ ), that is, the x-axis of C-8 in this article points in the direction of the thumb, where y ⊥ =(-y8+y7, x8-x7), sgn(V 5-9 ·y ⊥ ) is a vector V 5-9 With y ⊥ The sign function of the dot product, where, Normalize vectors y and x to obtain v y8 and v x8 ,then,

[0076] For any point on the image, its coordinates in the C8 coordinate system are: ((x-x8, y-y8)·v x8 ,(x-x8,y-y8)·v y8 ).

[0077] Scaled coordinates and potential acupoint sampling: Based on the 21 extracted navigation points, the k-nearest neighbor (KNN) algorithm is used to divide the gesture region, thus forming a unique region for each navigation point. The height h and width w of its bounding box are calculated. By calculating the maximum connectivity of each keypoint region, the scale information of the keypoints is obtained, and the potential acupoints are converted into scaled coordinates. Based on this, the C8 coordinates are further scaled to obtain the scale-C8 coordinates as: ((x-x8, y-y8)gv x8 / w,(x-x8,y-y8)gv y8 / h), this coordinate will serve as the precision input in the subsequent fuzzy inference system;

[0078] like Figure 6 As shown, the beacon point region and its elliptical bounding box are shown in (a) and (b) respectively. After constructing the local coordinate system of the beacon points, each point on the image can be represented in its local coordinate system. All the original pixels in the beacon point i region can be represented by a set. To represent, n i This represents the number of pixels in the region, and the vertex coordinates are then mapped to a scaled local coordinate system.

[0079] like Figure 7 As shown, step six, setting the corresponding membership function and fuzzy inference table according to the TCM diagnosis and treatment guidelines, includes the following steps:

[0080] (1) Fuzzification module: The input precise information x coordinate and y coordinate are fuzzified and converted into a fuzzy set on a given domain; for the ten acupoints, the coordinate system is C4, C8, C16, C20.

[0081] The x-coordinate can be set to linguistic values ​​P, ZE, N, and the y-coordinate can also be set to linguistic values ​​P, ZE, N. The fuzzy output is the probability that the x and y coordinates are 10, with values ​​ranging from [0 to 1]. We set their linguistic values ​​to L and H. Membership functions for fuzzy input and output: (a) Membership function of input fuzzy quantity x; (b) Membership function of input fuzzy quantity y; (c) Membership function of output fuzzy quantity.

[0082] (2) Fuzzy rule base: Establish a database containing fuzzy rules related to the Ten Xuan acupoints; the rules should be designed and verified based on TCM theory and practical experience.

[0083] The verification control table is as follows:

[0084]

[0085] (3) Fuzzy reasoning method: Based on the input fuzzy facts and activated fuzzy rules, an appropriate fuzzy reasoning method is selected for reasoning; this invention uses the Mamdani reasoning method for reasoning.

[0086] (4) A fuzzy control system is implemented to achieve fuzzy reasoning. The reasoning process should be able to handle imprecise and uncertain information and generate a set of possible conclusions. After fuzzy reasoning, fuzzy decisions will be generated. Figure 8 .

[0087] (5) Defuzzification module: Defuzzifies the fuzzy conclusions obtained from reasoning to obtain more specific and accurate output results.

[0088] like Figure 9 As shown, the fuzzy inference system detects the ten acupoints. (a) Ten acupoints detected by this invention, with blue dots representing the detected acupoints; (b) Manually labeled ten acupoints, with green dots representing the ten acupoints. The ten-xuan fuzzy inference system can be easily extended to other common acupoints, such as the ten-xuan, Shaofu, and Zhongchong acupoints.

[0089] This patent utilizes the design of a fuzzy inference system model and the application of a fuzzy inference system to enable the method to achieve high-precision automatic positioning of acupoints on the hand in traditional Chinese medicine without relying on a large amount of labeled data.

[0090] First, the design of the fuzzy inference system model fully considers the fuzzy characteristics of acupoint location in Traditional Chinese Medicine (TCM). By combining fuzzy logic with neural network technology, it successfully integrates empirical knowledge of TCM acupoints into the model. This enables the model to more accurately reflect the actual distribution patterns and characteristics of TCM acupoints when dealing with acupoint location problems, thereby improving the accuracy of acupoint location.

[0091] Secondly, the application of the fuzzy inference system further enhances the practicality of this patented technology. By further processing the predicted values ​​output by the fuzzy inference system, not only can the final acupoint location result be obtained, but also the confidence level that the point belongs to an acupoint can be output. This function provides patients with more comprehensive and reliable acupoint location information, helping them to better understand and master the relevant knowledge of acupoint therapy.

[0092] The above are merely specific embodiments of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial improvements made to the present invention using this concept should be considered as infringing on the protection scope of the present invention.

Claims

1. An automatic hand acupoint localization method based on a fuzzy inference system, characterized in that: The method includes the following specific steps: Step 1: Collect image data of the hand and label some acupoints to form the original sample; Step 2: Use Mediapipe's hand keypoint extraction network to obtain 21 hand landmarks from the sample image. Then, input the hand landmarks and the hand sample image from Step 1 into the MobileSAM segmentation network to obtain a mask for the hand region. Step 3: Based on TCM acupoint data, create a local coordinate system for each hand navigation point; Step 4: Based on Euclidean distance, deep learning algorithms are used to generate a dedicated region for each hand beacon point in the hand mask space, generating potential acupoint sampling points; Step 5: Obtain the scale information of key points from the maximum connectivity of each key point region on the hand, and convert potential acupoints into scaled coordinates as input for subsequent processing; Step Six: Based on the TCM diagnosis and treatment guidelines, set the corresponding membership function and fuzzy inference table to calculate the confidence of potential acupoint sampling points belonging to a certain type of acupoint. The final acupoint is determined by selecting the highest confidence, thus completing the automatic location of empty areas on the hand. Step three involves creating a local coordinate system for each hand-held navigation point, including: The line segment combination A of the thumb-sized navigation beacon points is: L1, L2, L3, L4; Line segment combination B of the index finger segment navigation beacon point: L5, L6, L7, L8; The line segment combination C of the middle finger section navigation beacon point is: L10, L11, L12; The line segment combination D of the navigation beacon point is: L14, L15, L16; The line segment combination E of the small finger segment navigation beacon point is: L18, L19, L20, L21; The line segment combination F for the navigation beacon points in the palm section: L9, L13, L17; Based on A~E, a local acupoint coordinate system of navigation points was constructed. The Vonoroi algorithm was used to divide the space into multiple regions according to the 21 navigation points. The bounding box of the region where the navigation point is located was extracted as the setting anchor quantity of the scale to mitigate the influence of scale transformation and individual finger shape differences. Taking the C8 coordinate system as an example, the origin of the coordinate system is set to p8, and the ordinate is set to the vector y = (x8-x7, y8-y7) between p7 and p8. The vector V between p5 and p9 is selected. 5-9 =(x9-x5,y9-y5) is used as a reference vector to determine the direction of the x-axis, which is x=(x8-x7,y8-y7)*sgn(V 5-9 ·y ⊥ ), that is, the x-axis of C-8 in this article points in the direction of the thumb, where y ⊥ =(-y8+y7,x8-x7), sgn(V) 5-9 ·y ⊥ ) is a vector V 5-9 With y ⊥ The sign function of the dot product, where, Normalize vectors y and x to obtain v y8 and v x8 ,then, For any point on the image, its coordinates in the C8 coordinate system are: ((x-x8, y-y8)·v x8 ,(x-x8,y-y8)·v y8 ); Based on the 21 extracted navigation points, the KNN algorithm is used to divide the gesture region, thus forming a unique region for each navigation point. The height h and width w of its bounding box are calculated. By calculating the maximum connectivity of each key point region, the scale information of the key points is obtained, and the potential acupoints are converted into scaled coordinates. Based on this, the C8 coordinates are further scaled to obtain the scale-C8 coordinates: This coordinate will serve as the precise input in the subsequent fuzzy inference system; After constructing the local coordinate system of the beacon points, every point on the image can be represented in its local coordinate system. All original pixels on the region are passed through a set To express, This represents the number of pixels in the region, and the vertex coordinates are then mapped to a scaled local coordinate system. .

2. The automatic hand acupoint localization method based on a fuzzy inference system according to claim 1, characterized in that: In step one, the hand image data is collected when the hand is in a relaxed state.

3. The automatic hand acupoint localization method based on a fuzzy inference system according to claim 1, characterized in that: In step two, the 21 hand navigation points include the joints of each finger and the joints of the wrist, and the hand navigation points are connected by lines.

4. The automatic hand acupoint localization method based on a fuzzy inference system according to claim 1, characterized in that: In step two, the data is input into the MobileSAM segmentation network, and the mask of the hand region is extracted using the k-nearest neighbor algorithm.

5. The automatic hand acupoint localization method based on a fuzzy inference system according to claim 4, characterized in that: Step six, in which the corresponding membership function and fuzzy inference table are set according to the TCM diagnosis and treatment guidelines, includes the following steps: (1) Fuzzification module: The input precise information x-coordinate and y-coordinate are fuzzified and converted into a fuzzy set on a given universe of discourse; (2) Membership functions of fuzzy input and output: (a) Membership function of input fuzzy quantity x; (b) Membership function of input fuzzy quantity y; (c) Membership function of output fuzzy quantity; (3) Fuzzy rule base: Establish a database containing fuzzy rules related to acupoints; (4) Fuzzy reasoning method: Based on the input fuzzy facts and activated fuzzy rules, select an appropriate fuzzy reasoning method to perform reasoning; (5) Defuzzification module: Defuzzifies the fuzzy conclusions obtained from reasoning to obtain more specific and accurate output results.

Images