Auricular point projection labeling positioning method based on multi-modal data fusion

Abstract

The invention discloses an auricular point projection labeling positioning method based on multi-modal data fusion, and particularly relates to the field of auricular point positioning, and the auricular point projection labeling positioning method comprises the steps of S1, collection terminal arrangement, S2, image mapping, S3, multi-source data feature extraction, S4, feature fusion, S5, offset correction, and S6, visual display. According to the method, through multi-angle image acquisition and three-dimensional point cloud modeling of acquisition terminal arrangement and image mapping, auricle depth features and auricle temperature features of multi-source data are fused, similarity analysis of comprehensive feature vectors and standard auricular point features is carried out, offset calculation and correction are carried out in combination with an image mapping result, and the accuracy of auricular point correction is improved. The method can dynamically adapt to auricle morphological variation of different individuals, realizes upgrading from standardized model mapping to individualized precise positioning, is especially suitable for auricle morphological variation under different ages, genders or pathological states, and enhances the universality and clinical practicability of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the ear acupoint positioning technical field, more particularly, the present application relates to an ear acupoint projection labeling positioning method based on multi-modal data fusion. BACKGROUND

[0002] Due to the significant individual differences in auricle morphology and the dense distribution of ear acupoints, the traditional positioning method relying on anatomical landmarks and doctor experience is easily affected by subjective judgment. In addition, the clinical effect of ear acupoint therapy is highly dependent on positioning accuracy, but the traditional method lacks objective quantitative standards, which restricts its promotion in modern medicine. Therefore, it is difficult for doctors who have not undergone long-term professional training to accurately determine the positions of ear acupoints of different people.

[0003] For example, the existing invention patent with publication number CN117934841A discloses an ear acupoint positioning method, which includes: obtaining an ear image; using a trained ASM model to extract features from the ear image, determining a plurality of first feature points constituting an acupoint region in the ear image, and the connection order of each first feature point; inputting the ear image into a trained ear acupoint positioning model to obtain each acupoint region in the ear image, and determining the ear acupoint features corresponding to each acupoint region; adjusting the first feature points based on the ear acupoint features to obtain second feature points of each acupoint region; and connecting the second feature points in the order of the connection of the first feature points to obtain each acupoint region in the ear image.

[0004] However, in actual use, there are still some shortcomings, first, the existing method relies on two-dimensional RGB images and cannot accurately reflect the spatial position of ear acupoints, resulting in three-dimensional space positioning deviation. Based on this situation, it is necessary to combine multi-source data for multi-modal data fusion to extract ear acupoint features, but the existing operation to deal with this positioning deviation is mostly limited to positioning ear acupoints based on only the morphological features of two-dimensional ear images, which limits the positioning effect and poses a risk of not meeting the positioning needs. Second, the existing method uses ASM to extract feature points, relies on a pre-set statistical shape model, and has poor robustness to individual differences, image noise and light and shadow interference. The local search strategy of ASM is easily affected by auricle wrinkles and hair obstruction, leading to misjudgment of feature points, and lacks a quantitative offset correction mechanism, which cannot adaptively adjust the morphological differences between individual auricles and standard models. SUMMARY

[0005] Therefore, the embodiments of the present application provide an ear acupoint projection labeling positioning method based on multi-modal data fusion, which effectively solves the problems raised in the background art.

[0006] To achieve the above purpose, the present application provides the following technical scheme: S1: Collect terminal layout: During the ear positioning process, image acquisition terminal and heat perception terminal are laid out, and multi-angle images of the auricle are acquired according to the image acquisition terminal, and the auricle heat state image is extracted according to the heat perception terminal; S2: Image mapping: The multi-angle image of the auricle is corrected to obtain the auricle depth image, and the auricle three-dimensional point cloud model is generated, and then the auricle three-dimensional point cloud model is matched with the standard ear acupuncture point model, so that the standard ear acupuncture point model is mapped on the auricle three-dimensional point cloud model; S3: Multi-source data feature extraction: auricle depth data and auricle temperature data are extracted according to the auricle depth image and the auricle heat state image respectively, so that the auricle depth feature and the auricle temperature feature are extracted; S4: Feature fusion: based on the auricle depth feature and the auricle temperature feature in S3, feature processing is carried out, including feature standardization, feature weighting and feature correlation analysis, so that the ear acupuncture related area and the ear acupuncture high probability point are extracted, and a comprehensive feature vector is generated; S5: Offset correction: based on the comprehensive feature vector, the feature similarity with each ear acupuncture standard feature is obtained, and the offset amount is analyzed according to the image mapping result of S2, and then the ear acupuncture related area is offset corrected; S6: Visual display: the final ear acupuncture positioning result is output in a visual way, and the corresponding anatomical semantics and feature information are displayed, and various data generated in the positioning process are stored.

[0007] Technical effects and advantages of the present application: 1. Through multi-angle image acquisition and three-dimensional point cloud modeling of image acquisition terminal layout and image mapping, combined with auricle depth feature and auricle temperature feature fusion of multi-source data, the dual constraints of geometric shape and physiological feature are realized, and the auricle positioning accuracy and reliability are improved; the three-dimensional point cloud model can accurately depict the individual anatomical differences of the auricle, and the temperature feature can reflect the physiological activity of the auricle area, after the integration of feature standardization, weighting and correlation analysis, the error of single morphological positioning is effectively avoided, so that the positioning result not only conforms to the anatomical structure but also fits the physiological function feature, and the accuracy and reliability of positioning are significantly improved; 2. Through similarity analysis of the comprehensive feature vector and the standard ear acupuncture feature, combined with the offset amount calculation and correction of the image mapping result, the auricle morphological variation of different individuals can be dynamically adapted, the traditional standard model mapping is easily affected by individual differences, and the method realizes the upgrade from the standardized model mapping to the individualized accurate positioning through the closed loop process of feature matching, offset analysis and accurate adjustment, especially suitable for auricle morphological changes in different ages, genders or pathological states, and enhances the universality and clinical practicability of the method. BRIEF DESCRIPTION OF DRAWINGS

[0008] Fig. 1The figure is a schematic diagram of the overall structure of the present application.

[0009] Fig. 2 The figure is a standardization process flowchart of the present application.

[0010] Fig. 3 The figure is a correlation analysis flowchart of the present application. DETAILED DESCRIPTION

[0011] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0012] As shown in the accompanying drawings Figs. 1-3 A ear acupoint projection marking positioning method based on multi-modal data fusion includes an image acquisition terminal and a heat perception terminal.

[0013] The image acquisition terminal is used for photographing the measured ear during ear acupoint positioning, and can be a high-precision industrial camera, which is fixed to an adjustable support at an angle of 120° around the auricle, and specifically covers the front side, back side and top surface of the auricle to ensure that more than 90% of the full surface of the auricle is covered by the multi-angle image.

[0014] The heat perception terminal is used for heat perception of the measured ear, and can be an infrared thermal sensor, which can non-contact measure the temperature of the measured ear to generate a heat map without direct contact, thereby avoiding the interference introduced by the contact sensor. On the other hand, by forming a heat map, the temperature distribution of each region of the measured ear can be comprehensively understood, and the infrared thermal sensor has a smaller volume and a convenient installation method, which is easy to deploy during ear acupoint positioning.

[0015] The specific embodiment of the present application includes the following steps: S1: terminal layout: during ear acupoint positioning, the image acquisition terminal and the heat perception terminal are laid out, and the multi-angle image of the auricle is obtained according to the image acquisition terminal, and the auricle heat state image is extracted according to the heat perception terminal.

[0016] S2: image mapping: the multi-angle image of the auricle is corrected to obtain an auricle depth image, and an auricle three-dimensional point cloud model is generated, and then the auricle three-dimensional point cloud model is matched with a standard ear acupoint model, so that the standard ear acupoint model is mapped to the corresponding ear acupoint in the auricle three-dimensional point cloud model.

[0017] In this embodiment, it needs to be specifically pointed out that the auricle depth image is obtained by correcting the multi-angle image of the auricle and then performing multi-view stereo matching. For example, the correction can be performed by Zhang Zhengyou calibration method to calibrate the internal parameters of the image acquisition terminal, and then perform pixel-by-pixel geometric correction. The matching relationship between the corresponding pixel points is determined by extracting feature points such as corner points and edges in the image, thereby generating the angle between each pixel point, and then generating an auricle three-dimensional point cloud model. The auricle three-dimensional point cloud model is used to represent the measured ear structure.

[0018] It needs to be added that multi-view image matching is a technology that acquires images of the same scene from different angles and aligns and fuses multiple images. The purpose is to obtain a larger 2D view or a 3D representation of the scanned scene. The curved surface characteristics of the auricle as a non-planar anatomical structure will cause radial distortion and tangential distortion in camera imaging. The Zhang Zhengyou calibration method can accurately calculate the camera internal parameter matrix and distortion coefficient, map the pixel coordinates of the original image to the ideal coordinate system without distortion, and provide a geometric reference for subsequent depth image generation. The specific process belongs to the prior art, which will not be repeated here.

[0019] It needs to be further explained that the mapping between the auricle three-dimensional point cloud model and the standard auricular point model is as follows: the auricle three-dimensional point cloud model is automatically identified to obtain auricle core points, for example, the auricle core points can be the tragus incisure, the starting point of the helix foot, the top point of the upper helix foot, the midpoint of the back edge of the concha, the lowest point of the ear lobe, and the tip of the antitragus. The auricle core points are numbered, and the corresponding auricle core points in the standard auricular point library are also numbered. According to the one-to-one correspondence principle, the auricle core points with corresponding numbers are extracted from the auricle three-dimensional point cloud model and the standard auricular point model to form a mapping group.

[0020] It needs to be added that the standard auricular point model is obtained according to the 91 auricular points in the GB / T13734-2008 national standard, a standard auricular point library is generated, and the positions of each auricular point and the corresponding auricle position are integrated.

[0021] S3: Multi-source data feature extraction: auricle depth data and auricle temperature data are extracted from the auricle depth image and the auricle heat state image, respectively, thereby extracting auricle depth features and auricle temperature features.

[0022] In this embodiment, it needs to be specifically pointed out that the auricle depth feature extraction is as follows: The auricle depth image is grid processed, taking the tragus incisure as the origin, the x-axis as the left-right direction, the y-axis as the front-back direction, and the z-axis as the up-down direction, to generate a three-dimensional point cloud coordinate system with a preset resolution, marked as P(x, y, z), wherein the preset resolution is 0.2mm, and then the curvature of each point is calculated, which is specifically represented as: , Where k represents the surface curvature at each point, Zx and Zy are the rates of change of depth Z with respect to the x-axis and y-axis, respectively. Specifically, by calculating the first-order partial derivatives of Z with respect to x and y, the degree of tilt of the surface in the horizontal and vertical directions can be quantified. Zxy represents the mixed partial derivatives of depth Z with respect to the x and y axes, reflecting the degree of distortion of the surface. Extract the z-axis coordinate value of each point in the 3D point cloud coordinate system as the depth value; The points of the ear to be tested are integrated into a set of test points, and its centroid is calculated. Similarly, the centroids of the corresponding points of the standard acupoint model are calculated. Then, the translation difference between the centroid of the ear to be tested and the centroid of the standard acupoint model is calculated. Specifically, it is expressed as: R=P-P1+Q1, where P0 represents the translation difference between the centroid of the ear to be tested and the centroid of the standard acupoint model, P represents the set of test points of the ear to be tested, and P1 and Q1 represent the centroid of the ear to be tested and the centroid of the standard acupoint model, respectively. The surface curvature, depth value, and translation difference corresponding to each point are integrated into auricular depth data, and then the auricular depth data is constructed into an auricular depth feature vector, labeled as He=[k, ta, R].

[0023] Furthermore, the auricular temperature feature extraction is as follows: Extract the number of temperature distribution regions and the chromaticity values ​​corresponding to the temperature distribution regions; The chromaticity values ​​corresponding to each temperature distribution area are compared with the temperatures represented by each chromaticity in the heat map to obtain the temperatures corresponding to each temperature distribution area. The average temperature and standard deviation of the entire map are calculated based on the temperature corresponding to each temperature distribution area. The segmentation threshold is set as the sum of the average temperature of the entire map and 0.8. Areas that are greater than or equal to the segmentation threshold are marked as heat-sensitive areas. Extract the area of ​​each heat-sensitive region and calculate the equivalent diameter of each heat-sensitive region, as shown below: Dh represents the equivalent diameter of each heat-sensitive region, S represents the area of ​​each heat-sensitive region, and π represents the natural constant. Extract the maximum and minimum temperature values ​​of each heat-sensitive region, combine them with the equivalent diameter, and then compare the difference between the maximum and minimum temperature values ​​with the equivalent diameter to obtain the temperature gradient change rate corresponding to each heat-sensitive region. Based on the average temperature and standard deviation of the entire map, combined with the temperature of each heat-sensitive area, the normal coefficient of ear acupoint temperature is obtained by subtracting the temperature of each heat-sensitive area from the average temperature of the entire map and comparing it with the standard deviation. The equivalent diameter, temperature gradient change rate, and normal coefficient of ear acupoint temperature of each heat-sensitive area are integrated into auricular temperature data, and then the auricular temperature data are constructed into an auricular temperature feature vector, labeled as Te=[Dh, Ca, Pn].

[0024] It should be noted that in a healthy state, there is a temperature difference between the ear acupoint and the surrounding tissue, and in a pathological state, the temperature difference can be expanded to form a recognizable hot area. At the same time, due to certain diseases such as digestive system diseases and cardiovascular diseases, the temperature abnormality rate of the corresponding ear acupoint is significantly higher than that of healthy people. Therefore, through temperature feature extraction, the ear acupoint can be located while assisting in disease screening or efficacy monitoring.

[0025] S4: Feature fusion: based on the auricle depth feature and the auricle temperature feature in S3, feature processing is performed, including feature standardization, feature weighting and feature correlation analysis, thereby extracting the ear acupoint related area and the ear acupoint high probability point, and generating a comprehensive feature vector.

[0026] In this embodiment, it needs to be specifically pointed out that the feature standardization specifically includes: standardizing the auricle depth feature and the auricle temperature feature: A1: Extract the curvature of the surface, the depth value and the translation difference corresponding to each point in the auricle depth feature, thereby calculating the average surface curvature and the curvature standard deviation, the average depth value and the depth value standard deviation, and the average translation difference and the translation difference standard deviation; A2: Subtract the average surface curvature, the average depth value and the average translation difference from the curvature of the surface, the depth value and the translation difference corresponding to each point respectively, and compare them with the curvature standard deviation, the depth value standard deviation and the translation difference standard deviation, thereby obtaining the standard curvature of the surface, the depth value and the translation difference; A3: Similarly, the equivalent diameter, the temperature gradient change rate and the ear acupoint temperature normal coefficient of each heat-sensitive region in the auricle temperature feature are extracted, and the standard equivalent diameter, the temperature gradient change rate and the ear acupoint temperature normal coefficient of each heat-sensitive region are obtained.

[0027] The feature correlation analysis specifically includes: performing correlation analysis on the auricle depth feature and the auricle temperature feature: B1: According to the mapping relationship between the auricle three-dimensional point cloud model and the standard ear acupoint model, the auricle depth feature and the auricle temperature feature are aligned in the auricle three-dimensional point cloud model; B2: Extract the auricle depth feature and establish an auricle depth feature model, thereby calculating the auricle depth feature factor, which is specifically represented as: , where He0 represents the auricle depth feature factor, k0, ta0 and R0 respectively represent the standard curvature of the surface, the depth value and the translation difference corresponding to each point, w1, w2 and w3 respectively represent the weight coefficients corresponding to the standard curvature of the surface, the depth value and the translation difference, wherein w1>0, w2>0 and w3>0, and w1+w2+w3=1, and exemplarily, w1, w2 and w3 are 0.4, 0.4 and 0.2 respectively; B3: Extract auricular temperature features and establish an auricular temperature feature model, thereby calculating the auricular temperature feature factor, specifically expressed as follows: , Where Te0 represents the auricle temperature characteristic factor, Dh0, Ca0 and Pn0 represent the standard equivalent diameter, temperature gradient change rate and auricular temperature normal coefficient corresponding to each point, respectively, and a1, a2 and a3 represent the weight coefficients corresponding to the standard equivalent diameter, temperature gradient change rate and auricular temperature normal coefficient, respectively, where a1>0, a2>0 and a3>0, and a1+a2+a3=1. For example, a1, a2 and a3 are 0.3, 0.4 and 0.3, respectively. B4: Calculate the correlation coefficient between the auricle depth feature factor and the auricle temperature feature factor to obtain the feature correlation degree; B5: Set an association threshold and calculate the absolute value of the feature association degree. When the feature association degree is greater than or equal to the association threshold, it indicates that there is a strong correlation between the auricular depth feature factor and the auricular temperature feature factor. At this time, the points corresponding to the auricular depth feature factor and the auricular temperature feature factor are regarded as high-probability points of auricular acupoints, and the corresponding areas are regarded as auricular acupoint related areas. When the feature association degree is less than the association threshold, it indicates that there is a weak correlation between the auricular depth feature factor and the auricular temperature feature factor. At this time, the points corresponding to the auricular depth feature factor are discarded.

[0028] It should be added that aligning the auricle depth features with the auricle temperature features in the auricle 3D point cloud model can ensure that each depth point matches the temperature value at the corresponding location; the association threshold can be calculated by performing feature association degree calculation on a large number of samples in clinical data and taking the maximum value as the association threshold.

[0029] It should be explained that the correlation coefficient usually refers to the Pearson correlation coefficient, which measures the strength and direction of the linear relationship between two variables. If the correlation coefficient is close to 1, it indicates that there is a strong positive correlation between the two variables, that is, they change in the same direction; if the correlation coefficient is close to -1, it indicates that there is a strong negative correlation between the two variables, that is, they change in opposite directions; if the correlation coefficient is close to 0, it indicates that there is no correlation between the two variables.

[0030] Feature weighting calculates the weighted sum of auricular depth and auricular temperature feature factors at high-probability points corresponding to auricular acupoints in the auricular acupoint-related regions. Specifically, this includes: C1: The weights corresponding to the auricle depth feature factors are calculated, specifically represented as: b H =|r|b H / (max|r|+V), where b Hrepresents the ear depth feature factor corresponding weight, r represents the feature correlation degree of the ear depth feature factor and the ear temperature feature factor, max|r| represents the maximum feature correlation degree in the auricular point related area, V is a natural constant, and exemplarily, V = 1e-6; C2: calculate b in the same way T as the ear temperature feature factor corresponding weight; C3: calculate the comprehensive feature vector by combining the ear depth feature factor and the ear temperature feature factor corresponding weight, and the specific representation is: Z = He0 x b H + Te0 x b T , wherein Te0 and He0 represent the ear temperature feature factor and the ear depth feature factor.

[0031] It should be noted that the maximum feature correlation degree in the auricular point related area is added to the natural constant to avoid meaningless denominator in the weight calculation formula.

[0032] S5: obtain the feature similarity with each auricular point standard feature based on the comprehensive feature vector, and perform offset analysis according to the image mapping result of S2, and then perform offset correction on the auricular point related area.

[0033] In this embodiment, it should be noted that the feature similarity is obtained by cosine similarity calculation of the comprehensive feature vector and each auricular point standard feature, and then the offset analysis is performed, including linear offset analysis and angle offset analysis, and the linear offset analysis is as follows: According to the image mapping result of S2, the initial positioning point coordinates of the auricular point are extracted, and the corresponding auricular point high probability points in the auricular point related area are extracted in the auricular point three-dimensional point cloud model, and the coordinates thereof are extracted as high probability positioning point coordinates; The comprehensive feature vector of the auricular point high probability point is calculated with each auricular point standard feature, and thus each feature similarity is compared, the highest feature similarity is extracted, and the initial positioning point coordinates of the auricular point corresponding to the highest feature similarity are recorded; The high probability positioning point coordinates are compared with the initial positioning point coordinates of the auricular point, and the linear offset is calculated, and the specific representation is: ΔP = Pm-Ps, wherein the initial positioning point coordinates of the auricular point represent the linear offset, Pm and Ps represent the initial positioning point coordinates of the auricular point and the high probability positioning point coordinates respectively; The auricular point linear rationality is calculated by formula 1-ΔP.

[0034] It needs to be explained that due to the mapping relationship between the auricle three-dimensional point cloud model and the standard auricular point model corresponding to the auricular point, at this time the feature similarity of the comprehensive feature vector acquisition corresponding to the auricular point high probability point and each auricular point standard feature is the feature similarity of the comprehensive feature vector acquisition corresponding to the auricular point high probability point and the auricular point initial positioning point, so the auricular point initial positioning point coordinates can be obtained by each auricular point standard feature.

[0035] It needs to be further explained that the angle offset amount analysis is as follows: the auricular point high probability point normal vector and each auricular point standard normal vector are compared, and the angle offset amount is calculated, which reflects the direction deviation caused by the difference of the auricle surface, which is specifically represented as: , Where ΔT represents the angle offset amount, n m and n s respectively represent the auricular point high probability point normal vector and each auricular point standard normal vector; The auricular point angle rationality is calculated by formula 1-ΔT.

[0036] Further, the auricular point linear rationality and the auricular point angle rationality are weighted and averaged to obtain the auricular point rationality index, and then a rationality threshold is set, the auricular point rationality index is compared with the rationality threshold, if the auricular point rationality index is greater than or equal to the rationality threshold, the auricular point high probability point is taken as the final auricular point, if the auricular point rationality index is less than the rationality threshold, the ICP algorithm is iteratively adjusted until the auricular point rationality index is greater than or equal to the rationality threshold.

[0037] S6: The final auricular point positioning result is output in a visual manner, such as marking the positions of each auricular point corresponding to the measured ear on the auricle three-dimensional model, and displaying the corresponding anatomical semantics and feature information, wherein the anatomical semantics is the corresponding name of each auricular point, which provides accurate position basis for subsequent auricular point diagnosis and treatment, and at the same time, various data generated in the positioning process are stored for subsequent analysis and research.

[0038] Secondly: the drawings of the disclosed embodiments of the present application only involve the structures involved in the disclosed embodiments of the present application, other structures can refer to the usual design, and in the case of no conflict, the same embodiments and different embodiments of the present application can be combined with each other; Finally: the above only describes the preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for ear acupoint projection annotation and localization based on multimodal data fusion, characterized in that, include: S1: Deployment of acquisition terminals: During the auricular acupoint positioning process, image acquisition terminals and heat sensing terminals are deployed, and multi-angle images of the auricle are obtained based on the image acquisition terminals, and the heat state images of the auricle are extracted based on the heat sensing terminals. S2: Image mapping: Correct the multi-angle image of the auricle, obtain the depth image of the auricle, generate a three-dimensional point cloud model of the auricle, and then match the three-dimensional point cloud model of the auricle with the standard auricular acupoint model. Thus, the auricular acupoints corresponding to the standard auricular acupoint model are mapped in the three-dimensional point cloud model of the auricle. S3: Multi-source data feature extraction: Extract the corresponding auricle depth data and auricle temperature data based on the auricle depth image and auricle thermal state image, respectively, and thereby extract auricle depth features and auricle temperature features; S4: Feature Fusion: Based on the auricle depth and auricle temperature features in S3, feature processing is performed, including feature standardization, feature weighting, and feature correlation analysis, thereby extracting auricle-related regions and high-probability auricle points, and generating a comprehensive feature vector. S5: Offset Correction: Based on the comprehensive feature vector, the feature similarity with the standard features of each ear acupoint is obtained, and the offset is analyzed according to the image mapping results of S2, and then the offset of the ear acupoint related areas is corrected. S6: Visualization: Output the final auricular acupoint location results in a visual manner, and display their corresponding anatomical semantics and feature information, and store various data generated during the location process.

2. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The specific process of mapping the three-dimensional point cloud model of the auricle to the standard auricular acupoint model is as follows: the core points of the auricle are automatically identified in the three-dimensional point cloud model of the auricle. For example, the core points of the auricle can be the tragus notch, the starting point of the crus of the helix, the apex of the superior crus of the antihelix, the midpoint of the posterior edge of the cymba conchae, the lowest point of the earlobe, and the tip of the antitragus. The core points of the auricle are numbered. Similarly, the core points of the auricle corresponding to the standard auricular acupoint database are numbered. According to the principle of one-to-one correspondence of numbers, the core points of the auricle with corresponding numbers are extracted from the three-dimensional point cloud model of the auricle and the standard auricular acupoint model to form a mapping group.

3. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The auricle depth feature extraction is specifically as follows: The auricle depth image is meshed, with the tragus notch as the origin, the x-axis representing the left-right direction, the y-axis the front-back direction, and the z-axis the up-down direction, generating a 3D point cloud coordinate system with a preset resolution, labeled P(x, y, z), where the preset resolution is 0.2 mm. The surface curvature of each point is then calculated, specifically as follows: ， Where k represents the surface curvature at each point, Zx and Zy are the rates of change of depth Z with respect to the x-axis and y-axis, respectively. Specifically, by calculating the first-order partial derivatives of Z with respect to x and y, the degree of tilt of the surface in the horizontal and vertical directions can be quantified. Zxy represents the mixed partial derivatives of depth Z with respect to the x and y axes, reflecting the degree of distortion of the surface. Extract the z-axis coordinate value of each point in the 3D point cloud coordinate system as the depth value; The points of the ear to be tested are integrated into a set of test points, and its centroid is calculated. Similarly, the centroids of the corresponding points of the standard acupoint model are calculated. Then, the translation difference between the centroid of the ear to be tested and the centroid of the standard acupoint model is calculated. Specifically, it is expressed as: R=P-P1+Q1, where P0 represents the translation difference between the centroid of the ear to be tested and the centroid of the standard acupoint model, P represents the set of test points of the ear to be tested, and P1 and Q1 represent the centroid of the ear to be tested and the centroid of the standard acupoint model, respectively. The surface curvature, depth value, and translation difference corresponding to each point are integrated into auricular depth data, and then the auricular depth data is constructed into an auricular depth feature vector, labeled as He=[k, ta, R].

4. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The auricular temperature feature extraction is specifically as follows: Extract the number of temperature distribution regions and the chromaticity values ​​corresponding to the temperature distribution regions; The chromaticity values ​​corresponding to each temperature distribution area are compared with the temperatures represented by each chromaticity in the heat map to obtain the temperatures corresponding to each temperature distribution area. The average temperature and standard deviation of the entire map are calculated based on the temperature corresponding to each temperature distribution area. The segmentation threshold is set as the sum of the average temperature of the entire map and 0.

8. Areas that are greater than or equal to the segmentation threshold are marked as heat-sensitive areas. Extract the area of ​​each heat-sensitive region and calculate the equivalent diameter of each heat-sensitive region, as shown below: Dh represents the equivalent diameter of each heat-sensitive region, S represents the area of ​​each heat-sensitive region, and π represents the natural constant. Extract the maximum and minimum temperature values ​​of each heat-sensitive region, combine them with the equivalent diameter, and then compare the difference between the maximum and minimum temperature values ​​with the equivalent diameter to obtain the temperature gradient change rate corresponding to each heat-sensitive region. Based on the average temperature and standard deviation of the entire map, combined with the temperature of each heat-sensitive area, the normal coefficient of ear acupoint temperature is obtained by subtracting the temperature of each heat-sensitive area from the average temperature of the entire map and comparing it with the standard deviation. The equivalent diameter, temperature gradient change rate, and normal coefficient of ear acupoint temperature of each heat-sensitive area are integrated into auricular temperature data, and then the auricular temperature data are constructed into an auricular temperature feature vector, labeled as Te=[Dh, Ca, Pn].

5. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The feature standardization specifically includes: standardizing the auricle depth feature and the auricle temperature feature. A1: Extract the surface curvature, depth value, and translation difference corresponding to each point in the auricle depth feature, and calculate the average surface curvature and curvature standard deviation, average depth value and depth value standard deviation, average translation difference and translation difference standard deviation. A2: Subtract the surface curvature, depth value, and translation difference corresponding to each point from the average surface curvature, average depth value, and average translation difference, and compare them with the standard deviation of curvature, the standard deviation of depth value, and the standard deviation of translation difference to obtain the standard surface curvature, depth value, and translation difference. A3: Similarly, the equivalent diameter, temperature gradient change rate, and normal coefficient of ear acupoint temperature of each heat-sensitive region in the auricle temperature characteristics are extracted, and the standard equivalent diameter, temperature gradient change rate, and normal coefficient of ear acupoint temperature of each heat-sensitive region are obtained.

6. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The feature correlation analysis specifically includes: performing correlation analysis on auricle depth features and auricle temperature features. B1: Based on the mapping relationship between the three-dimensional point cloud model of the auricle and the standard auricular acupoint model, align the auricle depth features and auricle temperature features in the three-dimensional point cloud model of the auricle. B2: Extract auricular depth features and establish an auricular depth feature model, thereby calculating the auricular depth feature factor, specifically expressed as follows: ， Where He0 represents the auricle depth feature factor, k0, ta0 and R0 represent the standard surface curvature, depth value and translation difference corresponding to each point, respectively, w1, w2 and w3 represent the weight coefficients corresponding to the standard surface curvature, depth value and translation difference, respectively, where w1>0, w2>0 and w3>0, and w1+w2+w3=1. For example, w1, w2 and w3 are 0.4, 0.4 and 0.2 respectively. B3: Extract auricular temperature features and establish an auricular temperature feature model, thereby calculating the auricular temperature feature factor, specifically expressed as follows: ， Where Te0 represents the auricle temperature characteristic factor, Dh0, Ca0 and Pn0 represent the standard equivalent diameter, temperature gradient change rate and auricular temperature normal coefficient corresponding to each point, respectively, and a1, a2 and a3 represent the weight coefficients corresponding to the standard equivalent diameter, temperature gradient change rate and auricular temperature normal coefficient, respectively, where a1>0, a2>0 and a3>0, and a1+a2+a3=1. For example, a1, a2 and a3 are 0.3, 0.4 and 0.3, respectively. B4: Calculate the correlation coefficient between the auricle depth feature factor and the auricle temperature feature factor to obtain the feature correlation degree; B5: Set an association threshold and calculate the absolute value of the feature association degree. When the feature association degree is greater than or equal to the association threshold, it indicates that there is a strong correlation between the auricular depth feature factor and the auricular temperature feature factor. At this time, the points corresponding to the auricular depth feature factor and the auricular temperature feature factor are regarded as high-probability points of auricular acupoints, and the corresponding areas are regarded as auricular acupoint related areas. When the feature association degree is less than the association threshold, it indicates that there is a weak correlation between the auricular depth feature factor and the auricular temperature feature factor. At this time, the points corresponding to the auricular depth feature factor are discarded.

7. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The feature weighting process calculates the weighted sum of the auricular depth feature factor and the auricular temperature feature factor at high-probability points corresponding to auricular acupoints in the auricular acupoint-related region. Specifically, it includes: C1: The weights corresponding to the auricle depth feature factors are calculated, specifically represented as: b H =|r|b H / (max|r|+V), where b H The weights corresponding to the auricular depth feature factors are represented by r, the feature correlation between the auricular depth feature factors and the auricular temperature feature factors are represented by max|r|, the maximum feature correlation in the auricular acupoint related region is represented by max|r|, and V is a natural constant, for example, V=1e-6; C2: Calculate b similarly T As the weight corresponding to the auricular temperature characteristic factor; C3: By combining the auricle depth feature factor and the auricle temperature feature factor with their corresponding weights, a comprehensive feature vector is calculated, specifically expressed as: Z = He0 × b H +Te0×b T , where Te0 and He0 represent the auricular temperature characteristic factor and the auricular depth characteristic factor, respectively.

8. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 1, characterized in that: The feature similarity is obtained by calculating the cosine similarity between the comprehensive feature vector and the standard features of each auricular acupoint, and then offset analysis is performed, including linear offset analysis and angular offset analysis. The linear offset analysis is as follows: Based on the image mapping results, the initial coordinates of the auricular acupoints are extracted for S2. At the same time, the high-probability points of the auricular acupoints in the relevant regions are mapped to the three-dimensional point cloud model of the auricle, and their coordinates are extracted as the coordinates of the high-probability positioning points. The feature similarity between the comprehensive feature vector corresponding to the high probability point of the ear acupoint and the standard feature of each ear acupoint is calculated. The feature similarity of each feature is then compared, the highest feature similarity is extracted, and the coordinates of the initial positioning point of the ear acupoint corresponding to the highest feature similarity are recorded. Compare the coordinates of the high-probability positioning point with the coordinates of the initial positioning point of the ear acupoint, and calculate the linear offset, which is specifically expressed as: ΔP=Pm-Ps, where the coordinates of the initial positioning point of the ear acupoint represent the linear offset, and Pm and Ps represent the coordinates of the initial positioning point of the ear acupoint and the coordinates of the high-probability positioning point, respectively. The linearity of auricular acupoints is calculated using formula 1-ΔP.

9. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 8, characterized in that: The angle offset analysis is as follows: The normal vector of the high-probability point of the auricle is compared with the standard normal vector of each auricle point, and the angle offset is calculated accordingly, reflecting the directional deviation caused by the difference in the curvature of the auricle. Specifically, it is expressed as follows: ， Where ΔT represents the angular offset, n m and n s These represent the normal vector of the high-probability point of the ear acupoint and the standard normal vector of each ear acupoint, respectively. The rationality of the ear acupoint angle is calculated using formula 1-ΔT.

10. The auricular acupoint projection annotation and localization method based on multimodal data fusion according to claim 8, characterized in that: The offset correction calculates a weighted average of the linear rationality and the angular rationality of the acupoint to obtain an acupoint rationality index. A rationality threshold is then set, and the acupoint rationality index is compared with the rationality threshold. If the acupoint rationality index is greater than or equal to the rationality threshold, the high-probability acupoint is selected as the final acupoint. If the acupoint rationality index is less than the rationality threshold, iterative adjustments are made using the ICP algorithm until the acupoint rationality index is greater than or equal to the rationality threshold.

Images