Tongue diagnosis basic information collection method and system

Abstract

The present invention discloses a tongue diagnosis basic information collection method and system, which relates to the field of artificial intelligence technology, including collecting infrared thermal imaging data, near-infrared multi-spectral reflectance data, RGB color images, three-dimensional point cloud data and wetness data of the tongue to generate a multimodal data set; extracting geometric features of the tongue surface from the three-dimensional point cloud data and the RGB color image, combining with dynamic microcirculation feature vectors to generate a morphology-microcirculation comprehensive feature matrix; extracting color and texture features in the RGB color image, combining with wetness features and the morphology-microcirculation comprehensive feature matrix to form a multimodal pathological feature vector; constructing an intelligent diagnosis model based on the multimodal pathological feature vector, performing disease risk assessment, and outputting a disease risk score result; through matching the intelligent diagnosis model with a disease feature template, quantitative assessment of disease risk can be achieved, and an intuitive risk score can be output.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for collecting basic information on tongue diagnosis. Background Art

[0002] In recent years, with the rapid development of artificial intelligence and medical image processing technology, tongue diagnosis, as one of the important means of TCM diagnosis, has gradually received attention from modern scientific research. Traditional tongue diagnosis mainly relies on the doctor's subjective experience, and disease judgment is made by observing the color, shape and other characteristics of the tongue. However, modern tongue diagnosis technology is developing in the direction of dataization and objectivity. Through multimodal data collection combined with artificial intelligence algorithms, it can more accurately extract the pathological characteristics of the tongue and improve the accuracy and reliability of disease diagnosis. This tongue diagnosis method based on multimodal data not only retains the theoretical advantages of TCM tongue diagnosis, but also can quantify tongue characteristics through modern technical means, providing a scientific basis for disease risk assessment.

[0003] However, the existing tongue diagnosis technology still has many shortcomings. First, the single-modality data collection method cannot fully reflect the multidimensional pathological characteristics of the tongue, especially in terms of microcirculatory function, tongue wetness, and morphological structure. In addition, most of the existing multimodal data fusion technologies remain at the simple data splicing level, lacking efficient feature extraction and cross-modal relationship modeling capabilities, making it difficult to fully explore the potential correlation between multimodal data. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a method for collecting basic information of tongue diagnosis, which solves the problem of how to comprehensively quantify tongue pathological characteristics and improve the accuracy of tongue diagnosis disease risk assessment through multimodal data fusion and dynamic feature analysis.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a tongue diagnosis basic information collection method, which comprises:

[0008] Collect infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data, and wetness data of the tongue to generate a multimodal data set;

[0009] The infrared thermal imaging data and near-infrared multispectral reflectance data in the multimodal data set are used to perform dynamic microcirculation feature analysis, extract the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtain the dynamic microcirculation feature vector;

[0010] The geometric features of the tongue surface were extracted from the 3D point cloud data and RGB color images, and combined with the dynamic microcirculation feature vectors to generate a morphological-microcirculation comprehensive feature matrix.

[0011] Extract color and texture features from RGB color images, combine moisture features and morphology-microcirculation comprehensive feature matrix to form a multimodal pathological feature vector;

[0012] Based on multimodal pathological feature vectors, an intelligent diagnosis model is constructed to conduct disease risk assessment and output disease risk score results;

[0013] Based on the risk score results, a health score is given to each area to generate personalized health recommendations.

[0014] As a preferred embodiment of the tongue diagnosis basic information collection method of the present invention, the infrared thermal imaging data, near-infrared multi-spectral reflectance data, RGB color image, three-dimensional point cloud data and wetness data of the tongue are collected to generate a multimodal data set, including the following steps:

[0015] Start the infrared thermal imager to record the temperature distribution on the surface of the tongue;

[0016] The multi-spectral reflectance data of the tongue surface is collected by near-infrared multi-spectral imaging equipment;

[0017] A high-resolution RGB camera is used to obtain color images of the tongue surface;

[0018] Use a structured light 3D scanner to perform a full-area scan of the tongue to obtain point cloud data of the tongue;

[0019] Start the wetness sensor to record the wetness and evaporation rate of the tongue surface;

[0020] The collected data are integrated to form a multimodal dataset.

[0021] As a preferred solution of the tongue diagnosis basic information collection method of the present invention, the infrared thermal imaging data and multi-spectral reflectance data in the multimodal data set are subjected to dynamic microcirculation feature analysis, the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface are extracted, and the dynamic microcirculation feature vector is obtained, which includes the following steps:

[0022] Start the infrared thermal imaging device and extract the temperature data of each area in the infrared thermal imaging data;

[0023] Perform time series analysis on the temperature data of each area to extract the changes in the temperature fluctuation amplitude and the temperature difference between areas, and perform discrete Fourier transform on the temperature time series data of each area to extract the frequency components of the temperature fluctuation and obtain the temperature fluctuation characteristics;

[0024] The collected multi-spectral reflectance data is normalized, and the blood oxygen saturation and blood flow velocity of the tongue are calculated to obtain the blood flow distribution characteristics;

[0025] The extracted temperature fluctuation features and blood flow distribution features are integrated to generate the tongue dynamic microcirculation feature vector.

[0026] As a preferred solution of the tongue diagnosis basic information collection method of the present invention, wherein: extracting the geometric features of the tongue surface from the three-dimensional point cloud data and the RGB color image, combining the dynamic microcirculation feature vector, and generating the morphology-microcirculation comprehensive feature matrix includes the following steps:

[0027] Statistical filtering method is used to remove noise points in 3D point cloud data;

[0028] The denoised point cloud data is reconstructed using the Poisson reconstruction algorithm to generate a continuous three-dimensional model of the tongue surface.

[0029] Using the collected RGB color images, the texture information of the tongue surface is mapped onto the continuous 3D model based on the nearest neighbor interpolation UV mapping method to obtain a complete 3D model of the tongue.

[0030] Calculate the curvature of each point in the complete three-dimensional model of the tongue and extract the geometric features of the tongue surface;

[0031] According to the anatomical features of the tongue, the complete 3D model is divided into three regions: the tongue tip, the middle of the tongue, and the tongue root. The extracted geometric features of the tongue surface are mapped to the corresponding regions to obtain complete 3D morphological features.

[0032] The three-dimensional morphological features are matched with the dynamic microcirculation feature vectors of the tongue to form a morphological-microcirculation comprehensive feature matrix.

[0033] As a preferred solution of the tongue diagnosis basic information collection method of the present invention, the following steps are included: extracting color and texture features in RGB color images, combining wetness features and morphology-microcirculation comprehensive feature matrix, and forming a multimodal pathological feature vector:

[0034] Extract the tongue color features and the tongue surface texture features through RGB images;

[0035] Extract the mean wetness value and evaporation rate of the tongue surface from the wetness data to obtain the wetness characteristics of the tongue;

[0036] The color features of the tongue, the texture features of the tongue surface, the wetness features of the tongue, the dynamic microcirculation feature vector and the morphology-microcirculation comprehensive feature matrix are spliced ​​to obtain a spliced ​​multimodal feature vector;

[0037] Linearly transform the distribution of each modal feature in the concatenated multimodal feature vector and map it to a unified embedding space;

[0038] The self-attention mechanism is used to model the cross-modal relationship of different modal features in a unified embedding space, forming a unified feature encoding for high-dimensional feature representation;

[0039] Extract the feature template corresponding to each disease from the disease database, use cosine similarity to match the patient characteristics and the corresponding disease feature template, and filter out the feature patterns related to the disease. The expression is:

[0040]

[0041] Among them, S(Z,Y) represents the similarity value between patient feature Z and disease feature template Y, Z represents patient feature, and Y represents disease feature template;

[0042] The disease-related feature patterns are comprehensively screened to generate a multimodal pathology feature vector.

[0043] As a preferred solution of the tongue diagnosis basic information collection method of the present invention, wherein: based on the multimodal pathological feature vector, an intelligent diagnosis model is constructed to perform disease risk assessment, and output of the disease risk score result includes the following steps:

[0044] Orthogonalize the pathological feature vector;

[0045] The orthogonalized pathological feature vector is input into the intelligent diagnosis model to output the diagnosis probability of each disease, which is expressed as:

[0046]

[0047] in, represents the diagnosis probability of the u-th disease, u represents the disease index, x represents the linear combination result based on patient characteristics in the diagnosis model, and W u represents the weight vector associated with the u-th disease, P represents the pathological feature vector after orthogonalization, b u represents the bias term corresponding to the u-th disease;

[0048] According to the diagnosis probability of each disease and the abnormal degree of the indicators in the pathological feature vector, the disease risk score is calculated, and the expression is:

[0049]

[0050] Among them, R u represents the risk score of the u-th disease, c represents the number of features for risk assessment, v represents the feature index, P v represents the vth eigenvalue in the pathological feature vector, μv represents the mean of the vth feature, σ v represents the standard deviation of the vth feature.

[0051] As a preferred solution of the tongue diagnosis basic information collection method of the present invention, the following steps are included: based on the risk score result, a health score is given to each area, and a personalized health recommendation is generated:

[0052] According to the risk score results, the health score of each tongue area is calculated, and the expression is:

[0053]

[0054] Among them, S i represents the health score of the ith tongue region, q represents the number of features involved in the scoring of each region, p represents the feature index involved in the scoring of each region, N represents the number of diseases, X p represents the actual value of the pth feature in the tongue area, μ p represents the mean of the pth feature in the tongue area, σ p represents the standard deviation of the pth feature in the tongue region;

[0055] The patient's health status was classified based on the tongue area's health score, disease risk score, and dynamic microcirculatory characteristics;

[0056] Generate personalized health recommendations based on the classification results.

[0057] In a second aspect, the present invention provides a tongue diagnosis basic information collection system, comprising:

[0058] The data acquisition module collects infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data, and wetness data of the tongue to generate a multimodal data set;

[0059] The microcirculation analysis module performs dynamic microcirculation feature analysis on the infrared thermal imaging data and multispectral reflectance data in the multimodal data set, extracts the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtains the dynamic microcirculation feature vector;

[0060] The geometric feature module extracts the geometric features of the tongue surface from the three-dimensional point cloud data and RGB color images, combines the dynamic microcirculation feature vectors, and generates a morphology-microcirculation comprehensive feature matrix;

[0061] The color and texture module extracts the color and texture features in the RGB color image, combines the wetness features and the morphology-microcirculation comprehensive feature matrix to form a multimodal pathology feature vector;

[0062] Intelligent diagnosis module, based on multimodal pathological feature vectors, builds an intelligent diagnosis model, conducts disease risk assessment, and outputs disease risk score results;

[0063] The health advice module performs health scoring for each area based on the risk scoring results and generates personalized health advice.

[0064] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the method for collecting basic information of tongue diagnosis as described in the first aspect of the present invention is implemented.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the method for collecting basic information of tongue diagnosis as described in the first aspect of the present invention is implemented.

[0066] The beneficial effects of the present invention are as follows: multimodal data acquisition can comprehensively reflect multidimensional information such as temperature fluctuation, blood flow distribution, and wetness of the tongue, providing rich feature input for disease diagnosis; dynamic microcirculation feature analysis is used to extract temperature fluctuation characteristics and blood flow distribution characteristics of the tongue, and a morphological-microcirculation comprehensive feature matrix is ​​generated in combination with three-dimensional morphological features, which can more finely characterize the pathological characteristics of the tongue; self-attention mechanism is used to model cross-modal relationships of multimodal features, and the potential correlation between different modal features is deeply explored, which effectively improves the robustness of feature representation and the intelligence level of the diagnostic model; through intelligent diagnostic models and disease feature template matching, quantitative assessment of disease risks can be achieved, and intuitive risk scores can be output; regionalized health scores and personalized health recommendations are further generated based on risk scores, which are helpful to comprehensively assess the patient's health status from the overall to the local. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0068] Figure 1 This is a flow chart of the method for collecting basic information of tongue diagnosis in Example 1.

[0069] Figure 2 Schematic diagram of the multimodal pathological feature vector in Example 1. DETAILED DESCRIPTION

[0070] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0071] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0072] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0073] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides a method for collecting basic information of tongue diagnosis, comprising the following steps:

[0074] S1. Collect infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data and wetness data of the tongue to generate a multimodal dataset.

[0075] Start the infrared thermal imager to record the temperature distribution on the tongue surface; use the near-infrared multispectral imaging device to collect multispectral reflectance data of the tongue surface; use a high-resolution RGB camera to obtain a color image of the tongue surface; use a structured light 3D scanner to perform a full-area scan of the tongue to obtain the point cloud data of the tongue; start the wetness sensor to record the wetness and evaporation rate of the tongue surface; integrate the collected data to form a multimodal data set.

[0076] S2. Perform dynamic microcirculation feature analysis on the infrared thermal imaging data and multispectral reflectance data in the multimodal data set, extract the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtain the dynamic microcirculation feature vector.

[0077] S2.1. Start the infrared thermal imaging device and extract the temperature data of each area in the infrared thermal imaging data; perform time series analysis on the temperature data of each area to extract the change in the temperature fluctuation amplitude and the temperature difference between areas, and perform discrete Fourier transform on the temperature time series data of each area to extract the frequency component of the temperature fluctuation and obtain the temperature fluctuation characteristics.

[0078] Specifically, the expression of temperature fluctuation amplitude is:

[0079]

[0080] in, represents the fluctuation amplitude of the temperature T of the ith tongue region, which is used to quantify the temperature variation amplitude of the tongue tip, middle and root regions, and reflects the thermal dynamic characteristics of the local region. i (t) represents the average value of the temperature T of the i-th tongue area at time t, i represents the area index of the tongue, t represents the time point, and T represents the temperature of the tongue.

[0081] The expression of the temperature difference between regions is:

[0082]

[0083] Where, ΔT ij represents the average temperature difference between the i-th tongue region and the j-th tongue region, j represents the index of another tongue region, k represents the time frame index, n represents the number of acquired time frames, T i (t k ) represents the i-th tongue area at time point t k The average temperature, T j (t k ) represents the jth tongue region at time point t k The average temperature value.

[0084] The expression for extracting the temperature change frequency is:

[0085]

[0086] Wherein, F(f) represents the Fourier coefficient at frequency f, f represents frequency, and m represents an imaginary unit.

[0087] S2.2. The collected multispectral reflectance data is normalized, and the blood oxygen saturation and blood flow velocity of the tongue are calculated to obtain the blood flow distribution characteristics; the extracted temperature fluctuation characteristics and blood flow distribution characteristics are integrated to generate the dynamic microcirculation feature vector of the tongue.

[0088] Specifically, the expression of blood oxygen saturation is:

[0089]

[0090] Among them, SO 2 Indicates the blood oxygen saturation in the local area of ​​the tongue. Represents the normalized reflectance value at 760nm wavelength, It represents the normalized reflectance value at 850nm wavelength.

[0091] The expression of blood flow velocity is:

[0092]

[0093] Wherein, V represents the blood flow velocity in the local area of ​​the tongue, λ represents the laser wavelength, Δb represents the Doppler frequency shift, a represents the refractive index of the tongue tissue, and θ represents the laser incident angle.

[0094] S3. The geometric features of the tongue surface were extracted from the 3D point cloud data and RGB color images, and combined with the dynamic microcirculation feature vectors to generate a morphological-microcirculatory comprehensive feature matrix.

[0095] S3.1. Use statistical filtering method to remove noise points in three-dimensional point cloud data; use Poisson reconstruction algorithm to reconstruct the three-dimensional surface of the denoised point cloud data to generate a continuous three-dimensional model of the tongue surface; use the collected RGB color image and the UV mapping method based on nearest neighbor interpolation to map the texture information of the tongue surface to the continuous three-dimensional model to obtain a complete three-dimensional model of the tongue.

[0096] S3.2. Calculate the curvature of each point in the complete three-dimensional model and extract the geometric features of the tongue surface.

[0097] Specifically, the geometric features of the tongue surface were extracted using Gaussian curvature and mean curvature;

[0098] The expression for Gaussian curvature is:

[0099] K=r 1 ·r 2 ;

[0100] Where, represents Gaussian curvature, r 1 , r 2 Represents the two principal curvatures of a surface.

[0101] The expression for mean curvature is:

[0102]

[0103] Here, H represents the mean curvature.

[0104] Through curvature analysis, the convex and concave areas of the tongue are detected. Among them:

[0105] K>0: the local surface is convex; K<0: the local surface is concave; K=0: the local surface is flat.

[0106] S3.3. According to the anatomical characteristics of the tongue, the complete three-dimensional model is divided into three regions: the tongue tip, the middle of the tongue and the tongue root, and the extracted geometric features of the tongue surface are mapped to the corresponding regions to obtain complete three-dimensional morphological features; the three-dimensional morphological features are matched with the dynamic microcirculation feature vectors of the tongue to form a morphological-microcirculation comprehensive feature matrix.

[0107] S4. Extract color and texture features from RGB color images, combine wetness features and morphology-microcirculation comprehensive feature matrix to form a multimodal pathology feature vector.

[0108] The color features of the tongue and the texture features of the tongue surface are extracted from the RGB image; the mean wetness and evaporation rate of the tongue surface in the wetness data are extracted to obtain the tongue wetness features; the color features of the tongue, the texture features of the tongue surface, the wetness features of the tongue, the dynamic microcirculation feature vector and the morphology-microcirculation comprehensive feature matrix are spliced ​​to obtain the spliced ​​multimodal feature vector; the distribution of each modal feature in the spliced ​​multimodal feature vector is linearly transformed and mapped to a unified embedding space; the cross-modal relationship of different modal features in the unified embedding space is modeled through the self-attention mechanism to form a unified feature encoding for high-dimensional feature representation; the feature template corresponding to each disease is extracted from the disease database, and the patient features and the corresponding disease feature templates are matched using cosine similarity to screen out the feature patterns related to the disease, which are expressed as:

[0109]

[0110] Among them, S(Z,Y) represents the similarity value between patient feature Z and disease feature template Y, Z represents patient feature, and Y represents disease feature template;

[0111] The disease-related feature patterns are comprehensively screened to generate a multimodal pathology feature vector.

[0112] S5. Based on the multimodal pathological feature vector, an intelligent diagnosis model is constructed to conduct disease risk assessment and output the disease risk score results.

[0113] Orthogonalize the pathological feature vector;

[0114] The orthogonalized pathological feature vector is input into the intelligent diagnosis model to output the diagnosis probability of each disease, which is expressed as:

[0115]

[0116] in, represents the diagnosis probability of the u-th disease, u represents the disease index, x represents the linear combination result based on patient characteristics in the diagnosis model, and W u represents the weight vector associated with the u-th disease, P represents the pathological feature vector after orthogonalization, b u represents the bias term corresponding to the u-th disease;

[0117] According to the diagnosis probability of each disease and the abnormal degree of the indicators in the pathological feature vector, the disease risk score is calculated, and the expression is:

[0118]

[0119] Among them, R u represents the risk score of the u-th disease, c represents the number of features for risk assessment, v represents the feature index, P v represents the vth eigenvalue in the pathological feature vector, μ v represents the mean of the vth feature, σ v represents the standard deviation of the vth feature.

[0120] S6. Based on the risk score results, a health score is given to each area to generate personalized health recommendations.

[0121] S6.1. Calculate the health score of each tongue area based on the risk score result. The expression is:

[0122]

[0123] Among them, S i represents the health score of the ith tongue region, q represents the number of features involved in the scoring of each region, p represents the feature index involved in the scoring of each region, N represents the number of diseases, X p represents the actual value of the pth feature in the tongue area, μ p represents the mean of the pth feature in the tongue area, σ p represents the standard deviation of the pth feature in the tongue region;

[0124] The health status of the patients was classified according to the health score, disease risk score, and dynamic microcirculation characteristics of the tongue area.

[0125] Specifically, the categories are as follows:

[0126] Health: If all S i >0.8 and disease risk R u All are at low risk;

[0127] Sub-health: If 0.5≤S i ≤0.8 or partial disease risk R u At medium risk;

[0128] Exception: If S i <0.5 or at least one disease risk R u At high risk.

[0129] S6.2. Generate personalized health recommendations based on the classification results.

[0130] Specifically, personalized recommendations are generated based on health status, including:

[0131] Healthy people: It is recommended to maintain good living habits and have regular check-ups;

[0132] Sub-healthy people: It is recommended to optimize diet, increase exercise, and pay attention to rest;

[0133] Abnormal population: It is recommended to undergo specialist examinations for high-risk diseases and provide preliminary intervention measures (such as improving blood microcirculation and adjusting body temperature fluctuations).

[0134] This embodiment also provides a tongue diagnosis basic information collection system, including:

[0135] The data acquisition module collects infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data, and wetness data of the tongue to generate a multimodal data set;

[0136] The microcirculation analysis module performs dynamic microcirculation feature analysis on the infrared thermal imaging data and multispectral reflectance data in the multimodal data set, extracts the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtains the dynamic microcirculation feature vector;

[0137] The geometric feature module extracts the geometric features of the tongue surface from the three-dimensional point cloud data and RGB color images, combines the dynamic microcirculation feature vectors, and generates a morphology-microcirculation comprehensive feature matrix;

[0138] The color and texture module extracts the color and texture features in the RGB color image, combines the wetness features and the morphology-microcirculation comprehensive feature matrix to form a multimodal pathology feature vector;

[0139] Intelligent diagnosis module, based on multimodal pathological feature vectors, builds an intelligent diagnosis model, conducts disease risk assessment, and outputs disease risk score results;

[0140] The health advice module performs health scoring for each area based on the risk scoring results and generates personalized health advice.

[0141] This embodiment also provides a computer device, which is suitable for the method of collecting basic information of tongue diagnosis, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the method of collecting basic information of tongue diagnosis proposed in the above embodiment.

[0142] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0143] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for collecting basic information of tongue diagnosis as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination of them, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0144] In summary, the present invention can comprehensively reflect the temperature fluctuation, blood flow distribution, wetness and other multi-dimensional information of the tongue through multimodal data acquisition, providing rich feature input for disease diagnosis; extracting the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue through dynamic microcirculation feature analysis, and generating a morphological-microcirculation comprehensive feature matrix in combination with three-dimensional morphological features, which can more finely characterize the pathological characteristics of the tongue; using the self-attention mechanism to model the cross-modal relationship of multimodal features, deeply explore the potential correlation between different modal features, and effectively improve the robustness of feature representation and the intelligence level of the diagnostic model; through the matching of intelligent diagnostic models and disease feature templates, it can realize the quantitative assessment of disease risks and output intuitive risk scores; based on the risk scores, regionalized health scores and personalized health recommendations are further generated, which is helpful to comprehensively evaluate the health status of patients from the overall to the local.

[0145] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A tongue diagnosis basic information collection method, characterized by: include, Collect infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data, and wetness data of the tongue to generate a multimodal data set; The infrared thermal imaging data and near-infrared multispectral reflectance data in the multimodal data set are used to perform dynamic microcirculation feature analysis, extract the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtain the dynamic microcirculation feature vector; The geometric features of the tongue surface were extracted from the 3D point cloud data and RGB color images, and combined with the dynamic microcirculation feature vectors to generate a morphological-microcirculation comprehensive feature matrix. Extract color and texture features from RGB color images, combine moisture features and morphology-microcirculation comprehensive feature matrix to form a multimodal pathological feature vector; Based on multimodal pathological feature vectors, an intelligent diagnosis model is constructed to conduct disease risk assessment and output disease risk score results; Based on the risk score results, a health score is given to each area to generate personalized health recommendations.

2. The tongue diagnosis basic information collection method according to claim 1, characterized in that: Collecting infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data and wetness data of the tongue, and generating a multimodal data set includes the following steps: Start the infrared thermal imager to record the temperature distribution on the surface of the tongue; The multi-spectral reflectance data of the tongue surface is collected by near-infrared multi-spectral imaging equipment; A high-resolution RGB camera is used to obtain color images of the tongue surface; Use a structured light 3D scanner to perform a full-area scan of the tongue to obtain point cloud data of the tongue; Start the wetness sensor to record the wetness and evaporation rate of the tongue surface; The collected data are integrated to form a multimodal dataset.

3. The tongue diagnosis basic information collection method according to claim 2, characterized in that: The infrared thermal imaging data and multispectral reflectance data in the multimodal data set are subjected to dynamic microcirculation feature analysis, and the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface are extracted to obtain the dynamic microcirculation feature vector, which includes the following steps: Start the infrared thermal imaging device and extract the temperature data of each area in the infrared thermal imaging data; Perform time series analysis on the temperature data of each area to extract the changes in the temperature fluctuation amplitude and the temperature difference between areas, and perform discrete Fourier transform on the temperature time series data of each area to extract the frequency components of the temperature fluctuation and obtain the temperature fluctuation characteristics; The collected multi-spectral reflectance data is normalized, and the blood oxygen saturation and blood flow velocity of the tongue are calculated to obtain the blood flow distribution characteristics; The extracted temperature fluctuation features and blood flow distribution features are integrated to generate the tongue dynamic microcirculation feature vector.

4. The tongue diagnosis basic information collection method according to claim 3, characterized in that: Extracting the geometric features of the tongue surface from the 3D point cloud data and RGB color images, combined with the dynamic microcirculation feature vector, and generating the morphology-microcirculation comprehensive feature matrix includes the following steps: Statistical filtering method is used to remove noise points in 3D point cloud data; The denoised point cloud data is reconstructed using the Poisson reconstruction algorithm to generate a continuous three-dimensional model of the tongue surface. Using the collected RGB color images, the texture information of the tongue surface is mapped onto the continuous 3D model based on the nearest neighbor interpolation UV mapping method to obtain a complete 3D model of the tongue. Calculate the curvature of each point in the complete three-dimensional model of the tongue and extract the geometric features of the tongue surface; According to the anatomical features of the tongue, the complete 3D model is divided into three regions: the tongue tip, the middle of the tongue, and the tongue root. The extracted geometric features of the tongue surface are mapped to the corresponding regions to obtain complete 3D morphological features. The three-dimensional morphological features are matched with the dynamic microcirculation feature vectors of the tongue to form a morphological-microcirculation comprehensive feature matrix.

5. The tongue diagnosis basic information collection method according to claim 4, characterized in that: Extracting color and texture features from RGB color images, combining wetness features and morphology-microcirculation comprehensive feature matrix, and forming a multimodal pathological feature vector includes the following steps: Extract the tongue color features and the tongue surface texture features through RGB images; Extract the mean wetness value and evaporation rate of the tongue surface from the wetness data to obtain the wetness characteristics of the tongue; The color features of the tongue, the texture features of the tongue surface, the wetness features of the tongue, the dynamic microcirculation feature vector and the morphology-microcirculation comprehensive feature matrix are spliced ​​to obtain a spliced ​​multimodal feature vector; Linearly transform the distribution of each modal feature in the concatenated multimodal feature vector and map it to a unified embedding space; The self-attention mechanism is used to model the cross-modal relationship of different modal features in a unified embedding space, forming a unified feature encoding for high-dimensional feature representation; Extract the feature template corresponding to each disease from the disease database, use cosine similarity to match the patient characteristics and the corresponding disease feature template, and filter out the feature patterns related to the disease. The expression is: Among them, S(Z,Y) represents the similarity value between patient feature Z and disease feature template Y, Z represents patient feature, and Y represents disease feature template; The disease-related feature patterns are comprehensively screened to generate a multimodal pathology feature vector.

6. The tongue diagnosis basic information collection method according to claim 5, characterized in that: Based on the multimodal pathological feature vector, an intelligent diagnosis model is constructed to conduct disease risk assessment and output the disease risk score results, including the following steps: Orthogonalize the pathological feature vector; The orthogonalized pathological feature vector is input into the intelligent diagnosis model to output the diagnosis probability of each disease, which is expressed as: in, represents the diagnosis probability of the u-th disease, u represents the disease index, x represents the linear combination result based on patient characteristics in the diagnosis model, and W u represents the weight vector associated with the u-th disease, P represents the pathological feature vector after orthogonalization, b u represents the bias term corresponding to the u-th disease; According to the diagnosis probability of each disease and the abnormal degree of the indicators in the pathological feature vector, the disease risk score is calculated, and the expression is: Among them, R u represents the risk score of the u-th disease, c represents the number of features for risk assessment, v represents the feature index, P v represents the vth eigenvalue in the pathological feature vector, μ v represents the mean of the vth feature, σ v represents the standard deviation of the vth feature.

7. The tongue diagnosis basic information collection method according to claim 6, characterized in that: Based on the risk score results, health scores are assigned to each area, and personalized health recommendations are generated, including the following steps: According to the risk score results, the health score of each tongue area is calculated, and the expression is: Among them, S i represents the health score of the ith tongue region, q represents the number of features involved in the scoring of each region, p represents the feature index involved in the scoring of each region, N represents the number of diseases, X p represents the actual value of the pth feature in the tongue area, μ p represents the mean of the pth feature in the tongue area, σ p represents the standard deviation of the pth feature in the tongue region; The patient's health status was classified based on the tongue area's health score, disease risk score, and dynamic microcirculatory characteristics; Generate personalized health recommendations based on the classification results.

8. A tongue diagnosis basic information collection system, based on the tongue diagnosis basic information collection method according to any one of claims 1 to 7, characterized in that: include, The data acquisition module collects infrared thermal imaging data, near-infrared multispectral reflectance data, RGB color images, three-dimensional point cloud data, and wetness data of the tongue to generate a multimodal data set; The microcirculation analysis module performs dynamic microcirculation feature analysis on the infrared thermal imaging data and multispectral reflectance data in the multimodal data set, extracts the temperature fluctuation characteristics and blood flow distribution characteristics of the tongue surface, and obtains the dynamic microcirculation feature vector; The geometric feature module extracts the geometric features of the tongue surface from the three-dimensional point cloud data and RGB color images, combines the dynamic microcirculation feature vectors, and generates a morphology-microcirculation comprehensive feature matrix; The color and texture module extracts the color and texture features in the RGB color image, combines the wetness features and the morphology-microcirculation comprehensive feature matrix to form a multimodal pathology feature vector; Intelligent diagnosis module, based on multimodal pathological feature vectors, builds an intelligent diagnosis model, conducts disease risk assessment, and outputs disease risk score results; The health advice module performs health scoring for each area based on the risk scoring results and generates personalized health advice.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the tongue diagnosis basic information collection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the tongue diagnosis basic information collection method according to any one of claims 1 to 7 are implemented.

Images