Traditional Chinese medicine cardiovascular health assessment method based on tongue picture image recognition

Abstract

The invention discloses a traditional Chinese medicine cardiovascular health assessment method based on tongue picture image recognition, and the method comprises the following steps: collecting original tongue picture image data, and carrying out the image preprocessing; collecting lingual surface spectrum original data, and executing spectrum preprocessing; extracting features of the standardized tongue picture image data, and generating tongue picture image feature vectors; extracting features of the standardized lingual surface spectral data, and generating lingual surface spectral feature vectors; constructing multi-modal hidden space representation of the tongue picture image feature vector and the tongue surface spectrum feature vector, and executing feature decoupling and feature fusion processing; establishing a mapping relation between the tongue picture-spectrum fusion implicit vector and a cardiovascular health index, and generating a cardiovascular risk prediction vector; and generating a cardiovascular health risk assessment result. According to the invention, tongue image and spectral analysis are fused, cardiovascular health assessment is realized, and the method has the advantages of high accuracy and high stability.

Description

Technical Field

[0001] This invention relates to the field of health assessment and analysis, and in particular to a traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition. Background Technology

[0002] Existing methods of TCM tongue diagnosis mainly rely on manual observation of the color, shape, and texture features of tongue images, combined with doctors' experience to assess cardiovascular health. This method is widely used in clinical practice, but it is highly subjective, with inconsistent judgment standards among different doctors, and it is difficult to establish an objective and quantitative correspondence between changes in tongue appearance and specific cardiovascular health indicators. In recent years, tongue image recognition technology has gradually developed, enabling the automatic extraction of tongue color, shape, and texture features, but it is still limited to macroscopic image information and lacks in-depth analysis of the molecular composition and metabolic state of the tongue surface.

[0003] On the other hand, modern spectroscopic detection technology can analyze the molecular signals on the tongue surface through near-infrared spectroscopy and Raman spectroscopy to obtain related metabolic characteristics such as blood glucose, lipids and hemoglobin, providing objective data support for cardiovascular health. However, existing spectroscopic methods and tongue image recognition technology are usually used separately and have not yet formed a fusion modeling mechanism. They cannot establish an effective connection between the macroscopic features of the tongue image and the microscopic molecular features of the tongue surface, thus limiting the scientific nature and clinical promotion value of TCM tongue diagnosis in cardiovascular health assessment. Summary of the Invention

[0004] One objective of this invention is to propose a traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition. This invention integrates tongue image and spectral analysis to achieve cardiovascular health assessment, and has the advantages of high accuracy and strong stability.

[0005] A method for assessing cardiovascular health in traditional Chinese medicine based on tongue image recognition according to an embodiment of the present invention includes the following steps: Raw tongue image data was collected, and image preprocessing was performed on the raw tongue image data under standard lighting conditions to obtain standardized tongue image data. Raw tongue surface spectral data were collected and spectral preprocessing was performed to generate standardized tongue surface spectral data; Extract tongue color features, tongue texture features, and tongue morphology features from standardized tongue image data to generate tongue image feature vectors; Standardized tongue surface spectral data are input into a tongue surface spectral feature extraction network to extract tongue surface molecular composition features and tongue surface metabolic features, and generate tongue surface spectral feature vectors. A multimodal latent space representation of tongue image feature vector and tongue surface spectral feature vector is constructed. Feature decoupling and feature fusion processing are performed on the multimodal latent space representation to generate tongue image-spectral fusion latent vector. The tongue image-spectral fusion latent vector is input into the cardiovascular health index mapping model to establish the mapping relationship between the tongue image-spectral fusion latent vector and cardiovascular health indicators, and to generate a cardiovascular risk prediction vector. Cardiovascular health risk assessment results are generated based on cardiovascular risk prediction vectors.

[0006] Optionally, the image preprocessing includes illumination correction, color normalization, background segmentation, and noise removal.

[0007] Optionally, the generation of the standardized tongue surface spectral data specifically includes: Raw spectral data of the tongue surface were acquired under immobilized spectral acquisition conditions. The raw spectral data of the tongue surface included near-infrared spectral data and Raman spectral data. Baseline correction was performed on the near-infrared spectral data and Raman spectral data respectively. At each wavelength position, the corresponding original spectral intensity value was subtracted from the spectral baseline fitted value to obtain the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data. Noise filtering was performed on the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data using the moving average method, respectively. Feature peaks were extracted from the filtered near-infrared spectral data and the filtered Raman spectral data respectively. The first and second derivatives of the spectrum at different wavelength positions were calculated. When the first derivative at a certain wavelength position is equal to zero and the second derivative at that position is less than zero, the position is determined to be the position of the spectral feature peak. The position of the spectral feature peak and the corresponding intensity value were recorded to obtain the set of near-infrared spectral feature peaks and the set of Raman spectral feature peaks. The characteristic peak sets of near-infrared and Raman spectra were normalized to obtain normalized characteristic peak sets of near-infrared and Raman spectra, respectively. The normalized set of near-infrared spectral characteristic peaks and the normalized set of Raman spectral characteristic peaks are spliced ​​together to generate standardized tongue-surface spectral data.

[0008] Optionally, the generation of the tongue image feature vector includes: Standardized tongue image data is input into a tongue image feature extraction network to construct a pixel matrix from the standardized tongue image data. The tongue image feature extraction network includes a color feature branch, a texture feature branch, a morphological feature branch, and a fully connected layer. Color feature extraction is performed on the standardized tongue image data through the color feature branch. The standardized tongue image is converted from the red-green-blue color space to the CIELab color space. In the CIELab color space, each pixel is represented by three parameters: brightness component, red-green difference component, and yellow-blue difference component. The average value of the three components of all pixels is calculated to obtain the average brightness value, average red-green difference value, and average yellow-blue difference value. The three are arranged in order to form the color feature vector of the tongue image. In the texture feature branch, the gray-level co-occurrence matrix method is used to extract texture features from standardized tongue image data. Under the conditions of preset direction angle and pixel distance, a gray-level co-occurrence matrix is ​​constructed, and four texture indices, namely energy, contrast, entropy and correlation, are calculated. The four texture indices are then combined into a texture feature vector of the tongue image. In the morphological feature branch, morphological feature extraction is performed on the standardized tongue image data. The boundary of the tongue region is obtained through binarization, the area of ​​the tongue region and the perimeter of the tongue boundary are calculated, the tongue morphological ratio is calculated, and the length of the long axis and the length of the short axis of the tongue are extracted. The aspect ratio of the tongue is calculated, and the area of ​​the tongue region, the perimeter of the tongue boundary, the aspect ratio of the tongue morphology, the length of the long axis of the tongue, the length of the short axis of the tongue, and the aspect ratio of the tongue are combined to form the morphological feature vector of the tongue image. The color feature vector, texture feature vector, and morphological feature vector of the tongue image are concatenated to generate a multidimensional feature vector of the tongue image, which is then normalized. The normalized multidimensional feature vector of the tongue image is input into a fully connected layer for dimensionality reduction and nonlinear mapping, and then normalized to generate the tongue image feature vector.

[0009] Optionally, the generation of the tongue surface spectral feature vector includes: Standardized tongue surface spectral data is input into a tongue surface spectral feature extraction network, which includes a frequency domain transformation layer, a wavelet decomposition layer, an energy statistics layer, a feature embedding layer, and a feature dimensionality reduction layer. The standardized tongue surface spectral data is transformed by a frequency domain transformation layer, converting the tongue surface spectral signal from a time domain representation to a frequency domain representation, thereby generating a frequency domain tongue surface spectral signal. Multi-scale wavelet decomposition is performed on the frequency domain tongue-shaped spectral signal through a wavelet decomposition layer to generate sets of low-frequency and high-frequency components. In the energy statistics layer, the energy of the low-frequency component and the high-frequency component set is statistically analyzed to form the tongue surface energy feature set; In the feature embedding layer of the tongue surface spectral feature extraction network, the set of tongue surface energy features is mapped to the embedding space to form a tongue surface spectral feature embedding vector; In the feature dimensionality reduction layer of the tongue surface spectral feature extraction network, the tongue surface spectral feature embedding vector is reduced in dimensionality using principal component analysis and then normalized to generate the tongue surface spectral feature vector.

[0010] Optionally, the generation of the tongue-spectral fusion latent vector specifically includes: The feature vectors of the tongue image and the spectral feature vectors of the tongue surface are concatenated to form an initial joint feature matrix; The initial joint feature matrix is ​​processed by latent space mapping to generate the tongue image latent subspace and the tongue surface spectral latent subspace; Covariance constraints are applied to the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. Means are extracted from all feature dimensions of the latent subspace of the tongue image to form the latent mean vector of the tongue image. Simultaneously, means are extracted from all feature dimensions of the latent subspace of the tongue surface spectrum to form the latent mean vector of the tongue surface spectrum. The latent mean vector of the tongue image is subtracted from the latent mean vector of the tongue image, and multiplied by the latent mean vector of the tongue surface spectrum, and the results are accumulated one by one. The accumulated result is then divided by the total number of samples to form the covariance value of the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. The covariance value is minimized to decouple the tongue image features and the tongue surface spectral features, resulting in the decoupled latent vector of the tongue image and the latent vector of the tongue surface spectrum. Adaptive weight allocation is performed using the decoupled latent vectors to generate a tongue image-spectral fusion latent vector. Specifically, this involves: performing a correlation assessment between the decoupled tongue image latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue image correlation score; performing a correlation assessment between the decoupled tongue surface spectral latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue surface spectral correlation score; dividing the tongue image correlation score by the sum of the tongue image correlation score and the tongue surface spectral correlation score to form a tongue image fusion weight; subtracting the tongue image fusion weight from the tongue image fusion weight to form a tongue surface spectral fusion weight; and multiplying the decoupled tongue image latent vector by the tongue image fusion weight and adding it to the decoupled tongue surface spectral latent vector multiplied by the tongue surface spectral fusion weight to obtain the tongue image-spectral fusion latent vector. Perform orthogonality and sparseness joint constraint processing on the hidden state vector of tongue image-spectrum fusion; Normalization is performed on the tongue-image-spectrum fusion latent vector after joint constraint optimization to generate the final tongue-image-spectrum fusion latent vector.

[0011] Optionally, the generation of the cardiovascular risk prediction vector specifically includes: The tongue image-spectral fusion latent vector is used as input to the cardiovascular health index mapping model, which includes a linear mapping layer, a pathological mechanism constraint layer, and an output layer. In the linear mapping layer, joint mapping processing is performed on the tongue image-spectral fusion latent vector, and the first-stage feature representation is obtained through linear transformation and nonlinear activation. The first-stage feature representation is multiplied by the pathological mechanism matrix through the pathological mechanism constraint layer to obtain the feature representation after mechanism constraint. In the output layer, the feature representations constrained by the mechanism are linearly combined to obtain the cardiovascular risk prediction vector.

[0012] Optionally, the generation of the cardiovascular health risk assessment results specifically includes: Each component of the cardiovascular risk prediction vector is compared with the corresponding standard health interval threshold. When the predicted value exceeds the corresponding standard interval, the indicator is marked as abnormal. Based on the anomaly labeling results, a set of major cardiovascular risk factors is generated; The overall deviation of the cardiovascular risk prediction vector is calculated to generate a comprehensive deviation. Based on the numerical range of the comprehensive deviation, cardiovascular health status is divided into different levels: normal, low risk, moderate risk, and high risk. Cardiovascular health level labels and the set of major cardiovascular risk factors are used as the results of cardiovascular health risk assessment.

[0013] The beneficial effects of this invention are: This invention combines tongue image recognition with tongue surface spectral analysis to establish a joint modeling mechanism for macroscopic tongue features and microscopic molecular features of the tongue surface, overcoming the limitation of existing TCM tongue diagnosis that relies solely on macroscopic images. By performing illumination correction, color normalization, background segmentation, and noise removal on the original tongue image data, the standardization and stability of the input tongue image are ensured. Simultaneously, by performing baseline correction, noise filtering, and feature peak extraction on the near-infrared and Raman spectral data of the tongue surface, standardized tongue surface spectral data is generated. This allows the molecular information of the tongue surface to participate in health assessment in an objective and quantitative manner. This dual-channel data preprocessing effectively improves the reliability and consistency of the input features, laying a solid foundation for subsequent feature fusion and model training.

[0014] In the feature extraction stage, this invention not only constructs color feature vectors, texture feature vectors, and morphological feature vectors of tongue images, but also extracts spectral feature vectors of the tongue surface through frequency domain transformation, wavelet decomposition, and energy statistics methods, forming a set of molecular energy features that can reflect blood glucose levels, lipid concentrations, and hemoglobin absorption intensity. Through a multimodal latent space mapping mechanism, this invention projects tongue image features and tongue surface spectral features into a latent subspace, and introduces covariance constraints and sparsity constraints to achieve feature decoupling and fusion, generating a tongue image-spectrum fused latent vector. This latent vector has both macroscopic image features and microscopic molecular features, ensuring strong discriminative power and complementarity at the feature level, and providing highly information-based input for the mapping of cardiovascular health indicators.

[0015] In the risk prediction and assessment stage, this invention establishes a correspondence between the tongue image-spectral fusion latent vector and key cardiovascular indicators such as blood pressure, blood lipid levels, blood glucose levels, and heart rate variability through a cardiovascular health indicator mapping model. A pathological mechanism matrix is ​​introduced to constrain the prediction process, ensuring that the prediction results conform to clinically known metabolic and circulatory coupling characteristics. Based on this, a comprehensive deviation is calculated, and health levels are assigned. This not only outputs a cardiovascular health level label but also identifies major risk factors. Compared to existing assessment methods that solely rely on tongue images or spectral signals, this invention provides a more objective cardiovascular health risk assessment conclusion at the outcome level. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition proposed in this invention; Figure 2 This is a schematic diagram of the tongue image-spectral fusion latent vector generation process in a traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition proposed in this invention. Figure 3 This is a flowchart illustrating the generation of cardiovascular risk prediction vectors using a cardiovascular health index mapping model for a traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition, as proposed in this invention. Detailed Implementation

[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0018] refer to Figures 1-3A traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition includes the following steps: Raw tongue image data was collected, and image preprocessing was performed on the raw tongue image data under standard lighting conditions to obtain standardized tongue image data. Raw tongue surface spectral data were collected and spectral preprocessing was performed to generate standardized tongue surface spectral data; Extract tongue color features, tongue texture features, and tongue morphology features from standardized tongue image data to generate tongue image feature vectors; Standardized tongue surface spectral data are input into a tongue surface spectral feature extraction network to extract tongue surface molecular composition features and tongue surface metabolic features, and generate tongue surface spectral feature vectors. A multimodal latent space representation of tongue image feature vector and tongue surface spectral feature vector is constructed. Feature decoupling and feature fusion processing are performed on the multimodal latent space representation to generate tongue image-spectral fusion latent vector. The tongue image-spectral fusion latent vector is input into the cardiovascular health index mapping model to establish the mapping relationship between the tongue image-spectral fusion latent vector and cardiovascular health indicators, and to generate a cardiovascular risk prediction vector. Cardiovascular health risk assessment results are generated based on cardiovascular risk prediction vectors.

[0019] In this embodiment, the image preprocessing includes illumination correction, color normalization, background segmentation, and noise removal.

[0020] In this embodiment, the generation of the standardized tongue surface spectral data specifically includes: Raw spectral data of the tongue surface is acquired under fixed spectral acquisition conditions. The raw spectral data of the tongue surface includes near-infrared spectral data and Raman spectral data. The spectral acquisition conditions consist of excitation source wavelength parameters, acquisition probe angle parameters, and acquisition time parameters. During the acquisition process, the excitation source wavelength parameters, acquisition probe angle parameters, and acquisition time parameters are kept within a preset constant range. Baseline correction was performed on the near-infrared spectral data and Raman spectral data respectively. At each wavelength position, the corresponding original spectral intensity value was subtracted from the spectral baseline fitted value to obtain the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data. The steps for obtaining the spectral baseline fitting value include: constructing a tongue surface spectral intensity sequence from the original tongue surface spectral data in wavelength order, and performing smoothing preprocessing on the tongue surface spectral intensity sequence across the entire spectral range to reduce high-frequency noise interference; then selecting the region in the tongue surface spectral intensity sequence that does not contain the main absorption peak as the baseline reference interval, and performing polynomial function fitting on the baseline reference interval using the least squares fitting method to obtain the tongue surface spectral baseline function; using the output value of the tongue surface spectral baseline function at each wavelength position as the tongue surface spectral baseline fitting value, thereby constructing a complete tongue surface spectral baseline fitting sequence; thus obtaining the spectral baseline fitting value. Noise filtering was performed on the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data using the moving average method. Within a set of continuous sampling points centered at a certain sampling point and determined by the filtering radius parameter, the sum of all spectral intensity values ​​in the set was divided by the number of sampling points in the set to obtain the filtered near-infrared spectral data and the filtered Raman spectral data. Feature peaks were extracted from the filtered near-infrared spectral data and the filtered Raman spectral data respectively. The first and second derivatives of the spectrum at different wavelength positions were calculated. When the first derivative at a certain wavelength position is equal to zero and the second derivative at that position is less than zero, the position is determined to be the position of the spectral feature peak. The position of the spectral feature peak and the corresponding intensity value were recorded to obtain the set of near-infrared spectral feature peaks and the set of Raman spectral feature peaks. Normalize the near-infrared spectral characteristic peak set and the Raman spectral characteristic peak set respectively. Subtract the minimum intensity value in the set from the intensity value of each spectral characteristic peak, and then divide by the difference between the maximum and minimum intensity values ​​in the set to obtain the normalized near-infrared spectral characteristic peak set and the normalized Raman spectral characteristic peak set. The normalized set of near-infrared spectral characteristic peaks and the normalized set of Raman spectral characteristic peaks are spliced ​​together to generate standardized tongue-surface spectral data.

[0021] In this embodiment, the generation of the tongue image feature vector includes: Standardized tongue image data is input into a tongue image feature extraction network to construct a pixel matrix from the standardized tongue image data. The tongue image feature extraction network includes a color feature branch, a texture feature branch, a morphological feature branch, and a fully connected layer. Color feature extraction is performed on standardized tongue image data through a color feature branch. The standardized tongue image is converted from the red-green-blue color space to the CIELab color space. In the CIELab color space, each pixel is represented by three parameters: brightness component, red-green difference component, and yellow-blue difference component. The brightness component is used to reflect the lightness and darkness of the tongue surface. The red-green difference component is used to reflect the relative difference of the tongue color in the red and green directions. The yellow-blue difference component is used to reflect the relative difference of the tongue color in the yellow and blue directions. The average value of the three components of all pixels is calculated to obtain the average brightness value, average red-green difference value, and average yellow-blue difference value. The three are arranged in order to form the color feature vector of the tongue image. In the texture feature branch, the gray-level co-occurrence matrix method is used to extract texture features from standardized tongue image data. Under the conditions of preset direction angle and pixel distance, a gray-level co-occurrence matrix is ​​constructed, and four texture indices, namely energy, contrast, entropy and correlation, are calculated. The four texture indices are then combined into a texture feature vector of the tongue image. In the morphological feature branch, morphological feature extraction is performed on the standardized tongue image data. The boundary of the tongue region is obtained through binarization, the area of ​​the tongue region and the perimeter of the tongue boundary are calculated, the tongue morphological ratio is calculated, and the long axis length and short axis length of the tongue are extracted. The length-to-width ratio of the tongue is calculated. The area of ​​the tongue region, the perimeter of the tongue boundary, the morphological ratio of the tongue, the long axis length of the tongue, the short axis length of the tongue, and the length-to-width ratio of the tongue are combined to form the morphological feature vector of the tongue image. The tongue morphological ratio is the square of the perimeter of the tongue boundary divided by the product of four times pi and the area of ​​the tongue region. The length-to-width ratio of the tongue is the long axis length of the tongue divided by the short axis length of the tongue. The major axis length of the tongue refers to the longest diameter in the circumscribed ellipse of the tongue body determined by the ellipse fitting method during the extraction of morphological features from the tongue image. This longest diameter, which is consistent with the main direction of the tongue body, is used to characterize the maximum geometric scale of the tongue body in the longitudinal direction. The minor axis length of the tongue refers to the longest diameter in the same circumscribed ellipse that is perpendicular to the major axis. This minor axis length is used to characterize the maximum geometric scale of the tongue body in the transverse direction. The major and minor axis lengths of the tongue body together reflect the overall geometric morphological features of the tongue body region and are further used to calculate the length-to-width ratio of the tongue body to describe the longitudinal and transverse proportional relationship of the tongue body. The color feature vector, texture feature vector, and morphological feature vector of the tongue image are concatenated to generate a multidimensional feature vector of the tongue image, which is then normalized. The normalized multidimensional feature vector of the tongue image is input into a fully connected layer for dimensionality reduction and nonlinear mapping, and then normalized to generate the tongue image feature vector.

[0022] In this embodiment, the generation of the tongue surface spectral feature vector includes: Standardized tongue surface spectral data is input into a tongue surface spectral feature extraction network, which includes a frequency domain transformation layer, a wavelet decomposition layer, an energy statistics layer, a feature embedding layer, and a feature dimensionality reduction layer. The standardized tongue surface spectral data is transformed by a frequency domain transformation layer, converting the tongue surface spectral signal from a time domain representation to a frequency domain representation, thereby generating a frequency domain tongue surface spectral signal. Multi-scale wavelet decomposition is performed on the frequency domain tongue surface spectral signal through a wavelet decomposition layer to generate sets of low-frequency and high-frequency components, which are used to extract the characteristic responses of tongue surface molecular groups in different frequency bands. In the energy statistics layer, the energy of the low-frequency component and the high-frequency component set is statistically analyzed to form the tongue surface energy feature set, which is used to characterize the molecular signals related to blood glucose level, lipid concentration and hemoglobin absorption characteristics. The set of tongue surface energy features includes low-frequency component energy features and high-frequency component energy features. The low-frequency component energy features reflect the energy distribution of the tongue surface molecular composition in an overall trend, while the high-frequency component energy features reflect the energy fluctuations of the tongue surface molecular composition in local details. Among the low-frequency component energy features and high-frequency component energy features, some energy features are closely related to the cardiovascular metabolic state. Specifically, the blood glucose-related energy features correspond to the energy response of glucose molecules in a specific wavelength range, the lipid-related energy features correspond to the energy response of fatty acid molecules and cholesterol molecules in a specific wavelength range, and the hemoglobin-related energy features correspond to the energy absorption intensity of hemoglobin molecules in a specific wavelength range. In the feature embedding layer of the tongue surface spectral feature extraction network, the set of tongue surface energy features is mapped to the embedding space to form a tongue surface spectral feature embedding vector. The embedding mapping process compresses redundant information and highlights tongue surface molecular signals related to cardiovascular metabolic abnormalities. The feature embedding layer takes the tongue surface energy feature set as input. The tongue surface energy feature set consists of energy statistics of each frequency band component, which includes molecular information related to blood glucose level, lipid concentration and hemoglobin absorption characteristics. For each energy statistics value in the tongue surface energy feature set, a linear mapping process is performed to project it into a high-dimensional embedding space, so that the low-dimensional statistical information in the tongue surface energy feature set is converted into a high-dimensional representation. Then, in the high-dimensional embedding space, the high-dimensional representations of different frequency bands are fused by weighted combination to highlight the energy components that are significant in cardiovascular metabolic abnormalities and suppress interference components that are unrelated to cardiovascular health status. The fused high-dimensional representation is compressed into a tongue surface spectral feature embedding vector. In the feature dimensionality reduction layer of the tongue surface spectral feature extraction network, the tongue surface spectral feature embedding vector is reduced in dimensionality using principal component analysis and then normalized to generate the tongue surface spectral feature vector.

[0023] In this embodiment, the generation of the tongue image-spectral fusion latent vector specifically includes: The feature vectors of the tongue image and the spectral feature vectors of the tongue surface are concatenated to form an initial joint feature matrix; The initial joint feature matrix is ​​processed by latent space mapping to generate the tongue image latent subspace and the tongue surface spectral latent subspace; The generation of the tongue image latent subspace and the tongue surface spectral latent subspace specifically includes: inputting the initial joint feature matrix into a multi-layer fully connected mapping structure, sequentially performing linear transformation and nonlinear activation processing, so that the tongue image feature components and tongue surface spectral feature components in the initial joint feature matrix are gradually projected into the latent representation space; then, the feature components are partitioned and mapped in the latent representation space, so that the tongue image feature components are separately partitioned to form the tongue image latent subspace, and the tongue surface spectral feature components are separately partitioned to form the tongue surface spectral latent subspace; then, normalization and regularization processing are performed on the feature components in their respective subspaces to ensure that the tongue image latent subspace and the tongue surface spectral latent subspace maintain consistency in numerical range and distribution characteristics; finally, the tongue image latent subspace and the tongue surface spectral latent subspace are output as inputs for feature decoupling and fusion processing. The tongue image latent subspace is composed of a set of latent vectors generated by the tongue image feature vectors during the latent mapping process, and the tongue surface spectral latent subspace is composed of a set of latent vectors generated by the tongue surface spectral feature vectors during the latent mapping process. Covariance constraints are applied to the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. Means are extracted from all feature dimensions of the latent subspace of the tongue image to form the latent mean vector of the tongue image. Simultaneously, means are extracted from all feature dimensions of the latent subspace of the tongue surface spectrum to form the latent mean vector of the tongue surface spectrum. The latent mean vector of the tongue image is subtracted from the latent mean vector of the tongue image, and multiplied by the latent mean vector of the tongue surface spectrum, and the results are accumulated one by one. The accumulated result is then divided by the total number of samples to form the covariance value of the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. The covariance value is minimized to decouple the tongue image features and the tongue surface spectral features, resulting in the decoupled latent vector of the tongue image and the latent vector of the tongue surface spectrum. Adaptive weight allocation is performed using the decoupled latent vectors to generate a tongue image-spectral fusion latent vector. Specifically, this involves: performing a correlation assessment between the decoupled tongue image latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue image correlation score; performing a correlation assessment between the decoupled tongue surface spectral latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue surface spectral correlation score; dividing the tongue image correlation score by the sum of the tongue image correlation score and the tongue surface spectral correlation score to form a tongue image fusion weight; subtracting the tongue image fusion weight from the tongue image fusion weight to form a tongue surface spectral fusion weight; and multiplying the decoupled tongue image latent vector by the tongue image fusion weight and adding it to the decoupled tongue surface spectral latent vector multiplied by the tongue surface spectral fusion weight to obtain the tongue image-spectral fusion latent vector. The cardiovascular health indicator vector consists of normalized systolic blood pressure, diastolic blood pressure, heart rate, blood glucose concentration, total cholesterol concentration, low-density lipoprotein concentration, high-density lipoprotein concentration, triglyceride concentration, blood oxygen saturation, hemoglobin concentration, blood viscosity, and hematocrit. The tongue-image-spectral fusion latent vector is subjected to joint orthogonality and sparsity constraints. The orthogonality constraint involves multiplying the tongue-image-spectral fusion latent vector with its transpose and subtracting the identity matrix of the same dimension to form a difference matrix. The norm of the difference matrix is ​​then minimized to achieve linear independence of the tongue-image-spectral fusion latent vector in the latent space. The sparsity constraint involves performing absolute value operations on each dimension of the tongue-image-spectral fusion latent vector and summing the results to form sparse values. The sparsity values ​​are then minimized to achieve a sparse representation of the tongue-image-spectral fusion latent vector. The minimization process is carried out during the modeling and training phases, in which the tongue image-spectrum fusion latent vector changes dynamically with parameter updates, and the constraint target is gradually reduced through iterative optimization each time. Normalization is performed on the tongue-image-spectrum fusion latent vector after joint constraint optimization to generate the final tongue-image-spectrum fusion latent vector.

[0024] In this embodiment, the generation of the cardiovascular risk prediction vector specifically includes: The tongue image-spectral fusion latent vector is used as input to the cardiovascular health index mapping model, which includes a linear mapping layer, a pathological mechanism constraint layer, and an output layer. In the linear mapping layer, joint mapping processing is performed on the tongue image-spectral fusion latent vector. The first-stage feature representation is obtained through linear transformation and nonlinear activation. The first-stage feature representation maintains consistency with the tongue image features and tongue surface spectral features in the vector dimension. The first-stage feature representation is multiplied by the pathological mechanism matrix through the pathological mechanism constraint layer to obtain the mechanism-constrained feature representation. The mechanism-constrained feature representation is adjusted to conform to the clinical coupling characteristics between blood glucose level, blood lipid level, blood pressure level and heart rate variability. The generation of the pathological mechanism matrix specifically includes: based on the tongue surface spectral features corresponding to the tongue image-spectral fusion latent vector, selecting key spectral peaks that can reflect the cardiovascular metabolic state, including molecular absorption peaks related to blood glucose levels, blood lipid levels, blood pressure levels, and heart rate variability; collecting a large amount of cardiovascular clinical test data to construct a paired sample set containing the spectral peak intensity and corresponding cardiovascular health indicators; using statistical analysis methods to perform correlation tests and significance analyses on the paired sample set to extract the pathological association between spectral features and various cardiovascular health indicators; representing the association according to the correspondence between the spectral feature dimension and the cardiovascular health indicator dimension in a matrix manner to obtain the pathological mechanism matrix, where each element of the matrix represents the quantitative association strength between a specific tongue surface spectral feature and a specific cardiovascular health indicator; and finally, normalizing the matrix elements to keep the pathological mechanism matrix stable in the numerical range, thereby forming a constraint structure that can be directly embedded into the cardiovascular health indicator mapping model. The cardiovascular health indicators refer to a set of multidimensional parameters that can objectively reflect the functional state of an individual's cardiovascular system, including blood pressure level, blood lipid level, blood glucose level, and heart rate variability. In the output layer, the feature representations constrained by the mechanism are linearly combined to obtain the cardiovascular risk prediction vector. The cardiovascular risk prediction vector is a multi-dimensional numerical vector, and each component corresponds to the blood pressure level, blood lipid level, blood glucose level and heart rate variability, respectively. The linear combination refers to the process in the output layer where the components of the feature representation after mechanistic constraints are weighted and summed according to preset weight coefficients, and the corresponding bias term is superimposed on the weighted sum to generate the components of the cardiovascular risk prediction vector. Specifically, for each component in the feature representation after mechanistic constraints, it is multiplied by the corresponding weight coefficient in the weight matrix of the output layer, and all products are added together and the bias value of the output layer is added to obtain one component of the cardiovascular risk prediction vector. In this way, the output layer can generate multiple components, each of which corresponds one-to-one with the blood pressure level, blood lipid level, blood glucose level and heart rate variability index.

[0025] In this embodiment, the generation of the cardiovascular health risk assessment results specifically includes: Each component of the cardiovascular risk prediction vector is compared with the corresponding standard health interval threshold. The standard interval for blood pressure level is determined by the lower limit and upper limit of blood pressure, the standard interval for blood lipid level is determined by the lower limit and upper limit of blood lipid, the standard interval for blood glucose level is determined by the lower limit and upper limit of blood glucose, and the standard interval for heart rate variability is determined by the lower limit and upper limit of heart rate variability. When the predicted value exceeds the corresponding standard interval, the indicator is marked as abnormal. Based on the anomaly labeling results, a set of major cardiovascular risk factors is generated, which consists of all health indicators that are labeled as abnormal. The overall deviation of the cardiovascular risk prediction vector is calculated to generate a comprehensive deviation. The overall deviation is the sum of the absolute values ​​of the differences between the predicted values ​​of each health indicator and their normal reference values, multiplied by the corresponding weight coefficients. The weight coefficients are used to distinguish the degree of influence of different health indicators in the comprehensive risk. Based on the numerical range of the comprehensive deviation, cardiovascular health status is divided into different levels: normal, low risk, moderate risk, and high risk. Cardiovascular health level labels and the set of major cardiovascular risk factors are used as the results of cardiovascular health risk assessment.

[0026] Example 1

[0027] To verify the feasibility of this invention in practical applications, it was applied to the clinical auxiliary diagnosis process of the cardiology department of a tertiary hospital. This hospital receives a large number of high-risk cardiovascular patients in its daily operations. Under traditional treatment models, these patients often rely on various biochemical tests and medical imaging examinations. While traditional Chinese medicine tongue diagnosis can provide some reference in initial screening, its reliance on physician subjective judgment leads to significant variability, a lack of quantitative standards, and difficulty in traceability and verification. This results in tongue diagnosis results not being directly correlated with modern clinical data. This invention addresses this problem by combining tongue image recognition and tongue surface spectral analysis to establish a cross-modal joint modeling mechanism, providing hospitals with a quantifiable, interpretable, and traceable approach to cardiovascular health assessment.

[0028] In practical applications, the subject first acquires tongue images under standardized lighting conditions. Through illumination correction, color normalization, background segmentation, and noise removal, image quality is ensured to be stable, reducing interference caused by ambient light and individual differences. At the same time, near-infrared spectroscopy and Raman spectroscopy are used to acquire images of the tongue surface. After baseline correction, noise filtering, and characteristic peak extraction, standardized spectral data is generated, thereby obtaining objective signals reflecting the molecular composition and metabolic state of the tongue surface. This process ensures that the input data is reliable at both the macroscopic visual and microscopic molecular levels.

[0029] During the data processing stage, tongue images are processed by a feature extraction network to extract color, texture, and morphological features, such as the brightness of the tongue color, the uniformity of the surface texture, and the geometric proportions of the tongue. These features reflect the external manifestations of blood circulation and overall metabolism. Meanwhile, the tongue surface spectral signal undergoes frequency domain transformation and wavelet decomposition to further extract low-frequency and high-frequency energy features, which contain key information related to blood glucose, lipids, and hemoglobin absorption. Through feature embedding and dimensionality reduction layers, a tongue surface spectral feature vector is finally formed. This vector, along with the tongue image feature vector, enters the multimodal latent space mapping module. This module combines macroscopic tongue image features with microscopic molecular features through feature decoupling, correlation evaluation, and weighted fusion to generate a tongue image-spectral fusion latent vector. This provides a highly concentrated and medically relevant feature representation for subsequent cardiovascular indicator prediction.

[0030] During the model inference process, the tongue image-spectral fusion latent vector is input into the cardiovascular health index mapping model. This model has a linear mapping layer, a pathological mechanism constraint layer, and an output layer. The pathological mechanism constraint layer plays a key role in this process. It establishes a quantitative correspondence between spectral molecular features and clinical indicators such as blood pressure, blood lipids, blood sugar, and heart rate variability, so that the output result is not only a risk prediction value, but also clearly correlated with specific molecular signals. The output layer generates a cardiovascular risk prediction vector through linear combination. Each component corresponds to a different cardiovascular health indicator, ensuring that the prediction result can be directly interpreted by comparing with medical test indicators.

[0031] During the assessment results output phase, the system compares the predicted vector with the health threshold range. If a certain indicator exceeds the normal range, it is marked as a risk factor in the results, and the final assessment report provides a cardiovascular health level label and a set of risk factors.

[0032] To verify the performance of the present invention in practice, it was compared with traditional methods, and the results are shown in Table 1.

[0033] Table 1. Performance Comparison of Tongue Image-Spectroscopic Methods and Traditional Methods in Cardiovascular Health Assessment Method category Accuracy (%) Sensitivity (%) Specificity (%) Data stability score (1-5) Traditional tongue image recognition method 78.6 75.2 80.3 2.3 Single tongue surface spectrum detection method 83.4 81.7 84.9 3.5 The method of the present invention 91.8 90.5 92.6 4.7 As can be seen from Table 1, traditional tongue image recognition methods have insufficient accuracy in cardiovascular health assessment, with an overall accuracy of less than 80%. Both sensitivity and specificity are affected by the doctor's experience and ambient lighting conditions, resulting in large fluctuations in the results. In addition, since this method is based only on macroscopic color and morphological features, it lacks persuasiveness in clinical interpretation.

[0034] The single-tongue surface spectral detection method is an improvement over the tongue image recognition method, with an accuracy of over 83%, a balanced sensitivity and specificity, and provides metabolic information at the molecular level, thus improving stability to some extent. However, this method lacks a correspondence with macroscopic tongue images and is difficult to corroborate with the context of tongue diagnosis in traditional Chinese medicine. Therefore, its reliability in comprehensive clinical applications is still insufficient.

[0035] The tongue image-spectral fusion method of this invention outperforms the comparative methods in three core indicators: accuracy, sensitivity, and specificity. The accuracy is close to 92%, and the sensitivity and specificity are both above 90%. The fundamental reason for this improvement is that the method of this invention constructs a multimodal latent space to deeply fuse the color, texture, and morphological features of the tongue image with the spectral molecular composition features, achieving complementarity between macroscopic appearance information and microscopic molecular information. At the same time, the introduction of a pathological mechanism matrix as a constraint in model training makes the prediction results not only more consistent with clinical patterns, but also able to output a set of risk factors. In terms of data stability, since the image and spectral data are standardized in the acquisition and preprocessing stages, the fusion method effectively suppresses noise interference from a single channel, and the overall stability score is close to full.

[0036] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A traditional Chinese medicine cardiovascular health assessment method based on tongue image recognition, characterized in that, Includes the following steps: Raw tongue image data was collected, and image preprocessing was performed on the raw tongue image data under standard lighting conditions to obtain standardized tongue image data. Raw tongue surface spectral data were collected and spectral preprocessing was performed to generate standardized tongue surface spectral data; Extract tongue color features, tongue texture features, and tongue morphology features from standardized tongue image data to generate tongue image feature vectors; Standardized tongue surface spectral data are input into a tongue surface spectral feature extraction network to extract tongue surface molecular composition features and tongue surface metabolic features, and generate tongue surface spectral feature vectors. A multimodal latent space representation of tongue image feature vector and tongue surface spectral feature vector is constructed. Feature decoupling and feature fusion processing are performed on the multimodal latent space representation to generate tongue image-spectral fusion latent vector. The tongue image-spectral fusion latent vector is input into the cardiovascular health index mapping model to establish the mapping relationship between the tongue image-spectral fusion latent vector and cardiovascular health indicators, and to generate a cardiovascular risk prediction vector. Cardiovascular health risk assessment results are generated based on cardiovascular risk prediction vectors.

2. The method for assessing cardiovascular health in Traditional Chinese Medicine based on tongue image recognition according to claim 1, characterized in that, The image preprocessing includes illumination correction, color normalization, background segmentation, and noise removal.

3. The method for assessing cardiovascular health in Traditional Chinese Medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the standardized tongue surface spectral data specifically includes: Raw spectral data of the tongue surface were acquired under immobilized spectral acquisition conditions. The raw spectral data of the tongue surface included near-infrared spectral data and Raman spectral data. Baseline correction was performed on the near-infrared spectral data and Raman spectral data respectively. At each wavelength position, the corresponding original spectral intensity value was subtracted from the spectral baseline fitted value to obtain the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data. Noise filtering was performed on the baseline-corrected near-infrared spectral data and the baseline-corrected Raman spectral data using the moving average method, respectively. Feature peaks were extracted from the filtered near-infrared spectral data and the filtered Raman spectral data respectively. The first and second derivatives of the spectrum at different wavelength positions were calculated. When the first derivative at a certain wavelength position is equal to zero and the second derivative at that position is less than zero, the position is determined to be the position of the spectral feature peak. The position of the spectral feature peak and the corresponding intensity value are recorded to obtain the set of near-infrared spectral feature peaks and the set of Raman spectral feature peaks. The characteristic peak sets of near-infrared and Raman spectra were normalized to obtain normalized characteristic peak sets of near-infrared and Raman spectra, respectively. The normalized set of near-infrared spectral characteristic peaks and the normalized set of Raman spectral characteristic peaks are spliced ​​together to generate standardized tongue-surface spectral data.

4. The method for assessing cardiovascular health in traditional Chinese medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the tongue image feature vector includes: Standardized tongue image data is input into a tongue image feature extraction network to construct a pixel matrix from the standardized tongue image data. The tongue image feature extraction network includes a color feature branch, a texture feature branch, a morphological feature branch, and a fully connected layer. Color feature extraction is performed on the standardized tongue image data through the color feature branch. The standardized tongue image is converted from the red-green-blue color space to the CIELab color space. In the CIELab color space, each pixel is represented by three parameters: brightness component, red-green difference component, and yellow-blue difference component. The average value of the three components of all pixels is calculated to obtain the average brightness value, average red-green difference value, and average yellow-blue difference value. The three are arranged in order to form the color feature vector of the tongue image. In the texture feature branch, the gray-level co-occurrence matrix method is used to extract texture features from standardized tongue image data. Under the conditions of preset direction angle and pixel distance, a gray-level co-occurrence matrix is ​​constructed, and four texture indices, namely energy, contrast, entropy and correlation, are calculated. The four texture indices are then combined into a texture feature vector of the tongue image. In the morphological feature branch, morphological feature extraction is performed on the standardized tongue image data. The boundary of the tongue region is obtained through binarization, the area of ​​the tongue region and the perimeter of the tongue boundary are calculated, the tongue morphological ratio is calculated, and the length of the long axis and the length of the short axis of the tongue are extracted. The aspect ratio of the tongue is calculated, and the area of ​​the tongue region, the perimeter of the tongue boundary, the aspect ratio of the tongue morphology, the length of the long axis of the tongue, the length of the short axis of the tongue, and the aspect ratio of the tongue are combined to form the morphological feature vector of the tongue image. The color feature vector, texture feature vector, and morphological feature vector of the tongue image are concatenated to generate a multidimensional feature vector of the tongue image, which is then normalized. The normalized multidimensional feature vector of the tongue image is input into a fully connected layer for dimensionality reduction and nonlinear mapping, and then normalized to generate the tongue image feature vector.

5. The method for assessing cardiovascular health in Traditional Chinese Medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the tongue surface spectral feature vector includes: Standardized tongue surface spectral data is input into a tongue surface spectral feature extraction network, which includes a frequency domain transformation layer, a wavelet decomposition layer, an energy statistics layer, a feature embedding layer, and a feature dimensionality reduction layer. The standardized tongue surface spectral data is transformed by a frequency domain transformation layer, converting the tongue surface spectral signal from a time domain representation to a frequency domain representation, thereby generating a frequency domain tongue surface spectral signal. Multi-scale wavelet decomposition is performed on the frequency domain tongue-shaped spectral signal through a wavelet decomposition layer to generate sets of low-frequency and high-frequency components. In the energy statistics layer, the energy of the low-frequency component and the high-frequency component set is statistically analyzed to form the tongue surface energy feature set; In the feature embedding layer of the tongue surface spectral feature extraction network, the set of tongue surface energy features is mapped to the embedding space to form a tongue surface spectral feature embedding vector; In the feature dimensionality reduction layer of the tongue surface spectral feature extraction network, the tongue surface spectral feature embedding vector is reduced in dimensionality using principal component analysis and then normalized to generate the tongue surface spectral feature vector.

6. The method for assessing cardiovascular health in traditional Chinese medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the tongue-spectral fusion latent vector specifically includes: The feature vectors of the tongue image and the spectral feature vectors of the tongue surface are concatenated to form an initial joint feature matrix; The initial joint feature matrix is ​​processed by latent space mapping to generate the tongue image latent subspace and the tongue surface spectral latent subspace; Covariance constraints are applied to the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. Means are extracted from all feature dimensions of the latent subspace of the tongue image to form the latent mean vector of the tongue image. Simultaneously, means are extracted from all feature dimensions of the latent subspace of the tongue surface spectrum to form the latent mean vector of the tongue surface spectrum. The latent mean vector of the tongue image is subtracted from the latent mean vector of the tongue image, and multiplied by the latent mean vector of the tongue surface spectrum, and the results are accumulated one by one. The accumulated result is then divided by the total number of samples to form the covariance value of the latent subspace of the tongue image and the latent subspace of the tongue surface spectrum. The covariance value is minimized to decouple the tongue image features and the tongue surface spectral features, resulting in the decoupled latent vector of the tongue image and the latent vector of the tongue surface spectrum. Adaptive weight allocation is performed using the decoupled latent vectors to generate a tongue image-spectral fusion latent vector. Specifically, this involves: performing a correlation assessment between the decoupled tongue image latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue image correlation score; performing a correlation assessment between the decoupled tongue surface spectral latent vector and the cardiovascular health indicator vector using the Pearson correlation coefficient to form a tongue surface spectral correlation score; dividing the tongue image correlation score by the sum of the tongue image correlation score and the tongue surface spectral correlation score to form a tongue image fusion weight; subtracting the tongue image fusion weight from the tongue image fusion weight to form a tongue surface spectral fusion weight; and multiplying the decoupled tongue image latent vector by the tongue image fusion weight and adding it to the decoupled tongue surface spectral latent vector multiplied by the tongue surface spectral fusion weight to obtain the tongue image-spectral fusion latent vector. Perform orthogonality and sparseness joint constraint processing on the hidden state vector of tongue image-spectrum fusion; Normalization is performed on the tongue-image-spectrum fusion latent vector after joint constraint optimization to generate the final tongue-image-spectrum fusion latent vector.

7. The method for assessing cardiovascular health in traditional Chinese medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the cardiovascular risk prediction vector specifically includes: The tongue image-spectral fusion latent vector is used as input to the cardiovascular health index mapping model, which includes a linear mapping layer, a pathological mechanism constraint layer, and an output layer. In the linear mapping layer, joint mapping processing is performed on the tongue image-spectral fusion latent vector, and the first-stage feature representation is obtained through linear transformation and nonlinear activation. The first-stage feature representation is multiplied by the pathological mechanism matrix through the pathological mechanism constraint layer to obtain the feature representation after mechanism constraint. In the output layer, the feature representations constrained by the mechanism are linearly combined to obtain the cardiovascular risk prediction vector.

8. A method for assessing cardiovascular health in traditional Chinese medicine based on tongue image recognition according to claim 1, characterized in that, The generation of the cardiovascular health risk assessment results specifically includes: Each component of the cardiovascular risk prediction vector is compared with the corresponding standard health interval threshold. When the predicted value exceeds the corresponding standard interval, the indicator is marked as abnormal. Based on the anomaly labeling results, a set of major cardiovascular risk factors is generated; The overall deviation of the cardiovascular risk prediction vector is calculated to generate a comprehensive deviation. Based on the numerical range of the comprehensive deviation, cardiovascular health status is divided into different levels: normal, low risk, moderate risk, and high risk. Cardiovascular health level labels and the set of major cardiovascular risk factors are used as the results of cardiovascular health risk assessment.

Images