AI health analysis method and system based on tongue appearance and pulse wave and related equipment

Abstract

This application relates to the field of health information processing technology, providing an AI health analysis method, system, and related equipment based on tongue image and pulse wave. It collects tongue images and heart pulse data from users through intelligent sensors; uses a convolutional neural network model in deep learning to extract features from the tongue image and output a tongue image feature vector; performs time-frequency analysis on the pulse data to generate a pulse feature vector; performs multimodal data fusion analysis on the tongue image feature vector and pulse feature vector based on deep learning to generate a feature vector, and generates a health status assessment result based on the feature vector; and generates health suggestions based on the assessment result and feeds them back to the user's terminal. This application combines artificial intelligence technology to realize a complete process from data collection to health suggestion feedback. It fully leverages the advantages of traditional Chinese medicine health analysis while using modern technology to improve the efficiency and accuracy of health analysis, providing users with personalized and precise health management solutions.

Description

Technical Field

[0001] This application relates to the field of health information processing technology, and more specifically, to an AI health analysis method, system, and related equipment based on tongue image and pulse wave. Background Technology

[0002] With the rapid development of society and the economy, people are paying increasing attention to health and pursuing more personalized health management solutions. In this context, traditional health analysis methods are gradually revealing their limitations in terms of efficiency, accuracy, and personalization. Traditional Chinese medicine (TCM), a treasure of Chinese medicine, has accumulated rich experience through long-term clinical practice, with tongue and pulse observation being particularly distinctive health analysis methods. However, traditional TCM analysis relies heavily on the doctor's personal experience and subjective judgment, which has certain limitations, resulting in an inability to efficiently and accurately capture human health information. Summary of the Invention

[0003] This application provides an AI health analysis method, system, and related equipment based on tongue image and pulse wave, which can at least partially solve the problem of the inability to efficiently and accurately capture human health information.

[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.

[0005] According to one aspect of this application, an AI health analysis method based on tongue image and pulse wave is provided, comprising: acquiring a user's tongue image and heart pulse data through a smart sensor; extracting features from the tongue image using a convolutional neural network model in deep learning to output a tongue image feature vector; performing time-frequency analysis on the pulse data to generate a pulse feature vector; performing multimodal data fusion analysis on the tongue image feature vector and the pulse feature vector based on deep learning to generate fused features, and generating a health status assessment result based on the fused features; generating health suggestions based on the health status assessment result, and feeding the health suggestions back to the user terminal.

[0006] In this application, based on the aforementioned scheme, the step of acquiring a user's tongue image and heart pulse data through a smart sensor includes: acquiring a user's tongue image through a camera device and preprocessing the tongue image according to median filtering; acquiring heart pulse data through a piezoelectric sensor and filtering out high-frequency noise in the pulse data through a low-pass filter.

[0007] In this application, based on the aforementioned scheme, the step of using a convolutional neural network model in deep learning to extract features from the tongue image and output a tongue image feature vector includes: obtaining a convolutional neural network model based on deep learning training; inputting the tongue image into the convolutional layer and pooling layer of the convolutional neural network for feature extraction and outputting a tongue image feature map; and after multiple rounds of convolution and pooling processing, flattening the feature map into a tongue image feature vector.

[0008] In this application, based on the aforementioned scheme, the step of performing time-frequency analysis on the pulse data to generate a pulse feature vector includes: determining the signal features corresponding to each set of scale factors and translation factors by adjusting the scale factors and translation factors based on a preset scale transformation function; and performing multi-scale time-frequency analysis on the pulse data by transforming the signal features to generate a pulse feature vector.

[0009] In this application, based on the aforementioned scheme, the step of performing multimodal data fusion analysis on the tongue image feature vector and pulse feature vector using deep learning to generate fused features, and generating a health status assessment result based on the fused features, includes: inputting the tongue image feature vector into the tongue image feature input layer of a pre-trained multimodal fusion network to output a first feature; inputting the pulse feature vector into the pulse wave feature input layer of the multimodal fusion network to output a second feature; performing attention-based fusion processing on the first feature and the second feature in the fusion layer of the multimodal fusion network to generate fused features; and transmitting the fused features to the fully connected layer of the multimodal fusion network, mining the mapping relationship between the fused features through nonlinear transformation to generate a health status assessment result.

[0010] In this application, based on the aforementioned scheme, the step of performing attention-based fusion processing on the first feature and the second feature in the fusion layer of the multimodal fusion network to generate fused features includes: constructing a tongue-pulse syndrome association map containing tongue image feature nodes, pulse feature nodes, and traditional Chinese medicine syndrome nodes in the fusion layer of the multimodal fusion network; constructing a matching function through the tongue-pulse correspondence in traditional Chinese medicine theory to calculate the association strength between each node in the tongue-pulse syndrome association map; and performing attention-based fusion processing on the first feature and the second feature based on a multi-head graph attention mechanism generated by the association strength to generate fused features.

[0011] In this application, based on the aforementioned scheme, the step of generating health suggestions based on the assessment results of the health status and feeding the health suggestions back to the user terminal includes: matching the health status assessment results in a preset health rule base to generate health suggestions; and feeding the health suggestions back to the user terminal.

[0012] In this application, based on the aforementioned scheme, after generating health recommendations based on the health status assessment results and feeding the health recommendations back to the user terminal, the method further includes: acquiring tongue images, pulse data, and lifestyle data of the user during the execution of the health recommendations; and adjusting the health recommendations based on the tongue images, pulse data, and lifestyle data.

[0013] According to one aspect of this application, an AI health analysis system based on tongue image and pulse wave is provided, comprising:

[0014] The acquisition module is used to collect images of the user's tongue and pulse data of the heart through intelligent sensors;

[0015] The extraction module is used to extract features from the tongue image using a convolutional neural network model in deep learning and output a tongue image feature vector.

[0016] The analysis module is used to perform time-frequency analysis on the pulse data and generate a pulse feature vector;

[0017] The evaluation module is used to perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector based on deep learning, generate fused features, and generate an evaluation result of health status based on the fused features.

[0018] The feedback module is used to generate health recommendations based on the assessment results of the health status and to send the health recommendations back to the user terminal.

[0019] According to one aspect of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the AI ​​health analysis method based on tongue image and pulse wave as described in the above embodiments.

[0020] According to one aspect of this application, an electronic device is provided, comprising: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the AI ​​health analysis method based on tongue image and pulse wave as described in the above embodiments.

[0021] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the AI ​​health analysis method based on tongue image and pulse wave provided in the various optional implementations described above.

[0022] The main differences and technical effects of the technical solution in this application compared with the prior art are as follows:

[0023] On the one hand, traditional Chinese medicine relies on a single modality, making it susceptible to individual differences and environmental interference. While existing AI systems can improve the efficiency of single-modality analysis, they lack cross-validation of multi-dimensional information. For example, a fissured tongue may be caused by yin deficiency or blood deficiency, which is difficult to distinguish based on tongue appearance alone; it requires comprehensive judgment in conjunction with pulse diagnosis (e.g., a thin pulse indicates blood deficiency, while a rapid pulse indicates yin deficiency). This solution uses a neural network model to simultaneously process tongue feature vectors and pulse feature vectors, dynamically allocating weights using an attention mechanism to focus on key feature associations. For example, when the tongue appearance shows a reddish-brown tongue body and a dry, yellow coating, the model can automatically associate it with a "rapid pulse" (increased pulse rate), combining this with the theory of *Shang Han Lun* (Treatise on Cold Damage) to generate a diagnostic conclusion of "excessive heat damaging body fluids," and recommend heat-clearing herbs such as Coptis chinensis. Based on the results of AI-physician collaborative analysis in complex cases, this multimodal fusion can significantly improve the accuracy of health analysis, significantly exceeding the results of single-modality or single-physician analysis.

[0024] Secondly, TCM data lacks unified quantitative standards, and tongue / pulse characteristics vary significantly among different regions, ages, and constitutions. For example, the threshold for yellow and greasy tongue coating in people from the Lingnan region needs to be lowered by 15% compared to those from the north to accommodate the characteristics of a damp-heat constitution. Existing AI models are mostly trained on general datasets, making it difficult to adapt to individual needs and lacking a dynamic update mechanism, leading to performance degradation over time. This solution achieves model self-optimization through an incremental learning system. Tens of thousands of new valid tongue data points (after manual verification) are automatically added to the training pool daily, and parameters are updated every 72 hours to improve diagnostic accuracy. It also supports regionally adaptive training, adjusting feature recognition thresholds for different regions. For example, the threshold for yellow and greasy tongue coating in Guangdong is lowered by 15% to ensure health analysis aligns with regional constitution characteristics. Furthermore, the system can track changes in a user's tongue appearance over the long term and dynamically adjust health recommendations based on pulse and lifestyle data. For instance, a hypertensive patient in Shenzhen, through system monitoring, noticed their tongue changing from pale red to dark red, combined with a "deep and wiry" pulse, providing a three-month advance warning of blood stasis risk and a basis for intervention.

[0025] Thirdly, traditional Chinese medicine analysis relies on physicians' experience, and high-quality resources are scarce in primary healthcare institutions. While existing AI analysis systems can improve efficiency, they often require specialized equipment, are costly, and struggle to cover remote areas. Furthermore, some systems output highly technical results that are difficult for patients to understand, impacting adherence. This solution achieves low-cost data collection through smart sensors (such as mobile phone cameras and piezoelectric sensors). Users can upload tongue / pulse data via mobile applications to receive personalized health advice. The output is presented in a visual report, accompanied by clear tongue images and easy-to-understand explanations. For example, if a patient's constitution leans towards phlegm and dampness, it is recommended to reduce oily foods, making the information readily understandable. In addition, it can assist in training primary care physicians, enabling them to rapidly improve their health analysis skills based on tongue and pulse waves through extensive case studies, thus narrowing the urban-rural healthcare gap.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1 The flowchart of an AI health analysis method based on tongue image and pulse wave is illustrated in one embodiment of this application.

[0029] Figure 2 The flowchart illustrating the extraction of tongue image feature vectors in one embodiment of this application is shown schematically.

[0030] Figure 3 The illustration shows a schematic diagram of an AI health analysis system based on tongue image and pulse wave in one embodiment of this application.

[0031] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0034] It should be noted that the data acquisition or information collection in this embodiment is performed after authorization by the user or the object of collection, and its process and purpose strictly follow the relevant regulations.

[0035] The block diagrams shown in the attached figures are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, or in one or more hardware modules composed of smart chips, smart integrated circuits, or application-specific integrated circuits (ASICs), or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] The implementation details of the technical solution of this application are described below:

[0038] Figure 1 A flowchart illustrating an AI health analysis method based on tongue image and pulse wave according to an embodiment of this application is shown. (Refer to...) Figure 1 As shown, this AI health analysis method based on tongue image and pulse wave includes at least steps S110 to S150, which are described in detail below:

[0039] The S110 uses smart sensors to collect images of the user's tongue and heart pulse data.

[0040] In this embodiment, a pre-deployed intelligent sensor system is used to collect images of the user's tongue and pulse data. A specialized intelligent high-definition camera automatically adjusts its focus and angle to clearly capture the user's tongue, ensuring an image that accurately reflects its color, shape, and coating. Simultaneously, intelligent sensors for collecting pulse data, such as pulse wave acquisition devices, are closely fitted to the corresponding parts of the user's body, precisely sensing the subtle physiological signal changes caused by the heartbeat and converting these changes into recordable pulse data. The entire acquisition process is efficient and orderly, providing foundational information for subsequent health analysis.

[0041] In one embodiment of this application, a user's tongue image and heart pulse data are collected using a smart sensor, including:

[0042] The user's tongue image is captured by a camera device, and the tongue image is preprocessed by median filtering;

[0043] Heart pulse data is acquired using a piezoelectric sensor, and high-frequency noise in the pulse data is filtered out using a low-pass filter.

[0044] By establishing a communication connection (such as via USB interface or Bluetooth) with professional tongue image acquisition equipment and pulse wave acquisition equipment, acquisition commands are sent to these devices. The high-definition camera in the tongue image acquisition device, in conjunction with a specific light source, captures an image of the tongue. The camera converts the light signal into an electrical signal, which is then converted into a digital tongue image by an image sensor. Tongue image acquisition is based on the principle of optical imaging; the specific light source provides uniform and suitable lighting conditions to ensure accurate representation of the color and details of the tongue image. The high-definition camera utilizes the photoelectric conversion characteristics of the image sensor to convert reflected light into a digital image.

[0045] High-precision sensors (such as piezoelectric sensors) in pulse wave acquisition devices detect the vibration signals of the human pulse, convert them into electrical signals, and then perform preliminary processing through internal signal conditioning circuits (such as amplification and filtering). Pulse wave acquisition utilizes physical principles such as the piezoelectric effect; the sensor converts the mechanical vibrations of the pulse into electrical signals for subsequent processing.

[0046] Optionally, after receiving the tongue image, a median filtering algorithm is used for denoising. Specifically, the image is divided into multiple small neighborhood windows (such as 3*3 or 5*5 pixel windows). For the pixel values ​​within each window, they are sorted, and the median value of the sorted values ​​is taken as the new value of the center pixel of that window. By sliding this window across the entire image, this operation is performed on each pixel, thereby achieving image denoising. Median filtering is a non-linear filtering method based on sorting statistics theory. Its core idea is to replace the value of a pixel with the median value of its neighborhood pixels. Because the values ​​of noisy pixels usually differ significantly from the values ​​of surrounding normal pixels, they are at opposite ends after sorting. Taking the median value can effectively remove these isolated noise points while preserving the edge information of the image.

[0047] Optionally, after receiving the pulse data, a low-pass filter can be used to remove high-frequency noise. In pulse data, useful physiological signals are typically located in the lower frequency range; filtering out high-frequency noise, such as electromagnetic interference, is crucial. The Butterworth low-pass filter features a maximally flat frequency response curve within its passband, effectively preserving the shape of the pulse data. Removing high-frequency interference components from the pulse data makes the pulse waveform smoother, accurately reflecting the physiological characteristics of the pulse, such as the timing and amplitude information of the main wave and tidal wave, providing reliable signal data for subsequent feature extraction and analysis.

[0048] The above process, through intelligent sensors, accurately acquires the user's tongue image and heart pulse data, providing a raw data foundation for subsequent analysis and ensuring reliable data support for later work. After acquiring the tongue image using a camera device, median filtering is used for preprocessing, effectively removing noise points from the image, making the tongue image clearer and more accurate, highlighting key tongue features such as tongue color, shape, and coating, creating favorable conditions for subsequent feature extraction. Using a piezoelectric sensor to acquire pulse data and filtering out high-frequency noise with a low-pass filter makes the pulse data purer, retaining useful low-frequency information, more realistically reflecting the heartbeat, and improving the quality of the pulse data.

[0049] S120, The convolutional neural network model in deep learning is used to extract features from the tongue image and output the tongue image feature vector.

[0050] In this embodiment, the preprocessed tongue image is input into a convolutional neural network model in deep learning. The convolutional layers use different convolutional kernels to slide and scan the image, capturing local features such as the tongue's color distribution and texture details. Pooling layers compress and filter the features output from the convolutional layers, retaining key features and reducing data volume. Subsequently, the fully connected layers of the convolutional neural network model integrate and organize the extracted features. Through this series of progressively deeper processing steps, features representing key information about the tongue are extracted from the complex tongue image and output as a tongue feature vector.

[0051] like Figure 2 As shown, in one embodiment of this application, a convolutional neural network model in deep learning is used to extract features from the tongue image and output a tongue image feature vector, including:

[0052] S210, a convolutional neural network model obtained based on deep learning training;

[0053] S220, The tongue image is input into the convolutional layer and pooling layer of the convolutional neural network for feature extraction, and the feature map of the tongue image is output.

[0054] S230, after multiple rounds of convolution and pooling, the feature map is flattened into a tongue image feature vector.

[0055] In this embodiment, a convolutional neural network model is constructed based on deep learning. Preprocessed tongue images and pulse data are read from memory or storage devices and loaded into the input layer of the convolutional neural network model as a multidimensional array data structure. Simultaneously, various parameters of the convolutional neural network model are initialized, including the weights of the convolutional kernels and bias terms. These parameters are randomly initialized before model training and are continuously adjusted and optimized as the training process progresses.

[0056] The input tongue image is fed into the first convolutional layer of a convolutional neural network. In this embodiment, the convolutional layer contains multiple convolutional kernels, each of which is a feature detector. The convolutional kernels slide across the tongue image with a preset stride. At each position, each element of the convolutional kernel is multiplied by the corresponding element of the local region of the image, and then the products are summed. Different convolutional kernels focus on different local features of the image; for example, some convolutional kernels may be sensitive to the edge information of the tongue image, while others may focus more on areas of color change.

[0057] The feature maps obtained after convolutional layer processing contain a large amount of data. To reduce computation and enhance the model's robustness to minor changes in tongue images, these feature maps are input into pooling layers. Pooling layers set a pooling window on the feature map, selecting the maximum value within the window as its output. In this way, the pooling layer downsamples the feature map, reducing the data dimensionality while retaining the most significant features. This allows the model to focus more on the main features of the tongue image while ignoring subtle, unimportant changes.

[0058] Specifically, the input tongue image is fed into the convolutional layer of a convolutional neural network, where the convolutional kernel slides across the image. For each location, a weighted sum is calculated based on the weights corresponding to the convolutional kernel at that location and the pixel values ​​of the local image region to obtain the feature response value at that location. The sum between the feature response value and the bias term is then calculated, and an activation function is used to process this sum to generate the feature map corresponding to that convolutional kernel. A complete feature map is generated by traversing the entire image. The feature map is generated according to the following formula. for:

[0059]

[0060] in, Indicates the first l The feature value at position (i,j) in the convolutional layer. σ The activation function can be σ( x )=max(0, x Its function is to truncate negative values ​​to 0, retain positive values, and increase the nonlinearity of the model; Indicates the first l The weights of the convolutional kernel at position (m,n) are learned and adjusted during model training using the backpropagation algorithm. Indicates the first Pooling layer (or input image, when) l When =1, the pixel value at position (i+m,j+n) Indicates the first l The layer's bias term is also a learnable parameter; M,N The size of the local region selected during the convolution operation is indicated by . In this way, the convolutional layer can extract various local feature information from the tongue image. Multiple convolutional kernels operate in parallel to extract features of different directions, textures, and colors to generate corresponding feature maps, thereby enhancing the model's ability to express multi-dimensional features of the tongue image.

[0061] To extract higher-level, more abstract features of the tongue image, convolutional and pooling operations are repeatedly performed. As the network depth increases, each convolutional operation extracts more complex features based on the feature map of the previous layer. Shallow convolutional layers mainly extract low-level features of the tongue image, such as simple edges and color blocks, while deeper convolutional layers can combine these low-level features to extract higher-level features such as more complex shapes and texture combinations. Pooling layers continuously downsample the feature map, further compressing the data volume while enhancing important features.

[0062] The above process, based on deep learning training of a convolutional neural network model, enables the model to learn features from tongue images. After training with a large amount of data, the model can master various feature patterns of the tongue. The tongue image is then input into the convolutional and pooling layers of the convolutional neural network for feature extraction, outputting a feature map of the tongue. Convolutional layers capture local features of the image, while pooling layers reduce data dimensionality and retain key features. Through multiple rounds of processing, deeper features of the tongue are gradually extracted. After multiple rounds of convolution and pooling, the feature map is flattened into tongue feature vectors, transforming the complex features of the tongue into a form that is easy to process and analyze, preparing for subsequent multimodal fusion analysis.

[0063] After multiple rounds of convolution and pooling operations, the final feature map is flattened into a one-dimensional tongue image feature vector, which is then input into the fully connected layer of the convolutional neural network model. Each neuron in the fully connected layer is connected to all neurons in the previous layer, comprehensively integrating and synthesizing the various local features extracted earlier. Through the nonlinear transformation of the fully connected layer of the convolutional neural network model, the intrinsic correlations and combination information between features are further explored, fusing various tongue image features into a complete, multi-dimensional feature representation. Finally, after processing by the fully connected layer of the convolutional neural network model, a multi-dimensional feature vector of the tongue image is output. This feature vector contains various features of the tongue image, such as tongue color, tongue shape, and tongue coating, providing crucial feature input for subsequent health analysis.

[0064] S130, perform time-frequency analysis on the pulse data to generate a pulse feature vector.

[0065] In this embodiment, the collected pulse data is fed into a time-frequency analysis stage to analyze the pulse data from two different dimensions: time and frequency. From the time dimension, the changing patterns of the pulse over time can be observed, such as the rhythm of the heartbeat; from the frequency dimension, the information contained in different frequency components of the pulse signal can be extracted. Through this comprehensive analysis, the features in the pulse data are deeply mined, much like sorting out valuable intelligence from a jumble of clues, ultimately integrating these features to generate a pulse feature vector.

[0066] In one embodiment of this application, time-frequency analysis is performed on the pulse data to generate a pulse feature vector, including:

[0067] Based on a preset scaling transformation function, the signal features corresponding to each set of scaling and shifting factors are determined by adjusting the scaling and shifting factors.

[0068] By transforming the signal characteristics, multi-scale time-frequency analysis is performed on the pulse data to generate a pulse feature vector.

[0069] The preprocessed pulse data is read from storage locations such as memory or hard drive. This data is a sequence of discrete signal points that has undergone denoising, normalization, and other processing. Subsequently, initial settings are configured for subsequent time-frequency analysis operations, including determining the scaling function and initial parameter values ​​for the time-frequency analysis method.

[0070] A time-frequency analysis method based on scaling transform functions is used to perform multi-scale analysis of pulse data. By adjusting the scaling factor, the width of the scaling transform function is changed. At large scales, the scaling transform function is wider, which is suitable for analyzing low-frequency components in the signal and can capture relatively slow changes in pulse data. At small scales, the scaling transform function is narrower, which is used to analyze high-frequency components and can detect rapid changes and detailed information in the signal. At the same time, the translation factor is adjusted to move the scaling transform function along the time axis, thereby covering the entire time range of the pulse data and analyzing the characteristics of the signal at different time positions.

[0071] Specifically, for each group of scale factors and translation factors of the current signal, the input is into the conjugate function of the scale transformation function to obtain the sampling time period and frequency band in the current signal. This part of the signal is used as the target signal for integration, and the result of the integration is linearly processed based on the scale factor to generate the feature information of the signal at the current scale factor and time position.

[0072] Specifically, signal characteristics are determined using the following formula. for:

[0073]

[0074] in, It represents the characteristics of the signal and reflects the signal. In scale factor a and time location Feature information of the location; The scale factor determines the scaling transformation function. The degree of stretching; The translation factor determines the position of the scaling function on the time axis. Represents the scaling function. The conjugate of the scaling function is used with the signal. Perform inner product operations to extract signal features; This represents the input pulse data.

[0075] The above process matches different signal features through scaling and translation of the scaling transformation function. By adjusting the values ​​of the scaling factor and translation factor, multi-scale analysis of the signal is performed. The scaling factor controls the width of the scaling transformation function, thereby enabling the analysis of different frequency components; the translation factor controls the position of the scaling transformation function on the time axis, used to analyze the signal's characteristics at different time points. By calculating the signal features, the characteristic representation of the signal at different scales and time positions can be obtained. In this process, the corresponding feature representation is calculated by interacting the pulse data with the scaling transformation function under different scales and translations.

[0076] Based on the signal characteristics obtained from time-frequency analysis, feature points in pulse data can be identified. For example, by analyzing information such as extreme points in the signal characteristics, the height and time interval of feature points such as pulse rate and pulse waveform can be determined, or the position of waveform feature points such as the main wave and tidal wave of the pulse wave can be determined.

[0077] For example, the main wave corresponds to the peak value of the waveform generated by blood ejection when the heart contracts, and the tidal wave corresponds to the propagation and reflection of blood in the blood vessels. Based on the positional information of these feature points, the height and time interval of the main wave, tidal wave, etc. are calculated.

[0078] For example, regarding the important parameter of pulse rate, by detecting the periodicity of pulse data and locating extreme points through mathematical differentiation and setting the derivative to zero, we can determine the height and time interval of these characteristic points, obtain the time interval between adjacent pulse waves, and thus derive the pulse rate value (number of beats per minute). Furthermore, based on the mathematical relationship between pulse rate value and time interval, we can obtain pulse rate characteristic parameters by dividing the pulse rate value by the average time interval. These are pulse wave characteristic parameters that reflect the physiological state of the human body.

[0079] Following the above analysis and calculations, the extracted pulse wave characteristic parameters, such as pulse rate, main wave height, and tidal wave time interval, are organized and encapsulated to generate a pulse feature vector. This provides crucial information about the pulse wave for subsequent comprehensive health analysis. The above process fully leverages the advantages of time-frequency analysis in data processing and analysis, accurately extracting valuable characteristic parameters from complex pulse data and providing strong support for subsequent health assessments.

[0080] The above process, based on a preset scaling transformation function, determines the signal characteristics corresponding to each set of scaling and shifting factors by adjusting the scaling and shifting factors. This step can flexibly adapt to different pulse signal characteristics, providing suitable parameters for subsequent time-frequency analysis. Generating a pulse feature vector and performing multi-scale time-frequency analysis on pulse data by transforming signal characteristics allows for the extraction of feature information from different perspectives, generating a pulse feature vector that comprehensively reflects pulse characteristics and providing important pulse-related information for health assessment.

[0081] S140, Based on deep learning, perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector to generate fused features, and generate health status assessment results based on the fused features.

[0082] In this embodiment, a multimodal fusion network constructed through deep learning organically integrates tongue image feature vectors and pulse feature vectors in the fusion layer. For example, attention mechanisms are used to dynamically allocate the weights of the two, or feature complementarity is achieved through splicing and transformation of fully connected layers. The fused features are further refined by deep neural networks to capture the implicit correlation patterns between tongue images and pulse waves. Finally, they are mapped to specific health status assessment results through classifiers or regression models, such as physical constitution type determination, sub-health risk level, and other health analyses. The entire process realizes the intelligent transformation from multi-source physiological signals to health status.

[0083] In one embodiment of this application, multimodal data fusion analysis is performed on the tongue image feature vector and pulse feature vector based on deep learning to generate fused features, and a health status assessment result is generated based on the fused features, including:

[0084] The tongue image feature vector is input into the tongue image feature input layer of the pre-trained multimodal fusion network to output the first feature; the pulse feature vector is input into the pulse wave feature input layer of the multimodal fusion network to output the second feature.

[0085] In the fusion layer of the multimodal fusion network, the first feature and the second feature are subjected to attention-based fusion processing to generate fused features;

[0086] The fused features are transmitted to the fully connected layer of the multimodal fusion network, and the mapping relationship between the fused features is mined through nonlinear transformation to generate the health status assessment result.

[0087] In this embodiment, the tongue image feature vector and pulse wave feature parameters generated in the preceding steps are read. These data carry key information about the tongue image and pulse wave. Subsequently, a pre-built neural network model is initialized. This neural network model includes: a tongue image feature input layer, a pulse wave feature input layer, a fusion layer, and an output layer. Each layer has its specific function and parameter settings. At this time, the model's parameters are in their initial state, waiting to be optimized through training.

[0088] Specifically, the read tongue image feature vector is input into the tongue image feature input layer, which outputs the first feature. The pulse wave feature parameters are then fed into the pulse wave feature input layer, which outputs the second feature. The input layer serves as the interface between the model and external data, responsible for introducing data from different modalities into the model, providing the model with raw materials for analyzing health status, and laying the foundation for subsequent processing and analysis.

[0089] Next, in the fusion layer, features from tongue images and pulse waves are integrated. Specifically, the model can automatically learn how to fuse these two types of features through training. Based on a large amount of sample data on tongue images, pulse waves, and health status, the model learns from this sample data to determine the complex relationship between different feature combinations and health status, thereby automatically adjusting the fusion method and degree. For example, in some cases, tongue image features may be more critical for judging health status, so the model will give tongue image features higher weight during fusion; while in other cases, pulse wave features may play a dominant role, and the model will adjust the weight allocation accordingly. This automatic learning mechanism allows the model to fuse multi-source data more flexibly and accurately.

[0090] Specifically, in this embodiment, the first feature and the second feature are subjected to attention-based fusion processing in the fusion layer of the multimodal fusion network to generate fused features, including:

[0091] In the fusion layer of the multimodal fusion network, a tongue and pulse syndrome correlation map is constructed, which includes tongue image feature nodes, pulse feature nodes, and TCM syndrome nodes.

[0092] A matching function is constructed based on the correspondence between tongue and pulse in traditional Chinese medicine theory to calculate the correlation strength between nodes in the tongue and pulse syndrome correlation map.

[0093] Based on the multi-head graph attention mechanism generated by the correlation strength, the first feature and the second feature are fused together to generate a fused feature.

[0094] Specifically, in one embodiment of this application, a TCM knowledge graph is first constructed, namely, a tongue and pulse syndrome association graph G=(V,E,A). Specifically, the correspondence between tongue appearance and pulse appearance in classical TCM theory is first converted into a computer-understandable language to construct the tongue and pulse syndrome association graph.

[0095] Next, detailed descriptions of the relationship between tongue appearance, pulse appearance and syndrome were extracted and used as three types of nodes V in this atlas: features extracted from tongue appearance (such as tongue color and coating), features extracted from pulse (such as waveform and rhythm), and TCM syndromes (such as yin deficiency with fire excess and qi and blood deficiency).

[0096] After treating tongue appearance features, pulse features, and syndromes as nodes in the atlas, the relationships between them are determined based on descriptions in classical literature, and these relationships are used as edges E to represent the strength of the relationships between nodes. To quantify the strength of the relationships, statistical prior information is extracted from classical literature. For example, the frequency of the simultaneous occurrence of red tongue with yellow coating and rapid pulse in paragraphs describing heat syndromes is statistically analyzed. This frequency, after a certain normalization process, is used as the theoretical weight of the first feature of tongue appearance (red tongue with yellow coating) and the second feature of pulse (rapid pulse) in the corresponding heat syndrome dimension.

[0097] Specifically, in determining the strength of the association, the weights of the connection edges between nodes are predefined based on empirical rules from classic texts such as Traditional Chinese Medicine, and the first feature of the tongue image is generated first. Second characteristic of the pulse The corresponding syndrome mapping function is used to calculate the basic product of the two syndrome mapping functions. Then, based on the preset theoretical weights of each syndrome dimension, a weighted sum operation is performed on the basic product to generate a matching function between the first feature of the tongue appearance and the second feature of the pulse. Based on the above process, a matching function between the first feature of the tongue appearance and the second feature of the pulse is constructed. for:

[0098]

[0099] in, These represent the first characteristics of the tongue. Second characteristic of the pulse The syndrome mapping function is used to map the original features to... K One symptom dimension; The theoretical weight of the kth syndrome dimension is derived from the statistical prior of classical Chinese medicine literature. Represents the basic product operation between features; i, j These represent the node identifiers corresponding to the first feature and the second feature, respectively. K This indicates the number of symptom dimensions.

[0100] Subsequently, the correlation strength between nodes in the tongue and pulse syndrome correlation map was determined based on the matching function. for:

[0101]

[0102] in, The first characteristic of tongue appearance Second characteristic of the pulse The manifold distance in the latent space; The standard deviation of the feature pair consisting of the first feature and the second feature; The temperature parameter is used to control the sparsity of the association; tanh is the hyperbolic tangent function; exp is the exponential function. For example, the red tongue node and the pulse node are strongly correlated with the heat excess syndrome node, thus injecting a priori TCM logic into the subsequent intelligent fusion.

[0103] In the feature fusion stage, the actual collected user tongue and pulse feature vectors are injected into the corresponding feature nodes in the graph. Subsequently, a graph-based attention neural network is activated, allowing information to be transmitted and interacted with along the graph edges between nodes. The symptom nodes actively collect information from the tongue and pulse feature nodes connected to them. During this process, the attention mechanism dynamically calculates and assigns weights based on the information content of the first and second features and the predefined correlation strength of the graph, automatically focusing on feature combinations that are highly relevant to traditional Chinese medicine theory and show significant performance in this test, achieving a deep fusion driven by both theory and data.

[0104] Next, a diagnostic feature vector is generated. Through information transmission and aggregation within the network, each syndrome node gathers related tongue and pulse feature information, forming sub-features with clear TCM semantics. Finally, the importance of these syndrome nodes is evaluated. This is not a simple addition, but a weighted aggregation based on the activation intensity and graph association contribution of each node during the fusion process. The final output is a comprehensive, structured fusion feature rich in TCM diagnostic significance. This fusion feature characterizes the user's constitution bias and syndrome tendencies, laying a core foundation for generating accurate health assessments and recommendations.

[0105] The fused features generated in the fusion layer are transmitted to the fully connected layer. The neurons in the fully connected layer are connected to all neurons in the previous layer, enabling a comprehensive analysis of the fused features. Through nonlinear transformations, the intrinsic connections and potential patterns between features are further explored, thereby achieving an accurate assessment of health status.

[0106] During training, the model learns the complex mapping relationship between features and health status from a large number of samples, enabling it to provide reasonable evaluation results when new fused features are input. After processing by fully connected layers, the model outputs a health status assessment result based on the previously learned mapping relationship between features and health status. These results include traditional Chinese medicine constitution types, such as balanced constitution and qi deficiency constitution, as well as potential health risks, such as cardiovascular disease risk and digestive system disease risk, providing users with comprehensive health information.

[0107] The above process, by inputting tongue image feature vectors and pulse feature vectors into different input layers of the neural network model, enables the model to simultaneously receive feature information from two different modalities, laying the foundation for fusion analysis. Through attention-based fusion processing of tongue image and pulse wave features by the fusion layer, weights are automatically assigned according to feature importance, highlighting key features and making the fused features more accurately reflect the user's health status. The fused features are then transmitted to a fully connected layer, where nonlinear transformations are used to mine the mapping relationships between the fused features. This allows for a comprehensive consideration of the feature information from both tongue image and pulse wave, generating a more comprehensive and accurate health status assessment result.

[0108] S150, generate health recommendations based on the assessment results of the health status, and feed the health recommendations back to the user terminal.

[0109] In this embodiment, after obtaining the health status assessment results, the system intelligently matches various suggestions suitable for the health condition based on a built-in rich knowledge base of traditional Chinese medicine health preservation and a clinical experience case database. Next, these personalized health suggestions, including dietary adjustments, appropriate exercise methods, and daily routine points, are integrated and organized. Finally, the generated health suggestions are accurately fed back to the user's terminal device via network communication channels, allowing the user to view them promptly and adjust their lifestyle accordingly to protect their health.

[0110] In one embodiment of this application, generating health recommendations based on the assessment results of the health status and feeding the health recommendations back to the user terminal includes:

[0111] Based on the assessment results of the health status, a matching is performed in a preset health rule base to generate health suggestions;

[0112] The health advice will be fed back to the user's terminal.

[0113] The health status assessment results generated based on the aforementioned steps include key information such as the user's TCM constitution type and potential health risks. Simultaneously, the system accesses a health rule base. In this embodiment, the health rule base is constructed based on a database of TCM health preservation knowledge and clinical experience. This health rule base stores a large amount of information corresponding to different health states, including dietary recommendations, exercise suggestions, and TCM conditioning plans, serving as the knowledge source for generating personalized health recommendations.

[0114] Optionally, a combination of rule-based reasoning and case-based reasoning can be employed. For rule-based reasoning, a series of rules related to health status and health recommendations are pre-set in a health rule base. For example, if the assessment results indicate that a user has a Qi deficiency constitution, the rules might recommend foods that replenish Qi, such as yam and red dates. Based on the health status assessment results, basic health recommendations that meet the criteria are selected from the database according to these rules.

[0115] In terms of case-based reasoning, the current user's health status assessment is matched with historical cases stored in the health rule base. Cases with similar health statuses to the current user are identified, and these cases contain previously provided effective health advice. By analyzing these similar cases, more targeted and practical health recommendations can be extracted.

[0116] It should be noted that this embodiment does not directly copy the health advice derived from rule-based and case-based reasoning, but rather makes appropriate adjustments and optimizations based on the specific circumstances of the current user. For example, factors such as the user's age, gender, and lifestyle habits (e.g., dietary preferences, exercise habits) are taken into consideration. If the user is elderly, the exercise advice may lean towards gentler forms of exercise, such as Tai Chi or walking; if the user is usually busy with work, the dietary advice may focus on easily prepared and nutritious foods.

[0117] After the above processing, the final personalized health recommendations are organized and formatted, outputting them in a way that is easy for users to understand and accept. These recommendations cover dietary suggestions, such as recommended food types and recipes; exercise suggestions, including exercise types, frequency, and intensity; and traditional Chinese medicine treatment plans, such as herbal formulas and acupressure, providing users with comprehensive health guidance. By fully utilizing the data processing and logical reasoning capabilities of computing devices, combined with a professional knowledge base and the user's specific situation, personalized health recommendations are generated and output to help users better manage their health.

[0118] The above process, based on the health status assessment results, matches the data against a pre-set health rule base to quickly and accurately generate health suggestions suitable for the user's health condition, providing specific health guidance. These health suggestions are then fed back to the user's device, allowing the user to promptly understand their health status and receive corresponding advice, facilitating action to improve their health.

[0119] In one embodiment of this application, after generating health recommendations based on the health status assessment results and feeding the health recommendations back to the user terminal, the method further includes:

[0120] Acquire tongue images, pulse data, and lifestyle data of the user during the execution of the health advice;

[0121] The health recommendations are adjusted based on the tongue image, pulse data, and lifestyle data.

[0122] In this embodiment, once the user begins to follow the health advice provided by the system, a new round of data collection is initiated. A dedicated tongue image acquisition device will, under suitable environmental conditions and according to preset rules and frequencies, photograph and record the user's tongue, capturing subtle changes in the tongue during the implementation of health advice, such as changes in color intensity and the thickness of the tongue coating. Simultaneously, a pulse data acquisition device will continuously monitor the user's heart rate and pulse, recording the rhythm and intensity of each heartbeat. Furthermore, various daily life data of the user, such as the type and quantity of daily food intake, exercise duration and intensity, and sleep quality and duration, are collected through manual input by the user or automatic transmission from smart wearable devices, providing comprehensive data support for subsequent adjustments to health advice.

[0123] After collecting users' tongue images, pulse data, and lifestyle data, intelligent analysis technology is used to conduct in-depth analysis of this data. For tongue images, image recognition and feature extraction technologies are used to compare with previous data, analyze the changing trends of tongue features, and the potential health implications of these changes. For pulse data, professional algorithms are used to analyze heart rate patterns to determine whether there has been any improvement or abnormality in cardiac function. For lifestyle data, the system assesses whether the user's diet, exercise, sleep, and other lifestyle habits meet the requirements of health recommendations, and the impact of these habits on the user's health. Based on these analytical results, the user's current health status is reassessed to determine the effectiveness of the health recommendations.

[0124] Based on the analysis and evaluation of users' tongue images, pulse data, and lifestyle data, the system intelligently adjusts and optimizes existing health recommendations. If the user's tongue image and pulse are trending positively, and the lifestyle data indicates that the user is adhering well to the health recommendations, some enhanced health guidance may be added to the existing recommendations to help the user further improve their health. Conversely, if the analysis results show that the user's health condition is not significantly improving or new problems have emerged, the system carefully investigates the causes and, combining traditional Chinese medicine knowledge and clinical experience, makes targeted modifications to the health recommendations. For example, it may adjust the food combinations in the diet plan or change the type and intensity of exercise recommended, ensuring that the health recommendations always align with the user's actual health needs and provide more precise and effective health guidance.

[0125] The above process, by acquiring tongue images, pulse data, and lifestyle data from users during the implementation of health advice, enables real-time tracking of changes in users' health and adjustments to their lifestyle habits, providing the latest data basis for adjusting health advice. Health advice is adjusted based on newly acquired tongue images, pulse data, and lifestyle data, allowing it to dynamically adapt to changes in users' health conditions and lifestyles, ensuring the effectiveness and relevance of health management at all times.

[0126] This application's technical solution collects a user's tongue image and heart pulse data using intelligent sensors; it employs a convolutional neural network model in deep learning to extract features from the tongue image, outputting a tongue image feature vector; it performs time-frequency analysis on the pulse data to generate a pulse feature vector; based on deep learning, it performs multimodal data fusion analysis on the tongue image feature vector and pulse feature vector to generate fused features, and generates a health status assessment result based on the fused features; it generates health suggestions based on the health status assessment result and feeds these suggestions back to the user's terminal. By comprehensively utilizing tongue image and pulse wave information, combined with artificial intelligence technology, a complete process from data collection to health suggestion feedback is realized. It fully leverages the advantages of traditional Chinese medicine health analysis while improving the efficiency and accuracy of health analysis with modern technological means, providing users with personalized and precise health management solutions.

[0127] The following describes embodiments of the AI ​​health analysis system based on tongue image and pulse wave of this application, which can be used to execute the AI ​​health analysis method based on tongue image and pulse wave in the above embodiments of this application. It is understood that the AI ​​health analysis system based on tongue image and pulse wave can be a computer program (including program code) running on a computer device. For example, the AI ​​health analysis system based on tongue image and pulse wave can install application software or management software to achieve cloud computing of the big data generated during the application process through a cloud platform. The AI ​​health analysis system based on tongue image and pulse wave can be used to execute the corresponding steps in the method provided in the embodiments of this application. For details not disclosed in the embodiments of the AI ​​health analysis system based on tongue image and pulse wave of this application, please refer to the embodiments of the AI ​​health analysis method based on tongue image and pulse wave described above.

[0128] Figure 3 A block diagram of an AI health analysis system based on tongue image and pulse wave according to an embodiment of this application is shown.

[0129] Reference Figure 3 As shown, an AI health analysis system based on tongue image and pulse wave according to an embodiment of this application includes:

[0130] The acquisition module 310 is used to acquire images of the user's tongue and pulse data of the heart through a smart sensor;

[0131] Extraction module 320 is used to extract features from the tongue image using a convolutional neural network model in deep learning and output a tongue image feature vector;

[0132] Analysis module 330 is used to perform time-frequency analysis on the pulse data and generate a pulse feature vector;

[0133] The evaluation module 340 is used to perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector based on deep learning, generate fused features, and generate an evaluation result of health status based on the fused features.

[0134] The feedback module 350 is used to generate health suggestions based on the assessment results of the health status and to feed the health suggestions back to the user terminal.

[0135] In this application, based on the aforementioned scheme, the step of acquiring a user's tongue image and heart pulse data through a smart sensor includes: acquiring a user's tongue image through a camera device and preprocessing the tongue image according to median filtering; acquiring heart pulse data through a piezoelectric sensor and filtering out high-frequency noise in the pulse data through a low-pass filter.

[0136] In this application, based on the aforementioned scheme, the step of using a convolutional neural network model in deep learning to extract features from the tongue image and output a tongue image feature vector includes: obtaining a convolutional neural network model based on deep learning training; inputting the tongue image into the convolutional layer and pooling layer of the convolutional neural network for feature extraction and outputting a tongue image feature map; and after multiple rounds of convolution and pooling processing, flattening the feature map into a tongue image feature vector.

[0137] In this application, based on the aforementioned scheme, the step of performing time-frequency analysis on the pulse data to generate a pulse feature vector includes: determining the signal features corresponding to each set of scale factors and translation factors by adjusting the scale factors and translation factors based on a preset scale transformation function; and performing multi-scale time-frequency analysis on the pulse data by transforming the signal features to generate a pulse feature vector.

[0138] In this application, based on the aforementioned scheme, the step of performing multimodal data fusion analysis on the tongue image feature vector and pulse feature vector using deep learning to generate fused features, and generating a health status assessment result based on the fused features, includes: inputting the tongue image feature vector into the tongue image feature input layer of a pre-trained multimodal fusion network to output a first feature; inputting the pulse feature vector into the pulse wave feature input layer of the multimodal fusion network to output a second feature; performing attention-based fusion processing on the first feature and the second feature in the fusion layer of the multimodal fusion network to generate fused features; and transmitting the fused features to the fully connected layer of the multimodal fusion network, mining the mapping relationship between the fused features through nonlinear transformation to generate a health status assessment result.

[0139] In this application, based on the aforementioned scheme, the step of performing attention-based fusion processing on the first feature and the second feature in the fusion layer of the multimodal fusion network to generate fused features includes: constructing a tongue-pulse syndrome association map containing tongue image feature nodes, pulse feature nodes, and traditional Chinese medicine syndrome nodes in the fusion layer of the multimodal fusion network; constructing a matching function through the tongue-pulse correspondence in traditional Chinese medicine theory to calculate the association strength between each node in the tongue-pulse syndrome association map; and performing attention-based fusion processing on the first feature and the second feature based on a multi-head graph attention mechanism generated by the association strength to generate fused features.

[0140] In this application, based on the aforementioned scheme, the step of generating health suggestions based on the assessment results of the health status and feeding the health suggestions back to the user terminal includes: matching the health status assessment results in a preset health rule base to generate health suggestions; and feeding the health suggestions back to the user terminal.

[0141] In this application, based on the aforementioned scheme, after generating health recommendations based on the health status assessment results and feeding the health recommendations back to the user terminal, the method further includes: acquiring tongue images, pulse data, and lifestyle data of the user during the execution of the health recommendations; and adjusting the health recommendations based on the tongue images, pulse data, and lifestyle data.

[0142] This application's technical solution collects a user's tongue image and heart pulse data using intelligent sensors; it employs a convolutional neural network model in deep learning to extract features from the tongue image, outputting a tongue image feature vector; it performs time-frequency analysis on the pulse data to generate a pulse feature vector; based on deep learning, it performs multimodal data fusion analysis on the tongue image feature vector and pulse feature vector to generate fused features, and generates a health status assessment result based on the fused features; it generates health suggestions based on the health status assessment result and feeds these suggestions back to the user's terminal. By comprehensively utilizing tongue image and pulse wave information, combined with artificial intelligence technology, a complete process from data collection to health suggestion feedback is realized. It fully leverages the advantages of traditional Chinese medicine health analysis while improving the efficiency and accuracy of health analysis with modern technological means, providing users with personalized and precise health management solutions.

[0143] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.

[0144] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not impose any limitations on the function and scope of use of the embodiments of this application.

[0145] In this embodiment, the computer system includes a central processing unit 401, which can perform various appropriate actions and processes based on programs stored in read-only memory 402 or programs loaded from storage section 408 into random access memory 403, such as executing the AI ​​health analysis method based on tongue image and pulse wave described in the above embodiment. The random access memory 403 also stores various programs and data required for system operation, thereby realizing big data storage and big data management. The central processing unit 401, read-only memory 402, and random access memory 403 are interconnected via bus 404. Input / output interface 405 is also connected to bus 404.

[0146] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, etc.; an output section 407 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 410 as needed so that computer programs read from it can be installed into the storage section 408 as needed.

[0147] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 409, and / or installed from removable medium 411. When the computer program is executed by central processing unit 401, it performs various functions defined in the system of this application.

[0148] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0150] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0151] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0152] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to implement the AI ​​health analysis method based on tongue image and pulse wave described in the above embodiments.

[0153] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0154] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the methods according to the embodiments of this application.

[0155] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0156] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An AI-based health analysis method based on tongue image and pulse wave, characterized in that, include: The system collects images of the user's tongue and heart pulse data using intelligent sensors. The tongue image is used to extract features using a convolutional neural network model in deep learning, and the tongue image feature vector is output. Perform time-frequency analysis on the pulse data to generate a pulse feature vector; Based on deep learning, multimodal data fusion analysis is performed on the tongue image feature vector and pulse feature vector to generate fused features, and an assessment result of health status is generated based on the fused features. Based on the assessment results of the health status, health recommendations are generated and fed back to the user's terminal; Specifically, deep learning is used to perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector to generate fused features, and health status assessment results are generated based on the fused features, including: The tongue image feature vector is input into the tongue image feature input layer of the pre-trained multimodal fusion network to output the first feature; the pulse feature vector is input into the pulse wave feature input layer of the multimodal fusion network to output the second feature. In the fusion layer of the multimodal fusion network, the first feature and the second feature are subjected to attention-based fusion processing to generate fused features; The fused features are transmitted to the fully connected layer of the multimodal fusion network, and the mapping relationship between the fused features is mined through nonlinear transformation to generate the health status assessment result. The first feature and the second feature are fused based on attention in the fusion layer of the multimodal fusion network to generate fused features, including: In the fusion layer of the multimodal fusion network, a tongue and pulse syndrome association map is constructed, which includes tongue image feature nodes, pulse feature nodes, and traditional Chinese medicine syndrome nodes; wherein, the traditional Chinese medicine syndrome includes yin deficiency with fire excess and qi and blood deficiency. A matching function is constructed based on the correspondence between tongue and pulse in Traditional Chinese Medicine theory. for: in, These represent the first characteristics of the tongue. Second characteristic of the pulse The symptom mapping function is used to map the original features to... K One symptom dimension; The theoretical weight of the k-th symptom dimension is derived from the statistical prior of classic Chinese medicine literature. Represents the basic product operation between features; i, j These represent the node identifiers corresponding to the first feature and the second feature, respectively. K Indicates the number of symptom dimensions; Calculate the correlation strength between nodes in the tongue and pulse syndrome correlation map. for: in, The first characteristic of tongue appearance Second characteristic of the pulse The manifold distance in the latent space; The standard deviation of the feature pair consisting of the first feature and the second feature; The temperature parameter is used to control the sparsity of the correlation; tanh is the hyperbolic tangent function; exp is an exponential function. Based on the multi-head graph attention mechanism generated by the correlation strength, the first feature and the second feature are fused together to generate a fused feature.

2. The AI ​​health analysis method based on tongue image and pulse wave as described in claim 1, characterized in that, The tongue image is processed using a convolutional neural network model from deep learning to extract features and output a tongue image feature vector, including: A convolutional neural network model is obtained based on deep learning training; The tongue image is input into the convolutional and pooling layers of the convolutional neural network for feature extraction, and the feature map of the tongue image is output. After multiple rounds of convolution and pooling, the feature map is flattened into a tongue image feature vector.

3. The AI ​​health analysis method based on tongue image and pulse wave as described in claim 1, characterized in that, Perform time-frequency analysis on the pulse data to generate a pulse feature vector, including: Based on a preset scaling transformation function, the signal features corresponding to each set of scaling and shifting factors are determined by adjusting the scaling and shifting factors. By transforming the signal characteristics, multi-scale time-frequency analysis is performed on the pulse data to generate a pulse feature vector.

4. The AI ​​health analysis method based on tongue image and pulse wave as described in claim 1, characterized in that, Based on the assessment results of the health status, health recommendations are generated and fed back to the user terminal, including: Based on the assessment results of the health status, a matching is performed in a preset health rule base to generate health suggestions; The health advice will be fed back to the user's terminal.

5. The AI ​​health analysis method based on tongue image and pulse wave according to any one of claims 1-4, characterized in that, After generating health recommendations based on the health status assessment results and feeding the health recommendations back to the user terminal, the process also includes: Acquire tongue images, pulse data, and lifestyle data of the user during the execution of the health advice; The health recommendations are adjusted based on the tongue image, pulse data, and lifestyle data.

6. An AI health analysis system based on tongue image and pulse wave, characterized in that, include: The acquisition module is used to collect images of the user's tongue and pulse data of the heart through intelligent sensors; The extraction module is used to extract features from the tongue image using a convolutional neural network model in deep learning and output a tongue image feature vector. The analysis module is used to perform time-frequency analysis on the pulse data and generate a pulse feature vector; The evaluation module is used to perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector based on deep learning, generate fused features, and generate an evaluation result of health status based on the fused features. The feedback module is used to generate health suggestions based on the assessment results of the health status and to feed the health suggestions back to the user terminal. Specifically, deep learning is used to perform multimodal data fusion analysis on the tongue image feature vector and pulse feature vector to generate fused features, and health status assessment results are generated based on the fused features, including: The tongue image feature vector is input into the tongue image feature input layer of the pre-trained multimodal fusion network to output the first feature; the pulse feature vector is input into the pulse wave feature input layer of the multimodal fusion network to output the second feature. In the fusion layer of the multimodal fusion network, the first feature and the second feature are subjected to attention-based fusion processing to generate fused features; The fused features are transmitted to the fully connected layer of the multimodal fusion network, and the mapping relationship between the fused features is mined through nonlinear transformation to generate the health status assessment result. The first feature and the second feature are fused based on attention in the fusion layer of the multimodal fusion network to generate fused features, including: In the fusion layer of the multimodal fusion network, a tongue and pulse syndrome association map is constructed, which includes tongue image feature nodes, pulse feature nodes, and traditional Chinese medicine syndrome nodes; wherein, the traditional Chinese medicine syndrome includes yin deficiency with fire excess and qi and blood deficiency. A matching function is constructed based on the correspondence between tongue and pulse in Traditional Chinese Medicine theory. for: in, These represent the first characteristics of the tongue. Second characteristic of the pulse The symptom mapping function is used to map the original features to... K One symptom dimension; The theoretical weight of the k-th symptom dimension is derived from the statistical prior of classic Chinese medicine literature. Represents the basic product operation between features; i, j These represent the node identifiers corresponding to the first feature and the second feature, respectively. K Indicates the number of symptom dimensions; Calculate the correlation strength between nodes in the tongue and pulse syndrome correlation map. for: in, The first characteristic of tongue appearance Second characteristic of the pulse The manifold distance in the latent space; The standard deviation of the feature pair consisting of the first feature and the second feature; The temperature parameter is used to control the sparsity of the correlation; tanh is the hyperbolic tangent function; exp is an exponential function. Based on the multi-head graph attention mechanism generated by the correlation strength, the first feature and the second feature are fused together to generate a fused feature.

7. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI ​​health analysis method based on tongue image and pulse wave as described in any one of claims 1 to 5.

8. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the AI ​​health analysis method based on tongue image and pulse wave as described in any one of claims 1 to 5.

Images