Method, system and device for recommending health management solutions based on tongue image monitoring

Abstract

The present application relates to the field of diagnostic management technology, and provides a method, system, and device for recommending a physical health management plan based on tongue image monitoring. Acquire the user's physiological information and tongue image; pre-process the physiological information according to the type of physiological information to generate a first eigenvector; monitor the tongue state in the tongue image, and generate a second eigenvector according to the tongue state; extract cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition, and fuse them to generate a fused eigenvector; determine the emotion probability distribution based on the fused eigenvector, generate a physical conditioning plan based on the emotion probability distribution, and send the physical conditioning plan to the user terminal. Through multimodal feature extraction, the physiological and tongue image information are fused, and the association between the tongue image crack features and the emotional state is mined based on the cross-modal correlation features to accurately evaluate the emotional state, thereby improving the accuracy and comprehensiveness of emotion recognition and management.

Description

Technical Field

[0001] The present application relates to the field of diagnostic management technology, and in particular to a method, system and device for recommending a health management plan based on tongue image monitoring. Background Art

[0002] With the accelerating pace of modern life and increasing awareness of health, emotional well-being management has become an increasingly important area. Smart wearable devices and tongue monitoring technology offer new methods and approaches for emotional well-being management. Smart wearable devices, such as smart watches and smart bracelets, are already widely used in daily life. They can monitor users' physiological data, such as heart rate, blood pressure, and sleep, in real time. This data is closely related to physical condition and provides an important reference for health management. Tongue monitoring has a long history in Traditional Chinese Medicine (TCM), where observation of tongue patterns can be used to assess a person's health. For example, the prior art study "Clinical Study on the Characteristics of Fissured Tongue and Its Correlation with Syndromes" observed 249 outpatient and inpatient patients with fissured tongue. The study showed that fissured tongue is most commonly associated with yin deficiency, which in TCM is often associated with emotional distress and mental stress. For example, the liver qi's function of dispersing qi and blood directly affects the circulation of qi and blood. Patients with liver qi stagnation often experience purple bruises and noticeable cracks on the edges of the tongue. Chronic emotional tension or excessive worry can easily lead to hyperactive heart fire, resulting in a red tip of the tongue with cracks often concentrated there.

[0003] In actual applications, the physiological data and tongue monitoring data collected by smart wearable devices belong to data of different modalities. Current emotion recognition methods are mostly based on limited sample data and simple models, which make it difficult to accurately identify complex emotional states and physical states, and their accuracy and comprehensiveness are difficult to guarantee. Summary of the Invention

[0004] The present application provides a method, system and device for recommending a physical health management plan based on tongue image monitoring, which can at least to some extent solve the problem of low accuracy and comprehensiveness of emotion recognition methods.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of the present application, a method for recommending a physical health management plan based on tongue image monitoring is provided, including: obtaining a user's physiological information and tongue image; preprocessing the physiological information according to the type of the physiological information to generate a first eigenvector; monitoring the tongue state in the tongue image, and generating a second eigenvector according to the tongue state; extracting cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology, and fusing them to generate a fused eigenvector; determining an emotion probability distribution based on the fused eigenvector, generating a physical conditioning plan based on the emotion probability distribution, and sending the physical conditioning plan to a user terminal.

[0007] In the present application, based on the aforementioned scheme, the physiological information is preprocessed separately according to the type of the physiological information to generate a first eigenvector, including: determining the interval parameters according to the heartbeat interval in the physiological information; determining the frequency domain power ratio according to the power spectral density of the heart rate data in the physiological information; determining the skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; normalizing and combining the interval parameters, the frequency domain power ratio and the skin conductance response frequency to generate a first eigenvector.

[0008] In the present application, based on the aforementioned scheme, the monitoring of the tongue state in the tongue image and the generation of a second feature vector according to the tongue state include: performing image recognition and segmentation on the tongue image to obtain the tongue area and the crack area; performing feature extraction on the tongue area to obtain tongue color features and tongue shape features; performing feature extraction on the crack area to obtain crack features; and combining the tongue color features, the tongue shape features and the crack features to generate a second feature vector.

[0009] In the present application, based on the above-mentioned scheme, the feature extraction of the crack area to obtain the crack feature includes: identifying the cracks in the tongue image; calculating the weighted sum of the pixel areas of the connected areas at the crack, calculating the pixel area of ​​the entire tongue image, dividing the weighted sum of the pixel areas of the connected areas at the crack by the pixel area of ​​the entire tongue image, and obtaining the result as the crack density of the tongue image; generating a binary image of the tongue image, overlaying a grid on the binary image, and determining the morphological complexity of the crack based on the number of grids containing cracks; determining the first spatial distribution at the crack based on the pixel area of ​​the connected areas at the crack, and determining the second spatial distribution of all cracks in the entire tongue image based on the first spatial distribution; combining the crack density, the morphological complexity and the second spatial distribution to generate a crack feature.

[0010] In the present application, based on the aforementioned scheme, the cross-modal correlation features in the first eigenvector and the second eigenvector are extracted by tensor decomposition technology, and fused to generate a fused feature vector, including: constructing a three-dimensional tensor according to the physiological, tongue image and time dimensions of the first eigenvector and the second eigenvector collected at the same time; decomposing the three-dimensional tensor, and generating a core tensor based on the interactive information between the physiological, tongue image and time dimensions as the cross-modal correlation feature; combining the core tensor according to the physiological, tongue image and time dimensions to generate a fused feature vector.

[0011] In the present application, based on the aforementioned scheme, the determination of the emotion probability distribution based on the fused feature vector includes: initializing the parameters and loss function of the deep learning model; inputting the fused feature vector into the deep learning model, and outputting the probability corresponding to each emotion category as the emotion probability distribution.

[0012] In the present application, based on the aforementioned scheme, a physical conditioning plan is generated according to the emotion probability distribution, and the physical conditioning plan is sent to the user terminal, including: obtaining a mapping rule between a preset emotion probability and a physical conditioning plan; determining the user's emotional tendency based on the probability value corresponding to each emotion in the emotion probability distribution; matching the emotional tendency and the mapping rule to generate a physical conditioning plan.

[0013] In the present application, based on the aforementioned scheme, after determining the emotion probability distribution based on the fused feature vector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to the user terminal, it also includes: generating a physical management report based on the emotion probability distribution and the physical conditioning plan; and sending the physical management report to the user terminal.

[0014] According to one aspect of the present application, a health management plan recommendation system based on tongue image monitoring is provided, comprising:

[0015] an acquisition unit, configured to acquire physiological information and tongue images of a user;

[0016] a first feature unit, configured to preprocess the physiological information according to the type of the physiological information to generate a first feature vector;

[0017] a second feature unit, configured to monitor a tongue state in the tongue image and generate a second feature vector according to the tongue state;

[0018] a correlation unit, configured to extract cross-modal correlation features from the first eigenvector and the second eigenvector by using a tensor decomposition technique, and fuse them to generate a fused feature vector;

[0019] A management unit is configured to determine an emotion probability distribution based on the fused feature vector, generate a body conditioning plan according to the emotion probability distribution, and send the body conditioning plan to a user terminal.

[0020] According to one aspect of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for recommending a physical health management plan based on tongue image monitoring as described in the above embodiment is implemented.

[0021] According to one aspect of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for recommending a physical health management plan based on tongue image monitoring as described in the above embodiments.

[0022] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method for recommending a health management plan based on tongue image monitoring, as provided in the various optional implementations described above.

[0023] The technical solution of the present application obtains the physiological information and tongue image of the user; pre-processes the physiological information according to the type of the physiological information to generate a first eigenvector; monitors the tongue state in the tongue image and generates a second eigenvector according to the tongue state; extracts cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology, and fuses them to generate a fused eigenvector; determines the emotion probability distribution based on the fused eigenvector, generates a body conditioning plan based on the emotion probability distribution, and sends the body conditioning plan to the user terminal. Physiological and tongue image information are fused through multimodal feature extraction, and the association between tongue crack features and emotional state is mined based on cross-modal correlation features to accurately assess the emotional state, generate a personalized body management report and send the body conditioning plan to the user terminal, thereby improving the accuracy and comprehensiveness of emotion recognition and management, and providing users with a scientific, accurate and convenient emotional health management plan.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0026] Figure 1 The flowchart of the method for recommending a health management plan based on tongue image monitoring in one embodiment of the present application is schematically shown.

[0027] Figure 2 The flowchart of generating the first feature vector in one embodiment of the present application is schematically shown.

[0028] Figure 3 The following schematically shows a system for recommending a health management plan based on tongue image monitoring in one embodiment of the present application.

[0029] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0031] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, systems, steps etc. can be adopted. In other cases, known methods, systems, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0032] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0033] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0034] During the application of this technical solution, users select the tongue image capture function in the terminal software and enter their customer information. The user then captures the tongue image through the terminal. The captured tongue image is analyzed, and the system automatically analyzes tongue characteristics, such as tongue color, coating color, and tongue shape. Combined with the body constitution, a tongue image report containing health risk warnings is generated. Finally, based on the analysis results, users are provided with personalized health solutions, including Traditional Chinese Medicine (TCM) treatments (acupuncture, massage, etc.), dietary recommendations (medicated food, tea, etc.), exercise plans (Tai Chi, Ba Duan Jin, etc.), and treatment schedules (frequency and duration), along with precautions and periodic assessment reminders.

[0035] For example, in the application scenario of a traditional Chinese medicine clinic, the patient takes a photo of the tongue on site, and the system immediately generates a physical constitution dialectical report, matches acupuncture, medicated diet and other conditioning plans, and assists doctors in formulating personalized treatment courses.

[0036] For example, at the physical examination and health management center, members can undergo regular tongue examinations to dynamically warn of cardiovascular / digestive risks and adjust their exercise plans and diet cycles.

[0037] For example, individual users complete tongue diagnosis through terminal devices, and AI generates sub-health conditioning plans, such as shoulder and neck massage, music, and accompanying physical therapy recommendations twice a week.

[0038] For example, family users can take photos of the tongue of the whole family, and according to the tongue analysis results of the whole family, they can uniformly push diet therapy, foot bath recipes and periodic assessment reminders to prevent chronic diseases.

[0039] Specifically, the generated health solutions are categorized in this embodiment into various types, including manual techniques, instrumental techniques, medicinal methods, and mental methods. Specifically, manual techniques (acupressure, patting, and massage) regulate qi and blood through physical manipulation, relieving pain and fatigue; instrumental techniques (acupuncture, scraping, cupping, and medicinal baths) utilize tools to dredge meridians and treat cold and dampness, skin conditions, and other issues; medicinal methods (herbal, biological, and mineral-based) combine medication to regulate internal organs; and mental methods (perception and meditation) integrate traditional theories with mind-body adjustments to analyze health trends and provide wellness recommendations. Each method is labeled with applicable symptoms, such as insomnia, uterine coldness, and eczema.

[0040] In addition, the embodiments of this application also have the function of providing personalized treatment course recommendations based on the user's health status and tongue diagnosis results. This includes time recommendations, frequency recommendations, staged assessments, and precautions. For example, in terms of treatment duration, short-term (1-3 months) or long-term (6 months or more) conditioning plans are recommended based on the type and severity of the disease; in terms of frequency recommendations, the frequency of TCM physiotherapy (such as twice a week), the adjustment cycle of the diet plan (once a month), and the frequency of exercise plan execution (such as 30 minutes a day) are clearly defined; in terms of staged assessments, a health assessment is conducted every 4-6 weeks, and the plan is adjusted according to the body's response to ensure the conditioning effect. At the same time, the user is reminded to keep warm during the conditioning period, avoid excessive fatigue, and maintain a good work and rest schedule and mentality.

[0041] The implementation details of the technical solution of this application are described in detail below:

[0042] Figure 1 FIG2 shows a flow chart of a method for recommending a health management plan based on tongue image monitoring according to an embodiment of the present application. Figure 1 As shown, the method for recommending a health management plan based on tongue image monitoring includes at least steps S110 to S150, which are described in detail as follows:

[0043] In step S110 , the user's physiological information and tongue image are acquired.

[0044] In an embodiment of the present application, the user's physiological information and tongue image can be obtained through a wearable device. Specifically, the sensors built into the wearable device, such as a heart rate sensor, accelerometer, and gyroscope, automatically perform self-tests and calibrations after the wearable device is turned on to ensure that data can be accurately collected. Initialization instructions are sent to each sensor through the control chip in the wearable device to set parameters such as the sensor's sampling frequency and data accuracy. For example, a heart rate sensor may be set to collect heart rate data several times per second to ensure that subtle changes in heart rate can be captured.

[0045] Optionally, a data buffer is created in the wearable device to temporarily store collected physiological information to prevent data loss. After initialization, the wearable device enters a standby state, ready to collect the user's physiological information at any time. When the user begins to move, the wearable device begins collecting physiological information in real time. The heart rate sensor detects pulse beats to obtain heart rate data, while the accelerometer and gyroscope are used to monitor the user's activity status, such as step count and movement trajectory.

[0046] Optionally, the pulse is measured by using the sensor in the wearable device using the absorbance change of hemoglobin in the blood vessels, and the measured pulse is marked to calculate the heart rate data and heartbeat interval.

[0047] Alternatively, wearable devices can use galvanic skin sensors to measure the skin's electrical properties, reflecting changes in sweat gland activity as a skin conductance response. For example, when a person is emotionally aroused or experiencing physical stress, sweat gland activity increases, resulting in a decrease in skin resistance and an increase in conductance, thereby assessing the user's emotional state or physical stress level.

[0048] Optionally, the wearable device reads and preprocesses the data from these sensors. This preprocessing includes operations such as data filtering and denoising. For example, for heart rate data, a digital filtering algorithm is used to remove interference caused by body movement, sensor noise, and other factors, resulting in a more accurate heart rate value. For motion data, the user's speed and direction of movement are calculated based on readings from the accelerometer and gyroscope. The collected timestamps are then associated with the physiological data to facilitate understanding of the data's temporal sequence during subsequent analysis.

[0049] Optionally, the wearable device establishes a connection with the user's phone or cloud server to facilitate subsequent transmission and storage of collected data. The collected physiological information is transmitted in real time to the wearable device's data buffer. When the buffer reaches a certain amount of data, the data is packaged and uploaded to the cloud server or the user's phone for further processing.

[0050] In addition to physiological information, wearable devices also capture images of the user's tongue. Following prompts, the user extends their tongue and places it in the capture area. The device's camera automatically activates and captures the tongue image. The wearable device controls camera parameters, such as exposure time and focal length, to ensure clear and accurate tongue images.

[0051] Optionally, after the image is captured, the tongue image is pre-processed, such as cropping, adjusting brightness and contrast, etc., to improve image quality. The tongue image is then associated with the physiological information data, for example, by using a timestamp to ensure that the two were captured at the same time.

[0052] Finally, the associated physiological information and tongue image data are uploaded to a cloud server or the user's mobile phone via wireless communication technologies such as Bluetooth and Wireless Fidelity (Wi-Fi). After receiving the data, the cloud server or mobile phone further stores and analyzes it, providing data support for subsequent emotional health management.

[0053] The above process acquires the user's physiological information and tongue images, which reflect some behavioral habits and physical conditions through tongue images, thus collecting data from two different dimensions: physiological and behavioral. Physiological information can reflect the body's functional status in real time, while tongue images contain rich information related to emotions in traditional medicine, providing a data foundation for subsequent comprehensive analysis of user emotions.

[0054] In step S120, the physiological information is preprocessed according to the type of the physiological information to generate a first feature vector.

[0055] In one embodiment of the present application, various types of collected physiological information are identified and pre-processed.

[0056] For example, a filtering algorithm is used to remove noise interference from the heart rate data, such as moving average filtering or median filtering, to smooth the data curve and eliminate abnormal fluctuations caused by slight body shaking or sensor instability. Then, the heart rate data is converted into representative features through feature extraction methods, such as calculating interval parameters to obtain heart rate variability indicators, forming heart rate characteristic data for subsequent analysis.

[0057] For example, outlier detection is performed on blood pressure data, and a statistical method is used to identify and eliminate blood pressure values ​​that are significantly deviated from the normal range. The remaining data is then normalized and mapped to a specific interval to form blood pressure characteristic data for subsequent analysis.

[0058] For example, for sleep duration data, according to the division of sleep stages, such as light sleep, deep sleep or rapid eye movement sleep, the proportion of the duration of each stage is counted to form sleep characteristic data for subsequent analysis.

[0059] Afterwards, the features obtained after preprocessing these different types of physiological information are further processed and analyzed to generate the first feature vector containing multi-dimensional physiological features, providing basic data for subsequent emotional state analysis.

[0060] like Figure 2 As shown, in one embodiment of the present application, the physiological information is preprocessed according to the type of the physiological information to generate a first feature vector, including:

[0061] S210, determining an interval parameter according to the heartbeat interval in the physiological information;

[0062] S220, determining a frequency domain power ratio according to the power spectrum density of the heart rate data of the physiological information;

[0063] S230, determining a skin conductance response frequency according to time intervals between adjacent skin conductance response events in the physiological information;

[0064] S240 , normalizing and combining the interval parameter, the frequency domain power ratio, and the skin conductance response frequency to generate a first eigenvector.

[0065] After acquiring heartbeat interval data from physiological information, it is typically stored as a time series. To calculate the interval parameter, the heartbeat interval sequence is first traversed, and the difference between each heartbeat interval and the average heartbeat interval is calculated. These differences are then squared and summed, and finally divided by the total number of heartbeat intervals minus one. The square root is then taken to obtain the interval parameter. The interval parameter reflects the degree of variability in heartbeat intervals and is an important indicator for assessing heart rate variability. By quantifying the interval parameter of the heartbeat interval, the state of the autonomic nervous system is reflected, thereby characterizing the user's emotional state. Specifically, higher values ​​of the calculated interval parameter indicate greater heart rate variability and a predominance of a parasympathetic nervous system-dominated relaxation state, indicating a state of mental relaxation. Conversely, lower values ​​indicate a more active sympathetic nervous system-dominated stress state, indicating a state of mental arousal.

[0066] For the heart rate data in physiological information, fast Fourier transform can be used for analysis to convert the time domain signal into frequency domain representation. In the frequency domain, different frequencies are identified. f The power spectral density of the components, especially the low frequencies (such as 0.04 Hz ≤ f <0.15 Hz, usually associated with sympathetic and parasympathetic activity) and high frequencies (e.g., 0.15 Hz ≤ f The power of the parasympathetic nervous system components (≤0.4 Hz, primarily related to respiration) is measured. The frequency domain power ratio is calculated by calculating the ratio of low-frequency power to high-frequency power, and this ratio reflects the balance of the autonomic nervous system.

[0067] Specifically, the frequency domain power ratio is determined according to the power spectrum density of the heart rate data of the physiological information. for:

[0068]

[0069] in, Indicates 0.04 Hz ≤ f Low-frequency power <0.15 Hz reflects the mixed regulation of sympathetic and parasympathetic nerves, such as; Indicates 0.15Hz≤ f High-frequency power ≤0.4 Hz, reflecting parasympathetic nervous activity, such as respiratory rhythm; represents the power spectral density of heart rate data, df Denotes a differential operation. The above calculation process quantifies the balance between the sympathetic and parasympathetic nervous systems. An increase in the frequency domain power ratio indicates sympathetic nervous system activity and a state of stress or anxiety. A decrease in the ratio indicates a dominant parasympathetic nervous system and a state of calm.

[0070] For the skin conductance response data in the physiological information, we identify events where the skin conductance level changes significantly (i.e., skin conductance response events), such as points where the conductance suddenly rises or falls. By traversing the timestamps of these skin conductance response events, we calculate the time intervals between adjacent skin conductance response events, and based on the length of these time intervals, we count the number of such events that occur per unit time as the skin conductance response frequency. Specifically, we calculate the total observation time. T The total number of such events divided by the total observation time T This indicator reflects the activity of skin conductance response and is closely related to psychological states such as emotional arousal and stress level.

[0071] Specifically, the skin conductance response frequency is determined based on the time interval between adjacent events of the skin conductance response in the physiological information. for:

[0072]

[0073] in, Represents the indicator function, the time interval between adjacent events when the skin conductance level changes significantly Take 1 when it is, otherwise take 0; represents the minimum interval threshold of skin conductance response events, k represents the index of skin conductance response events within time T, K Represents the total number of skin conductance response events within time T, T The above calculation process counts the number of skin conductance response events per unit time. A high frequency indicates sustained emotional arousal, such as long-term stress, while a low frequency reflects stable emotions.

[0074] Physiological signal feature extraction uses mathematical modeling and signal processing techniques to transform complex physiological data into quantifiable emotional indicators. It characterizes emotional states from the perspectives of time, frequency, and event frequency, providing highly discriminative input data for subsequent multimodal fusion and emotion classification.

[0075] After obtaining the interval parameters, frequency-domain power ratio, and skin conductance response frequency, these features are normalized. For example, each eigenvalue is subtracted from its minimum value and then divided by its range (maximum minus minimum), or Z-score normalization is used to ensure that each feature has zero mean and unit variance. The normalized eigenvalues ​​are combined to obtain the first eigenvector. This vector integrates multiple physiological information, including heart rate variability, autonomic nervous system balance, and skin conductance response, providing standardized input data for subsequent pattern recognition, classification, or regression analysis.

[0076] The above process preprocesses and generates the first eigenvector according to the type of physiological information. It can extract the most representative features for different types of physiological information. For example, features such as interval parameters, frequency domain power ratio of heart rate data, and skin conductance response frequency reflect the heart rhythm, nervous system activity, and skin stress response from different perspectives. These features are closely related to emotional state.

[0077] Furthermore, normalizing the extracted features eliminates the impact of varying dimensions and value ranges between features, making them comparable in subsequent analysis. These normalized features are combined to generate the first eigenvector, providing a unified data format for subsequent fusion with other modal features. Accurately extracted and processed physiological features can improve the accuracy of subsequent emotion classification models. As input to the model, these features can more comprehensively reflect the user's physiological state, leading to more accurate emotion prediction.

[0078] In step S130, the state of the tongue in the tongue image is monitored, and a second feature vector is generated according to the state of the tongue.

[0079] In one embodiment of the present application, the tongue image is preprocessed using image processing techniques, such as grayscale conversion and denoising, to enhance the contrast between the crack and surrounding tissue and reduce noise interference. An edge detection algorithm is then used to identify the crack edge in the image and separate the crack from the background.

[0080] After identifying the cracks, the distribution of the cracks is analyzed. The density of the cracks, that is, the sum of the lengths of the cracks per unit area, is calculated to reflect the density of the cracks. For the morphological complexity of the cracks, the contour features of the cracks, such as curvature and number of branches, are extracted, and these features are quantified through Fourier description to measure the complexity of the crack morphology. The spatial distribution of the cracks can also be analyzed to determine the position and direction of the cracks on the tongue surface, as well as the aggregation of cracks in different areas. Finally, the features such as crack density, morphological complexity and spatial distribution are quantified and encoded, arranged in order and integrated together to generate a second feature vector containing multiple features of the tongue cracks, providing key data for the subsequent emotional state assessment combined with physiological information.

[0081] In one embodiment of the present application, monitoring the tongue state in the tongue image and generating a second feature vector according to the tongue state includes:

[0082] Performing image recognition and segmentation on the tongue image to obtain a tongue area and a crack area;

[0083] Extracting features of the tongue area to obtain tongue color features and tongue shape features;

[0084] Extracting features from the crack region to obtain crack features;

[0085] The tongue color feature, the tongue shape feature, and the crack feature are combined to generate a second feature vector.

[0086] First, load a pre-trained deep learning model. This model is trained based on a large amount of labeled tongue image data and has powerful feature capture capabilities. After the tongue image is input, the deep learning model performs convolution, pooling and other operations on the image layer by layer to extract feature information at different levels in the tongue image. For tongue area recognition, the focus is on areas in the image with typical tongue texture, color distribution and other features. A classification algorithm is used to determine whether each pixel belongs to the tongue, thereby outlining the contour boundary of the tongue. For crack area detection, areas in the image where grayscale, gradient and other features have undergone sudden changes are captured. These areas often correspond to cracks on the tongue surface. After a series of complex calculations and analyses, the system can finally accurately segment the tongue image into the tongue area and the crack area, laying a solid foundation for subsequent feature extraction.

[0087] After completing tongue image segmentation, color space conversion technology is used to convert the tongue image from the common red, green, and blue color space to a color space more suitable for color analysis. Statistical analysis of the color values ​​of each pixel within the tongue region is performed to calculate statistical quantities such as the mean and variance of the tongue color. This information is used to describe the tongue's color characteristics, such as whether the tongue's primary hue is reddish, pale, or something else entirely. To extract the tongue's shape, an edge detection algorithm is used to accurately delineate the tongue's edges. Based on this outline data, geometric parameters such as the tongue's length, width, area, and circumference are calculated. The tongue's shape and contour are also analyzed for regularity and any significant concavities and convexities. These meticulous calculations and analyses enable comprehensive and accurate extraction of the tongue's color and shape characteristics, providing robust data support for subsequent analysis.

[0088] In one embodiment of the present application, feature extraction is performed on the crack region to obtain crack features, including:

[0089] identifying cracks in the tongue image;

[0090] Calculating the weighted sum of the pixel areas of the connected areas at the cracks, calculating the pixel area of ​​the entire tongue image, and dividing the weighted sum of the pixel areas of the connected areas at the cracks by the pixel area of ​​the entire tongue image to obtain the result as the crack density of the tongue image;

[0091] generating a binary image of the tongue image, overlaying a grid on the binary image, and determining the morphological complexity of the cracks based on the number of grids containing cracks;

[0092] Determining a first spatial distribution of the cracks based on the pixel areas of the connected regions at the cracks, and determining a second spatial distribution of all cracks in the entire tongue image based on the first spatial distribution of the cracks;

[0093] The crack density, the morphological complexity, and the second spatial distribution are combined to generate a crack feature.

[0094] In one embodiment of the present application, in the process of identifying cracks in the tongue image, the tongue image is loaded into memory and stored in the form of a pixel matrix, where each pixel contains information such as color and brightness. An edge detection algorithm is then used to calculate the gradient of each pixel in the tongue image, and the edge strength is determined by comparing the grayscale changes of the pixel with those of its neighboring pixels. For tongue images, the grayscale changes at the cracks are usually more obvious, and the edge detection algorithm is used to mark out these areas with significant grayscale changes. Then, the marked edge areas are binarized through threshold processing to distinguish the areas that may be cracks from the background.

[0095] Optionally, morphological operations such as dilation and erosion can be combined to remove some noise interference, fill small holes in the cracks, and make the outline of the cracks clearer and more complete, thereby accurately identifying the cracks in the tongue image.

[0096] To determine the crack density of a tongue image, the pixel areas of the connected regions at the cracks and the area of ​​the entire tongue image are first calculated. For the pixel areas of the connected regions at the cracks, a connected region labeling algorithm is used to iterate over each pixel in the image and label adjacent pixels of the same color as the same connected region. For example, the weighted sum of the pixel areas of the connected regions at the cracks is calculated by counting the number of pixels in each connected region. The area of ​​the entire tongue image can be determined by counting all pixels in the image. The total pixel area of ​​the connected regions at all cracks is then divided by the area of ​​the entire tongue image to obtain the crack density of the tongue image.

[0097] For example, consider a tongue with a first crack on the left side and a second crack on the right side. The first crack has N connected regions, with connected region N1 containing A1 pixels and connected region N2 containing A2 pixels. The sum of the pixel areas of the first crack is S1 = A1 + A2. The second crack has M connected regions, with connected region M1 containing B1 pixels and connected region M2 containing B2 pixels. The sum of the pixel areas of the second crack is S2 = B1 + B2. The sum of the pixel areas of the connected regions at the first and second cracks is then calculated as S1 + S2, and divided by the total area of ​​the tongue to obtain the crack density.

[0098] In addition, the weighted sum of the pixel areas of the connected areas at the cracks is calculated, and the pixel area of ​​the entire tongue image is calculated. The weighted sum of the pixel areas of the connected areas at the cracks is divided by the pixel area of ​​the entire tongue image, and the result is used as the crack density of the tongue image. for:

[0099]

[0100] in, represents the preset crack length threshold, Indicates cracks h length, h Indicates a crack mark. H Indicates the number of cracks, Indicates the h Crack connected region The sum of the pixel areas, Indicates the tongue area. represents the hyperbolic tangent function, which is used to calculate the contribution of cracks to emotional stress, such as long cracks ( ) contributes to emotional stress in a saturated manner, while short cracks are weighted proportionally.

[0101] Afterwards, based on the results of the previous crack identification, the tongue image is converted into a binary image with only two colors, black and white, where the crack area is black and the background is white. Next, a grid is overlaid on the binary image. The size and spacing of the grid can be adjusted according to actual needs. Each grid is traversed and the number of cracks contained in the grid is counted. For grids containing more cracks, it means that the crack morphology in this area is more complex; while for grids containing fewer cracks, the crack morphology is relatively simple. By counting and analyzing the number of cracks in all grids, some quantitative indicators such as the variance of the number of cracks and the distribution density are used to determine the morphological complexity of the cracks.

[0102] Specifically, a binary image of the tongue image is generated, a grid is overlaid on the binary image, and the number of grids containing cracks is counted. , according to the number of grids containing cracks, determine the complexity of the crack shape for:

[0103]

[0104] in, Indicates the i The side length of the grid, Indicates the total number of grids, It represents the mean of the natural logarithm of each grid side length; Indicates that the side length is The number of grids covering the crack in the grid, represents the mean of the natural logarithm of the number of cracks in each grid, Represents the natural logarithm.

[0105] The above calculations are used to convert the complexity of crack morphology into quantifiable values, which are then mapped to the interval through normalization. , and obtain the morphological complexity. The higher the value of morphological complexity, the more complex the crack morphology. For example, This suggests a syndrome of Yin deficiency and hyperactivity of fire, which is related to microcirculation disorders caused by long-term emotional stress.

[0106] When determining the spatial distribution of the cracks, each identified crack connection area is analyzed. First, the center of mass position of each connection area is recorded. The center of mass can be obtained by calculating the average value of all pixel coordinates in the connection area. Then, based on the position information of each center of mass, the position of each crack on the tongue image is analyzed, including which area of ​​the tongue image the cracks are mainly concentrated in, such as the tip of the tongue, the middle of the tongue, the root of the tongue, etc., as well as the relative position relationship between the cracks. Through these analyses, the spatial position characteristics of each crack can be determined, thereby obtaining the first spatial distribution of each crack. Afterwards, based on the spatial position characteristics of each crack, it is mapped to the entire tongue image to determine the spatial distribution of the cracks in the entire tongue image, so as to more comprehensively describe the spatial distribution of the cracks on the tongue image.

[0107] Specifically, the first spatial distribution of the crack is determined based on the pixel area of ​​the connected area at the crack, and the second spatial distribution of the entire tongue crack is determined based on the first spatial distribution of the crack. for:

[0108]

[0109] in, Z represents the normalization factor, represents the first spatial distribution of cracks; represents the gradient operation on the first spatial distribution, represents the tongue area delineated by image recognition, x,y Represents the coordinates on the tongue.

[0110] After determining the crack density, morphological complexity, and secondary spatial distribution, the three previously calculated eigenvalues ​​are quantized and normalized. Quantization ensures that each eigenvalue has the same data type and precision, facilitating subsequent calculations and processing. Normalization maps the eigenvalues ​​to a specific range, such as [0, 1], to eliminate dimensional differences between different eigenvalues. The three normalized eigenvalues ​​are then arranged in a specific order to form a vector. This vector is the crack signature, which comprehensively reflects the multiple characteristic features of tongue cracks. Once generated, the crack signature can be used for subsequent tasks such as tongue image analysis and disease diagnosis, providing valuable insights for medical research and clinical practice.

[0111] The above process quantifies the abstract characteristic of tongue cracks by identifying cracks in tongue images and calculating features such as crack density, morphological complexity, and spatial distribution. Crack density reflects the overall distribution of cracks on the tongue surface, morphological complexity describes the crack shape, and the secondary spatial distribution reflects the crack location on the tongue surface. These quantitative features provide concrete data support for subsequent analysis. Tongue crack characteristics are associated with emotional states, and quantifying these features helps to uncover this correlation. For example, certain emotional states may lead to the appearance or change of tongue cracks, and quantitative analysis can better understand this inherent connection. Combining tongue color, shape, and crack features to generate a second feature vector comprehensively characterizes the characteristics of tongue cracks. This feature vector, representing the tongue image modality, provides effective data for subsequent fusion with physiological modal features.

[0112] In step S140 , cross-modal correlation features in the first feature vector and the second feature vector are extracted by using tensor decomposition technology, and are fused to generate a fused feature vector.

[0113] In this embodiment, the first eigenvector (containing physiological information features) and the second eigenvector (containing tongue crackle features) are first integrated to construct a three-dimensional tensor. Two dimensions correspond to different characteristic elements of the two eigenvectors, and the third dimension can represent different moments in the sample or different individuals. This constructed three-dimensional tensor is then decomposed using tensor decomposition techniques. Specifically, during the decomposition process, the parameters of the core tensor are iteratively adjusted to minimize the error between the decomposed tensor and the original tensor. This allows the core tensor to capture the cross-modal correlation information between the first and second eigenvectors. Cross-modal correlation features are extracted from the decomposition results. These correlation features reflect the intrinsic connection between physiological information and tongue crackle features. These extracted cross-modal correlation features are then fused using a fusion strategy, such as a weighted fusion strategy (which assigns weights to each correlation feature based on its importance and then adds the weighted features together). Finally, the cross-modal correlation features, or core tensors, are combined to generate a fused feature vector. This vector integrates the cross-modal features of physiological information and tongue crackle information, providing more robust data support for subsequent tasks such as emotional state analysis or disease diagnosis.

[0114] In one embodiment of the present application, extracting cross-modal correlation features from the first feature vector and the second feature vector by using a tensor decomposition technique, and fusing them to generate a fused feature vector includes:

[0115] The first eigenvector and the second eigenvector collected at the same time are used to construct a three-dimensional tensor according to the physiological, tongue image and time dimensions;

[0116] Decomposing the three-dimensional tensor, generating a core tensor based on the interactive information among the physiological, tongue image, and time dimensions as the cross-modal correlation feature;

[0117] The core tensors are combined according to physiological, tongue image and time dimensions to generate a fused feature vector.

[0118] In one embodiment of the present application, a chronological sequence of first and second feature vectors is first obtained. These feature vectors are typically acquired by collecting and analyzing the same object (e.g., a patient's physiological data and tongue image data) at different time points. Each time point is assigned an index to facilitate accurate identification during subsequent processing.

[0119] Afterwards, the size of the three-dimensional tensor is determined based on the length of the time series, the dimension of the first eigenvector, and the dimension of the second eigenvector. For example, if the time series contains T time points, the dimension of the first eigenvector is M , the dimension of the second eigenvector is N , then the size of the constructed three-dimensional tensor is M×N×TThen, in chronological order, the first eigenvector and the second eigenvector corresponding to each time point are filled into the corresponding positions of the three-dimensional tensor. Specifically, for each time point t , fill the components of the first eigenvector into the first t The first dimension of each time slice is filled with the components of the second eigenvector, and the second dimension of the time slice is filled with the components of the second eigenvector. Through this operation, the first and second eigenvectors are integrated into a three-dimensional tensor in time series, providing a unified data structure for subsequent analysis and processing.

[0120] When decomposing a three-dimensional tensor, the three-dimensional tensor is first subjected to singular value decomposition, and decomposition operations are performed on the physiological, tongue image, and time dimensions respectively. The three-dimensional tensor is decomposed into the product of a core tensor and multiple factor matrices. These factor matrices and core tensors are determined by iterative processing to minimize the reconstruction error, thereby achieving dimensionality reduction and feature extraction of the three-dimensional tensor. For example, a set of orthogonal basis vectors and corresponding singular values ​​are obtained in each dimension through Tucker decomposition. These orthogonal basis vectors represent the main characteristic patterns in that dimension, and the singular values ​​reflect the importance of these characteristic patterns. Then, important features are screened out according to the size of the singular values, and a core tensor composed of multiple features is constructed as the cross-modal correlation feature.

[0121] In this embodiment, the core tensor is typically smaller than the original 3D tensor, but it contains the most critical interaction information from the original tensor. Each element of the core tensor reflects the interaction between physiology, tongue image, and time. By analyzing the core tensor, we can gain a deeper understanding of the interaction mechanism between different modal data at different time points.

[0122] After obtaining the core tensor, the core tensors are combined according to the modal dimensions to generate a fused feature vector. Based on the structure of the core tensor and the definition of the modal dimensions, the modal dimensions correspond to different data modalities such as physiology and tongue image. The elements of the core tensor in different modal dimensions are arranged and combined in a specific order. For example, if the core tensor has sizes P, Q, and R in the physiology, tongue image, and time dimensions, respectively, the core tensors can be combined into a one-dimensional vector of length P × Q × R, or into other feature vectors based on actual needs. During the combination process, the interaction information and timing information between the different modalities in the core tensor are retained, allowing the fused feature vector to comprehensively reflect multiple aspects of characteristics such as physiology, tongue image, and time based on timing information.

[0123] The resulting fused feature vector is highly information-rich and can be used as input for subsequent machine learning models (such as classifiers and regression models) for tasks such as health status assessment. By analyzing and processing the fused feature vector, the potential relationships between data from different modalities can be more accurately discovered, providing strong support for medical research and clinical applications.

[0124] The above process integrates the first and second eigenvectors into a three-dimensional tensor. Using tensor decomposition, cross-modal correlation features are extracted, revealing potential correlations between physiological and tongue image information. Tensor decomposition can capture high-order interactions between features from different modalities, information that is often difficult to detect using single-modality analysis.

[0125] Experimental verification shows that the emotion recognition accuracy of this embodiment is 5%-8% higher than that of emotion recognition in a single mode. The fused feature vector contains richer information, more comprehensively reflecting the user's emotional state, and has higher accuracy and reliability than feature vectors from a single modality. The process of tensor decomposition and feature fusion reduces the dimensionality of the data to a certain extent, removing redundant information while retaining key features. This helps improve the training efficiency and generalization ability of subsequent emotion classification models.

[0126] In step S150, an emotion probability distribution is determined based on the fused feature vector, a body conditioning plan is generated according to the emotion probability distribution, and the body conditioning plan is sent to a user terminal.

[0127] In this example, the fused feature vector is input into a pre-trained emotion classification model built using a deep learning algorithm. Based on the cross-modal information from physiology and tongue imagery contained in the fused feature vector, the model calculates the probabilities corresponding to different emotion categories (such as anxiety, depression, calmness, or joy). During this calculation, mathematical operations and activation functions are used to transform and extract the feature vector layer by layer, ultimately outputting an emotion probability distribution vector, where each element represents the probability value of the corresponding emotion.

[0128] This probability distribution is then analyzed to identify the emotion category with the highest probability as the primary emotional tendency. Subsequently, a pre-defined rule base of physical conditioning solutions, built on extensive psychological research, clinical experience, and expert knowledge, is used to match the primary emotional tendency with a corresponding physical conditioning solution. For example, if the primary emotion is anxiety, the rule base can also link it to recommendations such as deep breathing relaxation techniques, meditation exercises, and moderate exercise.

[0129] In one embodiment of the present application, determining the emotion probability distribution based on the fused feature vector includes:

[0130] Initialize the parameters and loss function of the deep learning model;

[0131] The fused feature vector is input into the deep learning model, and the probability corresponding to each emotion category is output as the emotion probability distribution.

[0132] In one embodiment of the present application, in the process of determining the probability distribution of emotions based on fused feature vectors, data preparation and model initialization are first required. The previously generated fused feature vectors are used as input data. These fused feature vectors integrate multi-dimensional information such as physiology, tongue image, and time, and can fully reflect the individual's state. At the same time, a large number of data samples with emotion labels are collected. These data samples contain fused feature vectors under different emotional states and their corresponding emotion categories. For example, for emotions such as anxiety, depression, and calmness, there are corresponding feature vectors and labels.

[0133] Based on a deep learning model, predict the probability distribution of emotions. Initialize the selected model and set its parameters, such as weights and biases. Initialization can use methods such as random initialization or pre-trained initialization to ensure that the model has a reasonable parameter distribution at the beginning of training. Furthermore, define a loss function, such as the cross-entropy loss function, to measure the difference between the model's predicted probability distribution of emotions and the true emotion labels, providing an optimization target for subsequent model training.

[0134] After model training and parameter optimization, the trained model is used to determine the emotion probability distribution. The fused feature vector is input into the trained model, which calculates the probabilities of various emotions corresponding to the input based on the previously learned patterns and parameters. These probability values ​​are organized into an emotion probability distribution vector, where each element corresponds to the probability of an emotion. For example, for the three emotions of anxiety, depression, and calmness, the emotion probability distribution vector might be represented as [P(anxiety), P(depression), P(calm)], where P(anxiety), P(depression), and P(calm) are the model's predicted probabilities of the input belonging to the anxiety, depression, and calm emotions, respectively.

[0135] Finally, the determined emotion probability distribution is output. This output can be presented in various formats, such as text displaying the probability values ​​of various emotions or a visual chart displaying the emotion probability distribution. These results can provide important references for fields such as medical diagnosis and mental health assessment, helping doctors and researchers better understand an individual's emotional state.

[0136] The above process, by initializing the parameters and loss function of the deep learning model, provides a foundation for model learning. Deep learning models possess powerful nonlinear fitting capabilities, enabling them to learn the complex mapping relationship between fused feature vectors and emotion probability distributions from extensive data. By inputting the fused feature vectors into the deep learning model, the model outputs the probabilities corresponding to each emotion category, enabling accurate prediction of the user's emotional state. This probability distribution provides more detailed information, revealing not only the user's most likely emotion but also the likelihood of other emotions. Deep learning models can be adjusted and optimized based on different datasets and tasks, demonstrating their high adaptability and flexibility. Through continuous training and model improvement, the accuracy of emotion probability predictions can be improved.

[0137] In one embodiment of the present application, generating a body conditioning plan according to the emotion probability distribution and sending the body conditioning plan to a user terminal includes:

[0138] Obtaining the mapping rules between preset emotional probabilities and physical conditioning plans;

[0139] Determining the user's emotional tendency based on the probability value corresponding to each emotion in the emotional probability distribution;

[0140] The emotional tendency is matched with the mapping rule to generate a physical conditioning plan.

[0141] In this embodiment, a mapping rule between emotional probabilities and physical conditioning plans is pre-established. This rule is constructed based on a large amount of psychological research, clinical experience, and existing knowledge of emotional management. For example, if the probability of anxiety is high, the mapping rule can be associated with some methods to relieve anxiety, such as deep breathing exercises, meditation, and relaxing exercises; if the probability of depression is high, it is recommended to seek professional psychological counseling, increase social activities, and cultivate interests and hobbies. These mapping rules are stored in the database so that corresponding suggestions can be quickly found and matched based on the emotional probability distribution. At the same time, the mutual influence between different emotions is taken into account. For example, anxiety and depression often appear together. When generating suggestions, the situation of these two emotions is comprehensively considered to provide more comprehensive and targeted suggestions.

[0142] When generating a physical conditioning plan based on the emotion probability distribution, the first step is to read the probability values ​​corresponding to each emotion in the emotion probability distribution vector, such as the probability of anxiety, depression, anger, calmness, etc. By comparing the probabilities of different emotions, the individual's most dominant emotional tendency is determined.

[0143] Based on the probability values ​​of the main emotions in the emotion probability distribution, the corresponding suggestions are searched in the mapping rule database. For each main emotion, multiple relevant suggestions can be found.

[0144] Optionally, these recommendations can be filtered. This filtering is based on factors such as the applicability and effectiveness of the recommendations, as well as individual characteristics. For example, for individuals with specific medical conditions (such as heart disease), certain recommendations for anxiety relief, such as strenuous exercise, may not be appropriate and may be excluded.

[0145] Optionally, the effectiveness of suggestions can be considered, prioritizing those proven effective in extensive practice. Furthermore, individual characteristics such as age, gender, and lifestyle can be factored in to further filter out suggestions that best suit the individual's circumstances. For example, young people might be more likely to be recommended suggestions that utilize technology (such as emotion management apps), while older adults might be more suited to traditional social activities.

[0146] The above process provides a clear basis for generating physical conditioning plans by obtaining pre-defined mapping rules between emotion probabilities and physical conditioning plans. These rules are developed based on professional knowledge and experience, ensuring that the generated recommendations are scientific and reasonable. The probability values ​​corresponding to each emotion in the emotion probability distribution are read to determine the user's emotional tendencies, and then a targeted physical conditioning plan is generated based on the mapping rules. This personalized recommendation can better meet user needs and improve the effectiveness of emotion management. Recommendations generated based on clear mapping rules are logically clear and easy to understand, which can improve user acceptance and execution.

[0147] In one embodiment of the present application, determining an emotion probability distribution based on the fused feature vector, generating a body conditioning plan according to the emotion probability distribution, and sending the body conditioning plan to a user terminal includes:

[0148] generating a physical management report based on the emotion probability distribution and the physical conditioning plan;

[0149] The body management report is sent to a user terminal.

[0150] After matching and screening, a physical management report is generated. The selected suggestions are collated and organized, arranged in a logical order. The report details the content, implementation methods, and expected results of each suggestion. For example, a deep breathing exercise suggestion would include the specific breathing rhythm, duration, and frequency of practice, as well as the anxiety-relieving effects of consistent practice. Additionally, the physical management report may include dietary recommendations, exercise training plans, and more.

[0151] Optionally, the report can also provide some additional resources and support information, such as relevant books, websites, hotlines, etc., to help individuals obtain more emotion management knowledge and help.

[0152] Finally, the generated personalized health management report is output in a suitable format, such as a text file, PDF document, or pushed to the individual's user terminal via a mobile app. This allows individuals to obtain targeted health conditioning plans based on their emotional probability distribution, helping them better cope with their emotional problems.

[0153] Furthermore, the body management report can be pushed to the user's designated or associated receiving end, such as restaurants, physiotherapy shops, gyms, etc., to help users achieve more comprehensive physical health management.

[0154] For example, a 35-year-old male user has a history of anxiety, depression, and chronic insomnia. Data collected from a smart wearable device (bracelet) revealed an average heartbeat interval of 850ms, a heart rate power spectrum of 2.8 (normal: 0.5-2), and a skin conductance response peak interval of 5 seconds. Tongue images were also collected, and visual assessment revealed significant depression, cracks, and flaking of the tongue coating in the middle.

[0155] After preprocessing physiological signals, the first eigenvector [0.32, 3.2, 5.0] was generated; tongue image analysis generated the second eigenvector [15, 0.7, 4.2, 1.7]. The first and second eigenvectors for seven consecutive days were constructed into a three-dimensional tensor, from which a cross-modal fused eigenvector [0.45, −0.12, 1.32] was extracted, reflecting the combined state of "acute anxiety + chronic stress (accumulated cracks)." The fused eigenvector was input into the LSTM model, and the output P = [0.68, 0.25, 0.07], representing the scores for anxiety, depression, and calmness, respectively.

[0156] Later, real-time intervention was carried out. When a sudden drop in heart rate was detected, deep breathing training was pushed through the smart wearable APP. After the user performed it, the heart rate increased by 12% within 5 minutes. In terms of long-term conditioning, the traditional Chinese medicine prescription "Bupleurum Liver-Soothing Powder" was recommended based on the cracks in the tongue. After taking it for 2 weeks, the anxiety probability dropped to 2.1%. In terms of the accuracy of emotion recognition in the above process, compared with the clinical questionnaire, the accuracy rate of identifying the user's anxiety state reached 91%; in terms of the effectiveness of health advice, the user's physiological indicators, such as the frequency domain power ratio LF / HF, dropped from 2.8 to 1.5, and the crack density of the tongue image dropped from 1.7 to 1.3, with significant improvements in physiological indicators and tongue conditions.

[0157] The above process generates a body management report based on the probability distribution of emotions and the physical conditioning plan, presenting complex data and analysis results to the user in a clear and easy-to-understand manner. The report can integrate information such as emotional status, possible causes, and recommended measures to help users fully understand their emotional state. The body management report is sent to the user's terminal for convenient access anytime, anywhere. Users can view the report according to their needs and schedule and take appropriate emotional management measures in a timely manner. Personalized body management reports can guide users to pay attention to their emotional health and adopt effective management strategies, thereby promoting overall health and well-being.

[0158] The technical solution of the present application obtains the physiological information and tongue image of the user; pre-processes the physiological information according to the type of the physiological information to generate a first eigenvector; monitors the tongue state in the tongue image and generates a second eigenvector according to the tongue state; extracts cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology, and fuses them to generate a fused eigenvector; determines the emotion probability distribution based on the fused eigenvector, generates a body conditioning plan based on the emotion probability distribution, and sends the body conditioning plan to the user terminal. By fusing the physiological and tongue image information through multimodal feature extraction, the emotional state is accurately assessed based on cross-modal correlation feature mining, a personalized body management report is generated and the body conditioning plan is sent to the user terminal, thereby improving the accuracy and comprehensiveness of emotion recognition and management, and providing users with a scientific, accurate and convenient emotional health management plan.

[0159] The following describes a system embodiment of the present application, which can be used to implement the method for recommending a health management plan based on tongue image monitoring described in the aforementioned embodiment of the present application. It is understood that the system can be a computer program (including program code) running on a computer device, for example, a software application; the system can be used to execute the corresponding steps of the method provided in the embodiment of the present application. For details not disclosed in the system embodiment of the present application, please refer to the aforementioned embodiment of the method for recommending a health management plan based on tongue image monitoring.

[0160] Figure 3 A block diagram of a health management solution recommendation system based on tongue image monitoring according to an embodiment of the present application is shown.

[0161] Reference Figure 3 As shown, according to one embodiment of the present application, a system for recommending a health management plan based on tongue image monitoring includes:

[0162] An acquisition unit 310 is configured to acquire physiological information and a tongue image of a user;

[0163] A first feature unit 320 is configured to pre-process the physiological information according to the type of the physiological information to generate a first feature vector;

[0164] A second feature unit 330 is configured to monitor the tongue state in the tongue image and generate a second feature vector according to the tongue state;

[0165] a correlation unit 340 configured to extract cross-modal correlation features from the first feature vector and the second feature vector using a tensor decomposition technique, and fuse them to generate a fused feature vector;

[0166] The management unit 350 is configured to determine an emotion probability distribution based on the fused feature vector, generate a body conditioning plan according to the emotion probability distribution, and send the body conditioning plan to the user terminal.

[0167] In the present application, based on the aforementioned scheme, the physiological information is preprocessed separately according to the type of the physiological information to generate a first eigenvector, including: determining the interval parameters according to the heartbeat interval in the physiological information; determining the frequency domain power ratio according to the power spectral density of the heart rate data in the physiological information; determining the skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; normalizing and combining the interval parameters, the frequency domain power ratio and the skin conductance response frequency to generate a first eigenvector.

[0168] In the present application, based on the aforementioned scheme, the monitoring of the tongue state in the tongue image and the generation of a second feature vector according to the tongue state include: performing image recognition and segmentation on the tongue image to obtain the tongue area and the crack area; performing feature extraction on the tongue area to obtain tongue color features and tongue shape features; performing feature extraction on the crack area to obtain crack features; and combining the tongue color features, the tongue shape features and the crack features to generate a second feature vector.

[0169] In the present application, based on the above-mentioned scheme, the feature extraction of the crack area to obtain the crack feature includes: identifying the cracks in the tongue image; calculating the weighted sum of the pixel areas of the connected areas at the cracks, calculating the pixel area of ​​the entire tongue image, dividing the weighted sum of the pixel areas of the connected areas at the cracks by the pixel area of ​​the entire tongue image, and obtaining the result as the crack density of the tongue image; generating a binary image of the tongue image, overlaying a grid on the binary image, and determining the morphological complexity of the cracks according to the number of grids containing cracks; determining the first spatial distribution of the cracks according to the pixel area of ​​the connected areas at the cracks, and determining the second spatial distribution of all cracks in the entire tongue image according to the first spatial distribution; combining the crack density, the morphological complexity and the second spatial distribution to generate a crack feature.

[0170] In the present application, based on the aforementioned scheme, the cross-modal correlation features in the first eigenvector and the second eigenvector are extracted by tensor decomposition technology, and fused to generate a fused feature vector, including: constructing a three-dimensional tensor according to the physiological, tongue image and time dimensions of the first eigenvector and the second eigenvector collected at the same time; decomposing the three-dimensional tensor, and generating a core tensor based on the interactive information between the physiological, tongue image and time dimensions as the cross-modal correlation feature; combining the core tensor according to the physiological, tongue image and time dimensions to generate a fused feature vector.

[0171] In the present application, based on the aforementioned scheme, the determination of the emotion probability distribution based on the fused feature vector includes: initializing the parameters and loss function of the deep learning model; inputting the fused feature vector into the deep learning model, and outputting the probability corresponding to each emotion category as the emotion probability distribution.

[0172] In the present application, based on the aforementioned scheme, a physical conditioning plan is generated according to the emotion probability distribution, and the physical conditioning plan is sent to the user terminal, including: obtaining a mapping rule between a preset emotion probability and a physical conditioning plan; determining the user's emotional tendency based on the probability value corresponding to each emotion in the emotion probability distribution; matching the emotional tendency and the mapping rule to generate a physical conditioning plan.

[0173] The technical solution of the present application obtains the physiological information and tongue image of the user; pre-processes the physiological information according to the type of the physiological information to generate a first eigenvector; monitors the tongue state in the tongue image and generates a second eigenvector according to the tongue state; extracts cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology, and fuses them to generate a fused eigenvector; determines the emotion probability distribution based on the fused eigenvector, generates a body conditioning plan based on the emotion probability distribution, and sends the body conditioning plan to the user terminal. By fusing the physiological and tongue image information through multimodal feature extraction, the emotional state is accurately assessed based on cross-modal correlation feature mining, a personalized body management report is generated and the body conditioning plan is sent to the user terminal, thereby improving the accuracy and comprehensiveness of emotion recognition and management, and providing users with a scientific, accurate and convenient emotional health management plan.

[0174] Figure 4 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0175] It should be noted that the computer system of the electronic device in this embodiment is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0176] In this embodiment, the computer system includes a central processing unit (CPU) 401, which can execute various appropriate actions and processes based on programs stored in a read-only memory (ROM) 402 or programs loaded from a storage unit 408 into a random access memory (RAM) 403. For example, this can include executing the method for recommending a health management plan based on tongue image monitoring described in the above embodiment. The RAM 403 also stores various programs and data required for system operation. The CPU 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output interface 405 is also connected to the bus 404.

[0177] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, mouse, and the like; an output section 407 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and speakers; a storage section 408 including devices such as a hard disk; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. Removable media 411, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 410 as needed, so that computer programs read from the removable media can be installed in the storage section 408 as needed.

[0178] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product that includes a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 409 and / or installed from a removable medium 411. When the computer program is executed by the central processing unit 401, the various functions defined in the system of the present application are performed.

[0179] It should be noted that the computer-readable medium described in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present application, a computer-readable signal medium may include a data signal transmitted in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0180] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0181] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0182] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0183] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method for recommending a health management plan based on tongue image monitoring described in the above embodiments.

[0184] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0185] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes a number of instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0186] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0187] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for recommending a health management plan based on tongue image monitoring, characterized in that: include: Obtaining the user's physiological information and tongue image; Preprocessing the physiological information according to the type of the physiological information to generate a first feature vector consisting of an interval parameter, a frequency domain power ratio, and a skin conductance response frequency; monitoring a tongue state in the tongue image, and generating a second feature vector consisting of a tongue color feature, a tongue shape feature, and a crack feature according to the tongue state; Extracting cross-modal correlation features from the first eigenvector and the second eigenvector by using tensor decomposition technology, and fusing them to generate a fused feature vector; determining an emotion probability distribution based on the fused feature vector, generating a body conditioning plan according to the emotion probability distribution, and sending the body conditioning plan to a user terminal; The extracting cross-modal correlation features from the first feature vector and the second feature vector by using tensor decomposition technology and fusing them to generate a fused feature vector includes: The first eigenvector and the second eigenvector collected at the same time are used to construct a three-dimensional tensor according to the physiological, tongue image and time dimensions; Decomposing the three-dimensional tensor, generating a core tensor based on the interactive information among the physiological, tongue image, and time dimensions as the cross-modal correlation feature; Combining the core tensors according to physiological, tongue image and time dimensions to generate a fused feature vector; The decomposition of the three-dimensional tensor and the generation of a core tensor as the cross-modal association feature based on the interactive information among the physiological, tongue image, and time dimensions include: Performing singular value decomposition on the three-dimensional tensor in physiological, tongue image and time dimensions respectively to obtain orthogonal basis vectors and corresponding singular values; According to the size of the singular value, a core tensor is selected from the orthogonal basis vectors as the cross-modal correlation feature; The step of determining the emotion probability distribution based on the fused feature vector includes: Initialize the parameters and loss function of the deep learning model; The fused feature vector is input into the deep learning model, and the probability corresponding to each emotion category is output as the emotion probability distribution.

2. The method for recommending a health management plan based on tongue image monitoring according to claim 1, characterized in that: Preprocessing the physiological information according to the type of the physiological information to generate a first feature vector consisting of an interval parameter, a frequency domain power ratio, and a skin conductance response frequency includes: Determining an interval parameter according to the heartbeat interval in the physiological information; determining a frequency domain power ratio according to the power spectral density of the heart rate data of the physiological information; determining a skin conductance response frequency according to time intervals between adjacent skin conductance response events in the physiological information; The interval parameter, the frequency domain power ratio, and the skin conductance response frequency are normalized and combined to generate the first feature vector.

3. The method for recommending a health management plan based on tongue image monitoring according to claim 1, characterized in that: The step of monitoring the tongue state in the tongue image and generating a second feature vector composed of tongue color features, tongue shape features, and crack features according to the tongue state includes: Performing image recognition and segmentation on the tongue image to obtain a tongue area and a crack area; Extracting features of the tongue area to obtain tongue color features and tongue shape features; Extracting features from the crack region to obtain crack features; The tongue color feature, the tongue shape feature, and the crack feature are combined to generate a second feature vector.

4. The method for recommending a health management plan based on tongue image monitoring according to claim 3, characterized in that: The extracting features of the crack region to obtain crack features includes: identifying cracks in the tongue image; Calculate the weighted sum of the pixel areas of the connected areas at the cracks and calculate the pixel area of ​​the entire tongue image; Dividing the weighted sum of the pixel areas of the connected areas at the crack by the pixel area of ​​the entire tongue image, and obtaining the result as the crack density of the tongue image; generating a binary image of the tongue image, overlaying a grid on the binary image, and determining the morphological complexity of the cracks based on the number of grids containing cracks; determining a first spatial distribution of the cracks based on the pixel areas of the connected regions at the cracks, and determining a second spatial distribution of all cracks in the entire tongue image based on the first spatial distribution; The crack density, the morphological complexity, and the second spatial distribution are combined to generate a crack feature.

5. The method for recommending a health management plan based on tongue image monitoring according to claim 1, characterized in that: Generating a body conditioning plan according to the emotion probability distribution and sending the body conditioning plan to a user terminal includes: Obtaining the mapping rules between preset emotional probabilities and physical conditioning plans; Determining the user's emotional tendency based on the probability value corresponding to each emotion in the emotional probability distribution; The emotional tendency is matched with the mapping rule to generate a physical conditioning plan.

6. A health management plan recommendation system based on tongue image monitoring, the system comprising a cloud device consisting of a cloud server and a local terminal device, characterized in that: include: an acquisition unit, configured to acquire physiological information and tongue images of a user; A first feature unit is configured to pre-process the physiological information according to the type of the physiological information, and generate a first feature vector consisting of an interval parameter, a frequency domain power ratio, and a skin conductance response frequency; a second feature unit, configured to monitor a tongue state in the tongue image and generate a second feature vector consisting of a tongue color feature, a tongue shape feature, and a crack feature according to the tongue state; a correlation unit, configured to extract cross-modal correlation features from the first eigenvector and the second eigenvector by using a tensor decomposition technique, and fuse them to generate a fused feature vector; a management unit, configured to determine an emotion probability distribution based on the fused feature vector, generate a body conditioning plan according to the emotion probability distribution, and send the body conditioning plan to a user terminal; The extracting cross-modal correlation features from the first feature vector and the second feature vector by using tensor decomposition technology and fusing them to generate a fused feature vector includes: The first eigenvector and the second eigenvector collected at the same time are used to construct a three-dimensional tensor according to the physiological, tongue image and time dimensions; Decomposing the three-dimensional tensor, generating a core tensor based on the interactive information among the physiological, tongue image, and time dimensions as the cross-modal correlation feature; Combining the core tensors according to physiological, tongue image and time dimensions to generate a fused feature vector; The decomposition of the three-dimensional tensor and the generation of a core tensor as the cross-modal association feature based on the interactive information among the physiological, tongue image, and time dimensions include: Performing singular value decomposition on the three-dimensional tensor in physiological, tongue image and time dimensions respectively to obtain orthogonal basis vectors and corresponding singular values; According to the size of the singular value, a core tensor is selected from the orthogonal basis vectors as the cross-modal correlation feature; The step of determining the emotion probability distribution based on the fused feature vector includes: Initialize the parameters and loss function of the deep learning model; The fused feature vector is input into the deep learning model, and the probability corresponding to each emotion category is output as the emotion probability distribution.

7. A terminal device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method for recommending a physical health management plan based on tongue image monitoring as described in any one of claims 1 to 5.

Images