Body health management scheme recommendation method, system and equipment based on tongue picture monitoring

Abstract

The invention relates to the technical field of diagnosis management, and provides a tongue picture monitoring-based body health management scheme recommendation method, system and device. Acquiring physiological information and a tongue image of a user; the physiological information is preprocessed according to the type of the physiological information, and a first feature vector is generated; monitoring a tongue body state in the tongue body image, and generating a second feature vector according to the tongue body state; extracting cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition, and fusing the cross-modal correlation features to generate a fused feature vector; and determining emotion probability distribution based on the fused feature vector, generating a body conditioning scheme according to the emotion probability distribution, and sending the body conditioning scheme to the user terminal. Physiological and tongue picture information is fused through multi-modal feature extraction, association between tongue picture crack features and emotional states is mined based on cross-modal association features, so that the emotional states are accurately evaluated, and the accuracy and comprehensiveness of emotion recognition and management are improved.

Description

Technical Field

[0001] This application relates to the technical field of diagnostic management. Specifically, it relates to a method, system, and device for recommending a physical health management solution based on tongue image monitoring. Background Art

[0002] With the accelerating pace of modern life and the increasing attention people pay to health, emotional health management has become an increasingly important field. Smart wearable devices and tongue image monitoring technology provide new methods and means for emotional health management. Smart wearable devices such as smart watches and smart bracelets have been widely used in daily life. They can real-time monitor users' physiological data, such as heart rate, blood pressure, sleep, etc. These data are closely related to the body state and provide important reference for physical health management. Tongue image monitoring has a long history in traditional Chinese medicine. By observing the tongue image, the health status of the human body can be judged. For example, in the prior art "Clinical Research on the Characteristics of Cracked Tongue and Its Correlation with Syndromes", 249 outpatients and inpatients with cracked tongue in the internal medicine department were observed. The research shows that the occurrence of cracked tongue is commonly seen in the syndrome type of yin deficiency, and yin deficiency in traditional Chinese medicine is often related to factors such as emotional discomfort and mental stress. For example, the function of liver qi dispersing directly affects the qi and blood circulation. Patients with liver qi stagnation often have obvious purple stasis and cracks on the edges of the tongue; long-term emotional tension or excessive thinking easily leads to hyperactivity of heart fire, with a red tip of the tongue and cracks mostly concentrated on the tip of the tongue.

[0003] In practical applications, the physiological data collected by smart wearable devices and the tongue image monitoring data belong to different modal data. Most of the current emotional recognition methods are based on limited sample data and simple models, and it is difficult to accurately recognize complex emotional states and body states, and their accuracy and comprehensiveness are difficult to guarantee. Summary of the Invention

[0004] This application provides a method, system, and device for recommending a physical health management solution based on tongue image monitoring, which can at least to some extent solve the problem of relatively low accuracy and comprehensiveness of emotional recognition methods.

[0005] Other features and advantages of this application will become apparent through the following detailed description, or be learned in part through the practice of this application.

[0006] According to an aspect of the present application, there is provided a method for recommending a physical health management plan based on tongue image monitoring, including: obtaining physiological information and tongue body images of a user; preprocessing the physiological information respectively according to the type of the physiological information to generate a first feature vector; monitoring the tongue body state in the tongue body image, and generating a second feature vector according to the tongue body state; extracting cross-modal correlation features from the first feature vector and the second feature vector through tensor decomposition technology, and fusing them to generate a fused feature vector; determining an emotion probability distribution based on the fused feature vector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to a user terminal.

[0007] In the present application, based on the foregoing solution, the step of preprocessing the physiological information respectively according to the type of the physiological information to generate a first feature vector includes: determining an interphase parameter according to the inter-beat interval in the physiological information; determining a frequency domain power ratio according to the power spectral density of the heart rate data in the physiological information; determining a skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; normalizing and combining the interphase parameter, the frequency domain power ratio, and the skin conductance response frequency to generate a first feature vector.

[0008] In the present application, based on the foregoing solution, the step of monitoring the tongue body state in the tongue body image and generating a second feature vector according to the tongue body state includes: performing image recognition and segmentation on the tongue body image to obtain a tongue body region and a crack region; extracting features from the tongue body region to obtain a tongue body color feature and a tongue body shape feature; extracting features from the crack region to obtain a crack feature; combining the tongue body color feature, the tongue body shape feature, and the crack feature to generate a second feature vector.

[0009] In the present application, based on the foregoing solution, the step of extracting features from the crack region to obtain a crack feature includes: identifying cracks in the tongue body image; calculating the weighted sum of the pixel areas of the connected regions at the cracks, calculating the pixel area of the entire tongue body image, and dividing the weighted sum of the pixel areas of the connected regions at the cracks by the pixel area of the entire tongue body image, and using the obtained result as the crack density of the tongue image; generating a binary image of the tongue body image, covering a grid on the binary image, and determining the morphological complexity of the cracks according to the number of grids containing cracks; determining a first spatial distribution of the cracks at the cracks according to the pixel area of the connected regions at the cracks, and determining a second spatial distribution of all cracks in the entire tongue image according to the first spatial distribution; combining the crack density, the morphological complexity, and the second spatial distribution to generate a crack feature.

[0010] In this application, based on the foregoing solution, extracting cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology and fusing them to generate a fused eigenvector includes: constructing a three-dimensional tensor from the first eigenvector and the second eigenvector collected at the same time in terms of physiology, tongue image, and time dimensions; decomposing the three-dimensional tensor, generating a core tensor based on the interaction information among the physiology, tongue image, and time dimensions as the cross-modal correlation features; and combining the core tensor in terms of physiology, tongue image, and time dimensions to generate a fused eigenvector.

[0011] In this application, based on the foregoing solution, determining the emotion probability distribution based on the fused eigenvector includes: initializing the parameters and loss function of the deep learning model; and inputting the fused eigenvector into the deep learning model to output the probabilities corresponding to each emotion category as the emotion probability distribution.

[0012] In this application, based on the foregoing solution, generating a physical conditioning plan according to the emotion probability distribution and sending the physical conditioning plan to the user terminal includes: obtaining the mapping rule between the preset emotion probability and the physical conditioning plan; determining the emotion tendency of the user based on the probability values corresponding to each emotion in the emotion probability distribution; and matching the emotion tendency with the mapping rule to generate a physical conditioning plan.

[0013] In this application, based on the foregoing solution, after determining the emotion probability distribution based on the fused eigenvector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to the user terminal, it further includes: generating a physical management report based on the emotion probability distribution and the physical conditioning plan; and sending the physical management report to the user terminal.

[0014] According to one aspect of this application, a recommendation system for a physical health management plan based on tongue image monitoring is provided, including: An acquisition unit for acquiring the physiological information and tongue body image of the user; A first feature unit for preprocessing the physiological information respectively according to the type of the physiological information to generate a first eigenvector; A second feature unit for monitoring the tongue body state in the tongue body image and generating a second eigenvector according to the tongue body state; An association unit for extracting cross-modal correlation features in the first eigenvector and the second eigenvector through tensor decomposition technology and fusing them to generate a fused eigenvector; A management unit for determining the emotion probability distribution based on the fused eigenvector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to the user terminal.

[0015] According to one aspect of the present application, there is provided a computer-readable medium having a computer program stored thereon, and when the computer program is executed by a processor, it implements the method for recommending a physical health management solution based on tongue image monitoring as described in the above embodiments.

[0016] According to one aspect of the present application, there is provided an electronic device, including: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method for recommending a physical health management solution based on tongue image monitoring as described in the above embodiments.

[0017] According to one aspect of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for recommending a physical health management solution based on tongue image monitoring provided in the above various alternative implementations.

[0018] The technical solution of the present application obtains the physiological information and tongue body image of a user; preprocesses the physiological information respectively according to the type of the physiological information to generate a first feature vector; monitors the tongue body state in the tongue body image and generates a second feature vector according to the tongue body state; extracts the cross-modal association features in the first feature vector and the second feature vector through tensor decomposition technology, and fuses them to generate a fused feature vector; determines the emotional probability distribution based on the fused feature vector, generates a physical conditioning plan according to the emotional probability distribution, and sends the physical conditioning plan to a user terminal. By fusing physiological and tongue image information through multi-modal feature extraction, and mining the association between tongue image crack features and emotional states based on cross-modal association features, the emotional state is accurately evaluated, a personalized physical management report is generated, and the physical conditioning plan is sent to the user terminal, improving the accuracy and comprehensiveness of emotion recognition and management, and providing a scientific, accurate and convenient emotion health management solution for users.

[0019] It should be understood that the above general description and subsequent detailed description are only exemplary and explanatory, and cannot limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0021] Figure 1 Schematically shows a flowchart of a method for recommending a physical health management solution based on tongue image monitoring in an embodiment of the present application.

[0022] Figure 2 Schematically shows a flowchart of generating a first feature vector in an embodiment of the present application.

[0023] Figure 3 Schematically shows a schematic diagram of a physical health management solution recommendation system based on tongue image monitoring in an embodiment of the present application.

[0024] Figure 4 Shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing the embodiments of the present application. Detailed implementation manners

[0025] Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more complete and comprehensive, and will fully convey the concept of the example embodiments to those skilled in the art.

[0026] In addition, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a thorough understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, systems, steps, etc. can be adopted. In other cases, well-known methods, systems, implementations, or operations are not shown or described in detail to avoid obscuring aspects of the present application.

[0027] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0028] The flowcharts shown in the accompanying drawings are only illustrative and do not necessarily include all the content and operations / steps, nor do they necessarily need to be executed in the described order. For example, some operations / steps can be decomposed, while some operations / steps can be combined or partially combined. Therefore, the actual execution order may change according to the actual situation.

[0029] During the application process of the technical solution of this application, select to take a tongue image in the terminal software and fill in customer information; then take a tongue image through the user terminal; analyze the obtained tongue image, and the system automatically analyzes tongue image features such as tongue color, coating color, and tongue shape; generate a tongue image report containing health risk warnings in combination with physical constitution dialectics. Finally, according to the analysis results, provide personalized health solutions for users, covering traditional Chinese medicine physiotherapy (acupuncture, massage, etc.), dietary suggestions (medicated diet, tea drinks, etc.), exercise plans (Tai Chi, Baduanjin, etc.) and treatment schedules (frequency, cycle), and attach precautions and phased assessment reminders.

[0030] Exemplarily, in the application scenario of a traditional Chinese medicine clinic, the patient takes a tongue image on-site, and the system immediately generates a physical constitution dialectics report, matches conditioning plans such as acupuncture and medicated diet, and assists the doctor in formulating a personalized treatment course.

[0031] Exemplarily, in a physical examination and health management center, members regularly undergo tongue image detection, dynamically warn of cardiovascular / digestive risks, and adjust exercise plans and dietary cycles.

[0032] Exemplarily, individual users complete tongue diagnosis through terminal devices, and the AI generates sub-healthy conditioning plans such as shoulder and neck massage and music, with 2 physiotherapy suggestions per week.

[0033] Exemplarily, family users can take the tongue images of the whole family, and uniformly push food therapy, foot bath formulas and phased assessment reminders according to the analysis results of the tongue images of the whole family to prevent chronic diseases.

[0034] Specifically, in the generated health solutions, in this embodiment, they are classified in detail according to the health plans, including manual methods, instrumental methods, pharmaceutical methods, and mental methods. Specifically, manual methods (acupoint pressing, patting, massage and manipulation) regulate qi and blood through physical operations to relieve pain and fatigue; instrumental methods (acupuncture, scraping, cupping, medicated bath, etc.) use tools to dredge the meridians and treat cold-dampness, skin problems, etc.; pharmaceutical methods (herbal medicines, biological medicines, mineral medicines) combine medicines to regulate the functions of the zang-fu organs; mental methods (perception, meditation) integrate traditional theories with physical and mental regulation, analyze health trends and provide health preservation suggestions. Each type of method is marked with applicable symptoms such as insomnia, uterine cold, eczema, etc.

[0035] In addition, the embodiments of the present application also have the function of providing personalized treatment course suggestions based on the user's health status and tongue diagnosis results. This includes time suggestions, frequency suggestions, stage evaluations, and precautions. For example, in terms of the treatment course time, according to the disease type and severity, short-term (1-3 months) or long-term (more than 6 months) conditioning plans are recommended; in terms of frequency suggestions, the frequency of traditional Chinese medicine physiotherapy (such as 2 times a week), the adjustment cycle of the diet plan (once a month), and the execution frequency of the exercise plan (such as 30 minutes a day) are clarified; in terms of stage evaluations, a health assessment is carried out every 4-6 weeks, and the plan is adjusted according to the body's reaction to ensure the conditioning effect; at the same time, the user is reminded to keep warm, avoid overexertion, and maintain a good work and rest and mentality during the conditioning period.

[0036] The implementation details of the technical solution of the present application are elaborated in detail below: Figure 1 The flowchart of a method for recommending a physical health management plan based on tongue image monitoring according to an embodiment of the present application is shown. Refer to Figure 1 As shown, the method for recommending a physical health management plan based on tongue image monitoring at least includes steps S110 to S150, which are introduced in detail as follows: In step S110, the physiological information and tongue image of the user are obtained.

[0037] In the embodiments of the present application, the physiological information and tongue image of the user can be obtained through wearable devices. Specifically, the sensors built into the wearable device, such as a heart rate sensor, an accelerometer, and a gyroscope, automatically perform self-check and calibration after the wearable device is powered on to ensure accurate data collection. Initialization instructions are sent to each sensor through the control chip in the wearable device to set parameters such as the sampling frequency and data accuracy of the sensor. For example, the heart rate sensor may be set to collect a certain number of heart rate data per second to ensure that subtle changes in the heart rate can be captured.

[0038] Optionally, a data buffer is created in the wearable device to temporarily store the collected physiological information data to prevent data loss. After initialization is completed, the wearable device enters a standby state and is ready to collect the user's physiological information at any time. When the user starts to move, the wearable device starts to collect physiological information in real time. The heart rate sensor obtains heart rate data by detecting the pulsation of the pulse, and the accelerometer and gyroscope are used to monitor the user's motion state, such as the number of steps and the motion trajectory.

[0039] Optionally, the pulse is measured by using the change in the absorbance of hemoglobin in the blood in the blood vessel through the sensor in the wearable device, and the measured pulse is marked to calculate the heart rate data and the heart beat interval.

[0040] Optionally, the electrical properties of the skin are measured through a galvanic skin sensor in the wearable device to reflect changes in sweat gland activity, as the skin conductance response. For example, when the human body is subjected to emotional stimuli or physiological stress, the sweat gland activity increases, resulting in a decrease in skin resistance and an increase in conductance, thereby evaluating the user's emotional state or physiological stress level.

[0041] Optionally, the wearable device reads the data from these sensors and performs preprocessing. The preprocessing process includes operations such as data filtering and denoising. For example, for heart rate data, a digital filtering algorithm is used to remove interference signals caused by body movement, sensor noise, etc., to obtain a more accurate heart rate value. For motion data, the user's motion speed and direction are calculated based on the readings of the accelerometer and gyroscope. At the same time, the collected timestamps are associated with the physiological information data so that the time sequence of the data can be understood during subsequent analysis.

[0042] Optionally, the wearable device establishes a connection with the user's mobile phone or cloud server for subsequent transmission and storage of the collected data. The collected physiological information data is transmitted in real time to the data buffer of the wearable device. When the data in the buffer reaches a certain amount, these data are packed and prepared to be uploaded to the cloud server or the user's mobile phone for further processing.

[0043] In addition to physiological information, the wearable device also needs to collect the user's tongue image. The user extends the tongue and places it in the collection area according to the prompt, and the camera in the device can be automatically activated to take a tongue image. The wearable device controls the parameters of the camera, such as exposure time, focal length, etc., to ensure that a clear and accurate tongue image is captured.

[0044] Optionally, after shooting, preprocessing is performed on the tongue image, such as cropping, adjusting brightness and contrast, etc., to improve the quality of the image. Then, the tongue image is associated with the physiological information data, for example, through timestamps to ensure that the two are collected at the same time.

[0045] Finally, the associated physiological information and tongue image data are uploaded to the cloud server or the user's mobile phone through wireless communication technologies such as Bluetooth and Wireless Fidelity (Wi-Fi). After receiving the data, the cloud server or the mobile phone performs further storage and analysis to provide data support for subsequent emotional health management.

[0046] In the above process, the user's physiological information and tongue image are obtained, and part of the behavior habits and physical conditions are reflected through tongue images, realizing the collection of data from two different dimensions of physiology and behavior. Physiological information can reflect the state of the body's functions in real time, while the tongue image contains rich information related to emotions in traditional medicine, providing a data basis for subsequent comprehensive analysis of the user's emotions.

[0047] In step S120, according to the type of the physiological information, the physiological information is preprocessed separately to generate a first feature vector.

[0048] In an embodiment of the present application, various types of collected physiological information are identified and preprocessed.

[0049] Exemplarily, for heart rate data, a filtering algorithm is used to remove noise interference therein, such as moving average filtering or median filtering, to smooth the data curve and eliminate abnormal fluctuations caused by slight body shaking or sensor instability. Then, through a feature extraction method, the heart rate data is converted into representative features, such as calculating inter-beat interval parameters to obtain heart rate variability indexes, forming heart rate feature data for subsequent analysis.

[0050] Exemplarily, for blood pressure data, outlier detection is performed, and a statistics-based method is used to identify and remove blood pressure values that significantly deviate from the normal range. Then, the remaining data is normalized to map it to a specific interval, forming blood pressure feature data for subsequent analysis.

[0051] Exemplarily, for sleep duration data, according to the division of sleep stages, such as light sleep, deep sleep, or rapid eye movement sleep, the proportion of the duration of each stage is statistically calculated to form sleep feature data for subsequent analysis.

[0052] After that, the features obtained after preprocessing these different types of physiological information are further processed and analyzed to generate a first feature vector containing multi-dimensional physiological features, providing basic data for subsequent emotional state analysis.

[0053] As Figure 2 shown, in an embodiment of the present application, according to the type of the physiological information, the physiological information is preprocessed separately to generate a first feature vector, including: S210, determining an inter-beat interval parameter according to the inter-beat interval in the physiological information; S220, determining a frequency-domain power ratio according to the power spectral density of the heart rate data in the physiological information; S230, determining a skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; S240, normalizing and combining the inter-beat interval parameter, the frequency-domain power ratio, and the skin conductance response frequency to generate a first feature vector.

[0054] After obtaining the inter-beat interval data in the physiological information, the inter-beat interval data is usually stored in the form of a time series. To calculate the interval parameter, first traverse the inter-beat interval sequence, calculate the difference between each inter-beat interval and the average inter-beat interval, then square and sum these differences, and finally divide by the total number of inter-beat intervals minus one and take the square root to obtain the interval parameter. The interval parameter reflects the variability of the inter-beat interval and is an important indicator for evaluating heart rate variability. By quantifying the interval parameter of the inter-beat interval, the state of the autonomic nervous system is reflected to characterize the user's emotional state. Specifically, the higher the calculated interval parameter value, the greater the heart rate variability, and the dominant relaxation state dominated by the parasympathetic nerve indicates that the user is in a state of mental relaxation; on the contrary, the active stress state dominated by the sympathetic nerve indicates that the user is in a state of mental excitement.

[0055] For the heart rate data in the physiological information, fast Fourier transform can be used for analysis to convert the time-domain signal into a frequency-domain representation. In the frequency domain, the power spectral density of different frequency f components is identified, especially the low-frequency (such as 0.04 Hz ≤ f <0.15 Hz, which is usually related to sympathetic and parasympathetic nerve activities) and high-frequency (such as 0.15 Hz ≤ f ≤0.4 Hz, mainly related to the parasympathetic nerve activity related to breathing) components. The frequency-domain power ratio is obtained by calculating the ratio of the low-frequency power to the high-frequency power, and this ratio reflects the balance state of the autonomic nervous system.

[0056] Specifically, according to the power spectral density of the heart rate data in the physiological information, the frequency-domain power ratio is determined as: Where represents the low-frequency power of 0.04 Hz ≤ f <0.15 Hz, reflecting the mixed regulation of the sympathetic and parasympathetic nerves, such as; represents the high-frequency power of 0.15 Hz ≤ f ≤0.4 Hz, reflecting the parasympathetic nerve activity, such as the breathing rhythm; represents the power spectral density of the heart rate data, df represents the differential operation. The above calculation process can quantify the balance state of the sympathetic and parasympathetic nerves. When the frequency-domain power ratio increases, it indicates that the sympathetic nerve is active and the user is in a state of stress or anxiety; when the ratio decreases, the parasympathetic nerve is dominant and the user is in a calm state.

[0057] For the skin conductance response data in physiological information, identify events where the skin conductance level changes significantly (i.e., skin conductance response events), such as points where the conductance suddenly rises or suddenly drops. By traversing the timestamps of these skin conductance response events, calculate the time intervals between adjacent skin conductance response events, and based on the lengths of these time intervals, count the number of such events occurring per unit time as the skin conductance response frequency. Specifically, by calculating the total observation time T within which the total number of occurrences of such events is divided by the total observation time T to obtain. This indicator reflects the activity level of the skin conductance response and is closely related to psychological states such as emotional arousal and stress levels.

[0058] Specifically, based on the time intervals between adjacent events of the skin conductance response in physiological information, determine the skin conductance response frequency as: where, represents the indicator function, which takes 1 when the time interval between adjacent events where the skin conductance level changes significantly, and 0 otherwise; represents the minimum interval threshold of the skin conductance response event, k represents the index of the skin conductance response event within time T, K represents the total number of skin conductance response events within time T, T represents the total observation time. The above calculation process counts the number of skin conductance response events occurring per unit time. A high frequency indicates continuous emotional arousal, such as long-term stress, while a low frequency reflects emotional stability.

[0059] Physiological signal feature extraction uses mathematical modeling and signal processing techniques to transform complex physiological data into quantifiable emotional indicators. Characterize the emotional state from the perspectives of time domain, frequency domain, and event frequency respectively, providing highly discriminative input data for subsequent multimodal fusion and emotion classification.

[0060] After obtaining the interphase parameters, frequency domain power ratios, and skin conductance response frequencies, normalize these features. For example: subtract the minimum value of each feature value and then divide by the range of the feature value (maximum value minus minimum value), or use the Z-score normalization method to make each feature have zero mean and unit variance. Combine the normalized feature values to obtain the first feature vector. This vector synthesizes physiological information in multiple aspects such as heart rate variability, autonomic nervous system balance, and skin conductance response, providing standardized input data for subsequent pattern recognition, classification, or regression analysis.

[0061] In the above process, preprocessing is performed separately according to the type of physiological information and the first feature vector is generated, which can extract the most representative features for different types of physiological information. For example, features such as interphase parameters, frequency domain power ratio of heart rate data, and skin conductance response frequency reflect the rhythm of the heart, the activity of the nervous system, and the stress response of the skin from different angles, and these features are closely related to the emotional state.

[0062] On the other hand, normalizing the extracted features eliminates the influence caused by different dimensions and value ranges between different features, making each feature comparable in subsequent analysis. Combining these normalized features to generate the first feature vector provides a unified data format for subsequent fusion with other modality features. Accurately extracting and processing physiological features can improve the accuracy of the subsequent emotion classification model. These features, as the input of the model, can more comprehensively reflect the physiological state of the user, thus more accurately predicting emotions.

[0063] In step S130, monitor the tongue body state in the tongue body image, and generate a second feature vector according to the tongue body state.

[0064] In an embodiment of the present application, first, the tongue body image is preprocessed by image processing techniques, such as graying, denoising, etc., to enhance the contrast between the cracks and the surrounding tissues and reduce noise interference. Then, an edge detection algorithm is used to identify the crack edges in the image and separate the cracks from the background.

[0065] After identifying the cracks, analyze the distribution of the cracks. Calculate the crack density, that is, the total length of the cracks per unit area, to reflect the density of the cracks. For the morphological complexity of the cracks, extract the contour features of the cracks, such as curvature, number of branches, etc., and quantify these features through Fourier description to measure the complexity of the crack morphology. The spatial distribution of the cracks can also be analyzed to determine the position, orientation of the cracks on the tongue surface, and the aggregation of cracks in different regions. Finally, quantify and encode the features such as crack density, morphological complexity, and spatial distribution, and arrange and integrate them in order to generate a second feature vector containing various aspects of the tongue image cracks, providing key data for subsequent evaluation of the emotional state in combination with physiological information.

[0066] In an embodiment of the present application, monitoring the tongue body state in the tongue body image and generating a second feature vector according to the tongue body state includes: Performing image recognition and segmentation on the tongue body image to obtain the tongue body region and the crack region; Extracting features from the tongue body region to obtain tongue body color features and tongue body shape features; Extracting features from the crack region to obtain crack features; Combine the tongue body color feature, the tongue body shape feature, and the crack feature to generate a second feature vector.

[0067] First, load a pre-trained deep learning model. This model is trained based on a large number of labeled tongue body image data and has powerful feature capture capabilities. When a tongue body image is input, the image is subjected to operations such as convolution and pooling layer by layer through the deep learning model to extract feature information at different levels in the tongue body image. For the recognition of the tongue body area, it will focus on the area in the image with typical textures, color distributions, etc. of the tongue body, and use a classification algorithm to determine whether each pixel point belongs to the tongue body, thereby outlining the contour boundary of the tongue body. For the detection of the crack area, capture the areas in the image where features such as gray level and gradient change suddenly, and these areas often correspond to the cracks on the surface of the tongue body. After a series of complex calculations and analyses, the system can finally accurately segment the tongue body image into the tongue body area and the crack area, laying a solid foundation for subsequent feature extraction.

[0068] After the segmentation of the tongue body image is completed, use color space conversion technology to convert the tongue body image from the common red-green-blue color space to a color space more suitable for color analysis. By statistically analyzing the color values of each pixel point in the tongue body area, calculate statistical quantities such as the average value and variance of the tongue body color to describe the color feature of the tongue body, such as whether the main color tone of the tongue body is reddish, pale white, or other colors. In the extraction of the tongue body shape feature, use an edge detection algorithm to accurately depict the edge contour of the tongue body. Based on these contour data, calculate geometric parameters such as the length, width, area, and perimeter of the tongue body, and also analyze morphological features such as whether the shape contour of the tongue body is regular and whether there are obvious concave and convex changes. Through these detailed calculations and analyses, the color feature and shape feature of the tongue body can be comprehensively and accurately extracted, providing strong data support for subsequent analysis.

[0069] In one embodiment of the present application, feature extraction is performed on the crack area to obtain crack features, including: Identify the cracks in the tongue body image; Calculate the weighted sum of the pixel areas of the connected regions at the cracks, calculate the pixel area of the entire tongue body image, and divide the weighted sum of the pixel areas of the connected regions at the cracks by the pixel area of the entire tongue body image. The result obtained is used as the crack density of the tongue image; Generate a binary image of the tongue body image, overlay a grid on the binary image, and determine the morphological complexity of the cracks according to the number of grids containing cracks; Determine the first spatial distribution at the cracks according to the pixel area of the connected regions at the cracks, and determine the second spatial distribution of all the cracks in the entire tongue image according to the first spatial distribution at the cracks; Combining the crack density, the morphological complexity, and the second spatial distribution to generate crack features.

[0070] In an embodiment of the present application, during the process of identifying cracks in the tongue image, the tongue image is loaded into the memory and stored in the form of a pixel matrix. Each pixel contains information such as color and brightness. Then, an edge detection algorithm is used to calculate the gradient of each pixel in the tongue image, and the edge intensity is determined by comparing the gray-scale changes of the pixel with its neighboring pixels. For the tongue image, the gray-scale changes at the crack locations are usually more obvious, and these regions with significant gray-scale changes are marked out through the edge detection algorithm. Then, through threshold processing, the marked edge regions are binarized to distinguish the regions that may be cracks from the background.

[0071] Optionally, morphological operations such as dilation and erosion can also be combined to remove some noise interference, fill small holes in the cracks, and make the contour of the cracks clearer and more complete, so as to accurately identify the cracks in the tongue image.

[0072] When determining the crack density of the tongue image, first calculate the pixel area of the connected region at the crack and the area of the entire tongue image respectively. For the pixel area of the connected region at the crack, a connected region labeling algorithm is used to traverse each pixel in the image and label adjacent pixels with the same color as the same connected region. For example, then, by counting the number of pixels in each connected region, the weighted sum of the pixel areas of the connected regions at the crack is obtained. The area of the entire tongue image can be obtained by counting the number of all pixels in the image. Then, the sum of the pixel areas of all connected regions at the crack is divided by the area of the entire tongue image, and the result obtained is the crack density of the tongue image.

[0073] For example, the left side of the tongue has a first crack, and the right side of the tongue surface has a second crack. The first crack has N connected regions, the number of pixels in connected region N1 is A1, and the number of pixels in connected region N2 is A2. Then the sum of the pixel areas of the first crack is S1 = A1 + A2. The second crack has M connected regions, the number of pixels in connected region M1 is B1, and the number of pixels in connected region M2 is B2. The sum of the pixel areas of the second crack is S2 = B1 + B2. Then calculate the sum of the pixel areas of the connected regions at the first crack and the second crack as S1 + S2, and divide it by the area of the entire tongue image. The result obtained is the crack density of the tongue image.

[0074] In addition, calculate the weighted sum of the pixel areas of the connected regions at the crack, calculate the pixel area of the entire tongue image, and divide the weighted sum of the pixel areas of the connected regions at the crack by the pixel area of the entire tongue image. The result obtained is used as the crack density of the tongue image It is: Among them, represents a preset crack length threshold, represents the crack h length, h represents the identifier of the crack, H represents the number of cracks, represents the h th crack-connected region total pixel area, represents the tongue image area, represents the hyperbolic tangent function, which is used to calculate the contribution of cracks to emotional stress. For example, the contribution of long cracks ( ) to emotional stress is saturated, and short cracks are weighted proportionally.

[0075] After that, according to the results of previously identified cracks, the tongue image is converted into a binary image with only two colors, black and white, where the crack region is black and the background is white. Then, a grid is overlaid on the binary image, and the size and spacing of the grid can be adjusted according to actual needs. Each grid is traversed, and the number of cracks contained in the grid is counted. For grids with more cracks, it indicates that the crack morphology in this area is more complex; while grids with fewer cracks have a relatively simple crack morphology. By counting and analyzing the number of cracks in all grids, some quantitative indicators, such as the variance and distribution density of the number of cracks, are used to determine the morphological complexity of the cracks.

[0076] Specifically, generate the binary image of the tongue image, overlay a grid on the binary image, and count the number of grids containing cracks , and determine the morphological complexity of the cracks according to the number of grids containing cracks as: Among them, represents the i th side length of the grid, represents the total number of grids, represents the mean value of the natural logarithms of the side lengths of each grid; represents the side length of and the number of grids covering cracks in the grid, represents the mean value of the natural logarithms of the number of cracks in each grid, represents the natural logarithm.

[0077] By the above calculation, the crack morphological complexity is converted into a quantifiable value, and then through normalization processing, the value is mapped to the interval , to obtain the morphological complexity. The higher the value of the morphological complexity indicates that the crack morphology is more complex. Exemplarily, It indicates the syndrome of yin deficiency and excessive fire, which is related to the microcirculation disorder caused by long-term emotional stress.

[0078] When determining the spatial distribution at the crack, each identified crack-connected region is analyzed. First, the centroid position of each connected region is recorded. The centroid can be obtained by calculating the average value of all pixel coordinates within the connected region. Then, based on the position information of each centroid, the position of each crack on the tongue image is analyzed, including which region of the tongue image the cracks mainly concentrate on, such as the tip of the tongue, the middle of the tongue, the root of the tongue, etc., and the relative position relationship between the cracks. Through these analyses, the spatial position characteristics at each crack can be determined, thereby obtaining the first spatial distribution at each crack. Then, according to the spatial position characteristics at each crack, it is mapped onto the entire tongue image, so as to determine the spatial distribution of the cracks in the entire tongue image, in order to more comprehensively describe the spatial distribution of the cracks on the tongue image.

[0079] Specifically, the first spatial distribution at the crack is determined according to the pixel area of the connected region at the crack, and the second spatial distribution of the cracks in the entire tongue image is determined according to the first spatial distribution at the crack. is: where, Z represents the normalization factor, represents the first spatial distribution of the crack; represents the gradient operation on the first spatial distribution, represents the tongue region delimited by image recognition, x, y represents the coordinates on the tongue body.

[0080] After determining the crack density, morphological complexity, and the second spatial distribution, first, these three calculated characteristic values are quantified and standardized. Quantification is to ensure that each characteristic value has the same data type and precision, facilitating subsequent calculations and processing. Standardization is to map the characteristic values to a specific range, such as [0, 1], to eliminate the dimensional differences between different characteristic values. Then, these three standardized characteristic values are arranged together in a certain order to form a vector. This vector is the crack feature, which can comprehensively reflect multiple characteristic information of the cracks in the tongue image. After generating the crack feature, it can be used for subsequent tongue image analysis, disease diagnosis, etc., providing valuable references for medical research and clinical practice.

[0081] In the above process, by identifying the cracks in the tongue image and calculating features such as crack density, morphological complexity, and spatial distribution, the abstract feature of tongue image cracks is quantified. The crack density reflects the overall distribution of cracks on the tongue surface, the morphological complexity describes the shape characteristics of the cracks, and the second spatial distribution reflects the position information of the cracks on the tongue surface. These quantified features provide specific data support for subsequent analysis. There is a certain correlation between the tongue image crack features and the emotional state. Quantifying these features helps to explore this correlation. For example, certain emotional states may lead to the appearance or change of tongue image cracks, and through quantitative analysis, this internal connection can be better understood. Combining the tongue body color features, tongue body shape features, and crack features to generate a second feature vector can comprehensively represent the features of tongue image cracks. This feature vector, as the representative of the tongue image modality, provides effective data for subsequent fusion with physiological modality features.

[0082] In step S140, cross-modal association features in the first feature vector and the second feature vector are extracted through tensor decomposition technology, and fused to generate a fused feature vector.

[0083] In this embodiment, first, the first feature vector (including physiological information features) and the second feature vector (including tongue image crack features) are integrated to construct a three-dimensional tensor, where two dimensions respectively correspond to different feature elements of the two feature vectors, and the third dimension can represent different moments or different individuals of the samples, etc. Then, the constructed three-dimensional tensor is decomposed through tensor decomposition technology. Specifically, in the decomposition process, the parameters of the core tensor are continuously adjusted through iteration to minimize the error between the decomposed tensor and the original tensor, so as to capture the cross-modal association information between the first feature vector and the second feature vector through the core tensor. The cross-modal association features are extracted from the decomposition results, and these association features reflect the internal connection between physiological information and tongue image crack features. Then, a fusion strategy is adopted to fuse the extracted cross-modal association features, such as a weighted fusion strategy (assigning weights to each association feature according to the importance of the feature, and then adding the weighted features). Finally, the cross-modal association features, that is, the core tensor, are combined to generate a fused feature vector, which comprehensively combines the cross-modal features of physiological information and tongue image information, and provides more powerful data support for subsequent tasks such as emotional state analysis or disease diagnosis.

[0084] In an embodiment of the present application, extracting cross-modal association features in the first feature vector and the second feature vector through tensor decomposition technology and fusing them to generate a fused feature vector includes: Construct a three-dimensional tensor from the first feature vector and the second feature vector collected at the same time according to the physiological, tongue image, and time dimensions; Decompose the three-dimensional tensor, and generate a core tensor based on the interaction information between the physiological, tongue image, and time dimensions as the cross-modal correlation feature; Combine the core tensor according to the physiological, tongue image, and time dimensions to generate a fused feature vector.

[0085] In an embodiment of the present application, first obtain a first feature vector and a second feature vector sequence arranged in chronological order. These feature vectors are usually obtained by collecting and analyzing the same object (such as the physiological data and tongue image data of a patient) at different time points. Assign an index to each time point so that it can be accurately identified during subsequent processing.

[0086] After that, determine the size of the three-dimensional tensor according to the length of the time series, the dimension of the first feature vector, and the dimension of the second feature vector. For example, if the time series contains T time points, the dimension of the first feature vector is M , and the dimension of the second feature vector is N , then the size of the constructed three-dimensional tensor is M × N × T . Then, in chronological order, fill the first feature vector and the second feature vector corresponding to each time point into the corresponding positions of the three-dimensional tensor. Specifically, for each time point t , fill the components of the first feature vector into the first dimension of the t th time slice of the three-dimensional tensor, and fill the components of the second feature vector into the second dimension of this time slice. Through such an operation, the first feature vector and the second feature vector are integrated into a three-dimensional tensor according to the time series, providing a unified data structure for subsequent analysis and processing.

[0087] When decomposing the three-dimensional tensor, first perform a singular value decomposition on the three-dimensional tensor, and perform decomposition operations on the physiological, tongue image, and time dimensions respectively. Decompose the three-dimensional tensor into the product of a core tensor and multiple factor matrices. Determine these factor matrices and the core tensor by iterative processing to minimize the reconstruction error, thereby realizing the dimensionality reduction and feature extraction of the three-dimensional tensor. For example, through Tucker decomposition, a set of orthogonal basis vectors and corresponding singular values are obtained in each dimension. These orthogonal basis vectors represent the main feature patterns in that dimension, and the singular values reflect the importance of these feature patterns. Then, screen out important features according to the magnitude of the singular values, and construct a core tensor composed of multiple features as the cross-modal correlation feature.

[0088] In this embodiment, the size of the core tensor is usually smaller than that of the original three-dimensional tensor, but it contains the most critical interaction information in the original tensor. Each element of the core tensor reflects the interaction relationship between the physiological, tongue image, and time dimensions. By analyzing the core tensor, the interaction mechanism of different modality data at different time points can be deeply understood.

[0089] After obtaining the core tensor, the core tensor is combined according to the modality dimension to generate a fused feature vector. According to the structure of the core tensor and the definition of the modality dimension, the modality dimension corresponds to different data modalities such as physiology and tongue image. In a specific order, the elements of the core tensor on different modality dimensions are arranged and combined. For example, if the sizes of the core tensor in the physiological, tongue image, and time dimensions are P, Q, and R respectively, the core tensor can be combined into a one-dimensional vector with a length of P×Q×R, or combined into other forms of feature vectors according to actual needs. During the combination process, the interaction information and temporal information between different modalities in the core tensor are retained, so that the fused feature vector can comprehensively reflect the characteristics of multiple aspects such as physiology, tongue image, and time based on the temporal information.

[0090] The generated fused feature vector has a high information richness. It can be used as the input of subsequent machine learning models (such as classifiers, regression models, etc.) for tasks such as health status assessment. By analyzing and processing the fused feature vector, the potential relationship between different modality data can be more accurately mined, providing strong support for medical research and clinical applications.

[0091] In the above process, the first feature vector and the second feature vector are integrated into a three-dimensional tensor, and cross-modal correlation features are extracted through tensor decomposition, which can mine the potential correlation between physiological information and tongue image information. Tensor decomposition can capture the high-order interaction information between different modality features, which is often difficult to discover in single-modal analysis.

[0092] Experimental verification shows that the emotion recognition accuracy in this embodiment is improved by 5%-8% compared with emotion recognition in a single mode. The fused feature vector contains richer information and can more comprehensively reflect the user's emotional state. Compared with the feature vector of a single modality, it has higher accuracy and reliability. The process of tensor decomposition and feature fusion reduces the data dimension to a certain extent, removes redundant information, and retains key features at the same time. This helps to improve the training efficiency and generalization ability of subsequent emotion classification models.

[0093] In step S150, based on the fused feature vector, an emotion probability distribution is determined, a physical conditioning plan is generated according to the emotion probability distribution, and the physical conditioning plan is sent to the user terminal.

[0094] In this embodiment, the fused feature vector is input into a pre-trained emotion classification model, which is constructed based on a deep learning algorithm. The model calculates the probabilities corresponding to different emotion categories (such as anxiety, depression, calmness, or pleasure, etc.) according to the physiological and tongue image cross-modal information contained in the fused feature vector. During the calculation process, mathematical operations and activation functions are used to perform layer-by-layer transformation and extraction on the feature vector, and finally an emotion probability distribution vector is output, where each element represents the probability value of the corresponding emotion.

[0095] Then, this probability distribution is analyzed to determine the emotion category with the highest probability as the main emotion tendency. Subsequently, according to the pre-set rule library of physical conditioning plans, which is constructed based on a large amount of psychological research, clinical experience, and expert knowledge, the corresponding physical conditioning plan is matched according to the main emotion tendency. For example, if the main emotion is anxiety, the rule library can also be associated with suggestions such as deep breathing relaxation methods, meditation practices, and moderate exercise.

[0096] In an embodiment of the present application, determining the emotion probability distribution based on the fused feature vector includes: Initializing the parameters and loss function of the deep learning model; Inputting the fused feature vector into the deep learning model and outputting the probabilities corresponding to each emotion category as the emotion probability distribution.

[0097] In an embodiment of the present application, during the process of determining the emotion probability distribution based on the fused feature vector, data preparation and model initialization are first required. The previously generated fused feature vectors are used as input data, and these fused feature vectors synthesize multi-dimensional information such as physiology, tongue image, and time, and can comprehensively reflect the state of an individual. At the same time, a large number of data samples with emotion labels are collected, and these data samples contain the fused feature vectors in different emotion states and their corresponding emotion categories. For example, for emotions such as anxiety, depression, and calmness, there are corresponding feature vectors and labels.

[0098] Based on the deep learning model, the prediction of the emotion probability distribution is carried out. The selected model is initialized, and the parameters of the model, such as weights and biases, are set. The initialization can adopt methods such as random initialization or pre-trained initialization to ensure that the model has a reasonable parameter distribution at the beginning of training. In addition, a loss function, such as the cross-entropy loss function, is defined to measure the difference between the emotion probability distribution predicted by the model and the true emotion labels, providing an optimization goal for subsequent model training.

[0099] After model training and parameter optimization, the trained model is used to determine the emotion probability distribution. The fused feature vector is input into the trained model, and the model calculates the probabilities of various emotions corresponding to this input according to the patterns and parameters learned previously. These probability values are organized into an emotion probability distribution vector, where each element in the vector corresponds to the probability of an emotion. For example, for three emotions of anxiety, depression, and calmness, the emotion probability distribution vector may be expressed as [P(anxiety), P(depression), P(calmness)], where P(anxiety), P(depression), and P(calmness) are the probabilities that the model predicts this input belongs to the emotions of anxiety, depression, and calmness respectively.

[0100] Finally, the determined emotion probability distribution result is output, and the output result can be presented in various forms. For example, the probability values of various emotions are displayed in text form, or the emotion probability distribution is shown in the form of a visual chart. These results can provide important reference bases for fields such as medical diagnosis and mental health assessment, helping doctors and researchers better understand the emotional state of individuals.

[0101] The above process provides a basis for the learning of the model by initializing the parameters and loss function of the deep learning model. The deep learning model has a powerful non-linear fitting ability and can learn the complex mapping relationship between the fused feature vector and the emotion probability distribution through a large amount of data. Inputting the fused feature vector into the deep learning model and outputting the probabilities corresponding to each emotion category can accurately predict the emotional state of the user. The form of the probability distribution provides more detailed information, not only can we know the most likely emotion of the user, but also can understand the possibilities of other emotions. The deep learning model can be adjusted and optimized according to different data sets and tasks, and has strong adaptability and flexibility. By continuously training and improving the model, the accuracy of emotion probability prediction can be improved.

[0102] In an embodiment of the present application, generating a physical conditioning plan according to the emotion probability distribution and sending the physical conditioning plan to a user terminal includes: Obtaining the mapping rule between the preset emotion probability and the physical conditioning plan; Based on the probability values corresponding to each emotion in the emotion probability distribution, determining the emotion tendency of the user; Matching the emotion tendency and the mapping rule to generate a physical conditioning plan.

[0103] In this embodiment, a mapping rule between emotion probabilities and physical conditioning plans is established in advance. This rule is constructed based on a large number of psychological studies, clinical experiences, and existing emotion management knowledge. For example, if the probability of anxiety is relatively high, the mapping rule can be associated with some methods to relieve anxiety, such as deep breathing exercises, meditation, and engaging in relaxing exercises; if the probability of depression is relatively large, it is recommended to seek professional psychological counseling, increase social activities, and cultivate hobbies. These mapping rules are stored in a database for subsequent quick search and matching of corresponding suggestions according to the emotion probability distribution. At the same time, considering the mutual influence between different emotions, for example, anxiety and depression often occur together, the situations of these two emotions are comprehensively considered when generating suggestions to provide more comprehensive and targeted suggestions.

[0104] When generating a physical conditioning plan based on the emotion probability distribution, first read the probability values corresponding to each emotion in the emotion probability distribution vector, such as the probabilities of emotions like anxiety, depression, anger, calmness, etc. By comparing the probability magnitudes of different emotions, determine the most dominant emotion tendency of the current individual.

[0105] According to the probability value of the dominant emotion in the emotion probability distribution, search for the corresponding suggestions in the mapping rule database. For each dominant emotion, multiple relevant suggestions can be found.

[0106] Optionally, these suggestions can also be screened. The screening basis includes factors such as the applicability, effectiveness of the suggestions, and the characteristics of the individual. For example, for some individuals with special physical conditions (such as heart disease), certain suggestions for relieving anxiety through strenuous exercise may not be applicable, so they are excluded.

[0107] Optionally, the effectiveness of the suggestions can also be considered, and priority is given to selecting suggestions that have been verified through a large number of practices and have good effects. In addition, combined with the characteristics of the individual such as age, gender, and living habits, further screen out suggestions that are more in line with the actual situation of the individual. For example, for young people, suggestions that utilize technological means (such as emotion management APPs) may be more inclined to be recommended, while for the elderly, some traditional social activity suggestions may be more suitable.

[0108] The above process provides a clear basis for the generation of physical conditioning plans by obtaining the preset mapping rule between emotion probabilities and physical conditioning plans. These rules are formulated based on professional knowledge and experience, which can ensure that the generated suggestions are scientific and reasonable. Read the probability values corresponding to each emotion in the emotion probability distribution, determine the user's emotion tendency, and generate a targeted physical conditioning plan according to the mapping rule. This personalized suggestion can better meet the user's needs and improve the effect of emotion management. The suggestions generated based on clear mapping rules are logically clear, easy to understand, and can improve the user's acceptance and execution of the suggestions.

[0109] In one embodiment of the present application, determining an emotion probability distribution based on the fusion feature vector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to a user terminal, includes: Generating a physical management report based on the emotion probability distribution and the physical conditioning plan; Sending the physical management report to the user terminal.

[0110] After matching and screening, a physical management report is generated. The selected suggestions are sorted and organized, arranged in a certain logical order. In the report, the content, implementation method, and expected effect of each suggestion are elaborated in detail. For example, for the deep breathing exercise suggestion, specify the specific breathing rhythm, practice time, and frequency, as well as the effect of anxiety relief that may be brought about by persistent practice; in addition, the physical management report can also include dietary suggestions, exercise training programs, and so on.

[0111] Optionally, the report can also provide some additional resources and support information, such as relevant books, websites, hotline numbers, etc., to facilitate individuals to obtain more emotion management knowledge and assistance.

[0112] Finally, the generated personalized physical management report is output in a suitable form, such as a text file, PDF document, or pushed to the individual's user terminal through a mobile application. In this way, individuals can obtain a targeted physical conditioning plan based on their emotion probability distribution and better cope with their own emotion problems.

[0113] Furthermore, the physical management report can also be pushed to a receiving end designated or associated by the user, such as a restaurant, physiotherapy store, gym, etc., to help the user more comprehensively achieve physical health management.

[0114] Exemplarily, the user is a 35-year-old male with a history of anxiety disorder and depression and long-term insomnia. Through the data of a smart wearable device (a bracelet), the average inter-beat interval collected is 850 ms, the heart rate power spectrum is 2.8 (normal 0.5 - 2), and the peak interval of the skin conductance response is 5 seconds. At the same time, a tongue image is collected, and visually judged that the middle jiao of the tongue is significantly sunken, with cracks and exfoliated coating.

[0115] After preprocessing the physiological signals, the first feature vector [0.32, 3.2, 5.0] is generated; tongue image analysis generates the second feature vector [15, 0.7, 4.2, 1.7]. The first and second feature vectors of the user for 7 consecutive days are constructed into a three-dimensional tensor, from which a cross-modal fusion feature vector [0.45, -0.12, 1.32] is extracted, reflecting the composite state of "acute anxiety + chronic stress (crack accumulation)". The fusion feature vector is input into the LSTM model, and the output P = [0.68, 0.25, 0.07], representing the scores corresponding to anxiety, depression, and calmness respectively.

[0116] For real-time intervention in the later stage, when a sudden drop in heart rate is detected, deep breathing training is pushed through the smart wearable APP. After the user executes it, the heart rate increases by 12% within 5 minutes. In terms of long-term conditioning, according to the tongue image cracks, the traditional Chinese medicine formula "Chaihu Shugan Powder" is recommended. After taking it for 2 weeks, the probability of anxiety drops to 2.1%. In terms of the accuracy of emotion recognition in the above process, compared with the clinical questionnaire, the recognition accuracy of the user's anxiety state reaches 91%; in terms of the effectiveness of health advice, the user's physiological indicators, such as the frequency domain power ratio LF / HF, drops from 2.8 to 1.5, and the crack density of the tongue image drops from 1.7 to 1.3, showing significant improvement in physiological indicators and tongue image conditions.

[0117] In the above process, a body management report is generated based on the emotion probability distribution and the body conditioning plan, presenting complex data and analysis results to the user in a clear and understandable way. The report can integrate information such as the emotional state, possible reasons, and recommended measures to help the user comprehensively understand their emotional condition. The body management report is sent to the user's terminal for the user to access at any time and place. The user can view the report according to their own needs and time arrangements and take corresponding emotion management measures in a timely manner. The personalized body management report can guide the user to pay attention to their emotional health, adopt effective management strategies, and thus promote the improvement of the overall health level.

[0118] The technical solution of this application obtains the physiological information and tongue image of the user; preprocesses the physiological information respectively according to the type of the physiological information to generate a first feature vector; monitors the tongue state in the tongue image and generates a second feature vector according to the tongue state; extracts the cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition technology, and fuses them to generate a fused feature vector; determines the emotion probability distribution based on the fused feature vector, generates a physical conditioning plan according to the emotion probability distribution, and sends the physical conditioning plan to the user terminal. By fusing physiological and tongue image information through multi-modal feature extraction, mining cross-modal correlation features to accurately evaluate the emotional state, generating a personalized physical management report and sending the physical conditioning plan to the user terminal, it improves the accuracy and comprehensiveness of emotion recognition and management, and provides a scientific, accurate and convenient emotion health management plan for users.

[0119] The following introduces the system embodiments of this application, which can be used to execute the method for recommending a physical health management plan based on tongue image monitoring in the above embodiments of this application. It can be understood that the system can be a computer program (including program code) running in a computer device, for example, the system is an application software; the system can be used to execute the corresponding steps in the method provided in the embodiments of this application. For the details not disclosed in the system embodiments of this application, please refer to the embodiments of the method for recommending a physical health management plan based on tongue image monitoring in the above of this application.

[0120] Figure 3 The block diagram of a system for recommending a physical health management plan based on tongue image monitoring according to an embodiment of this application is shown.

[0121] Referring to Figure 3 As shown, a system for recommending a physical health management plan based on tongue image monitoring according to an embodiment of this application includes: An acquisition unit 310, configured to acquire the physiological information and tongue image of the user; A first feature unit 320, configured to preprocess the physiological information respectively according to the type of the physiological information to generate a first feature vector; A second feature unit 330, configured to monitor the tongue state in the tongue image and generate a second feature vector according to the tongue state; An association unit 340, configured to extract the cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition technology, and fuse them to generate a fused feature vector; A management unit 350, configured to determine the emotion probability distribution based on the fused feature vector, generate a physical conditioning plan according to the emotion probability distribution, and send the physical conditioning plan to the user terminal.

[0122] In this application, based on the foregoing solution, the preprocessing of the physiological information according to the type of the physiological information to generate a first feature vector includes: determining an interval parameter according to the inter-beat interval in the physiological information; determining a frequency-domain power ratio according to the power spectral density of the heart rate data in the physiological information; determining a skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; normalizing and combining the interval parameter, the frequency-domain power ratio, and the skin conductance response frequency to generate a first feature vector.

[0123] In this application, based on the foregoing solution, the monitoring of the tongue body state in the tongue body image and generating a second feature vector according to the tongue body state includes: performing image recognition and segmentation on the tongue body image to obtain a tongue body region and a crack region; extracting features from the tongue body region to obtain a tongue body color feature and a tongue body shape feature; extracting features from the crack region to obtain a crack feature; combining the tongue body color feature, the tongue body shape feature, and the crack feature to generate a second feature vector.

[0124] In this application, based on the foregoing solution, the extracting of the crack feature from the crack region includes: identifying cracks in the tongue body image; calculating the weighted sum of the pixel areas of the connected regions at the cracks, calculating the pixel area of the entire tongue body image, and dividing the weighted sum of the pixel areas of the connected regions at the cracks by the pixel area of the entire tongue body image, and using the obtained result as the crack density of the tongue image; generating a binary image of the tongue body image, covering a grid on the binary image, and determining the morphological complexity of the cracks according to the number of grids containing cracks; determining a first spatial distribution of the cracks at the cracks according to the pixel area of the connected regions at the cracks, and determining a second spatial distribution of all the cracks in the entire tongue image according to the first spatial distribution; combining the crack density, the morphological complexity, and the second spatial distribution to generate a crack feature.

[0125] In this application, based on the foregoing solution, the extracting of the cross-modal correlation features from the first feature vector and the second feature vector by tensor decomposition technology and fusing them to generate a fused feature vector includes: constructing a three-dimensional tensor from the first feature vector and the second feature vector collected at the same time according to the physiological, tongue image, and time dimensions; decomposing the three-dimensional tensor, and generating a core tensor based on the interaction information between the physiological, tongue image, and time dimensions as the cross-modal correlation feature; combining the core tensor according to the physiological, tongue image, and time dimensions to generate a fused feature vector.

[0126] In this application, based on the foregoing solution, determining the emotion probability distribution based on the fused feature vector includes: initializing the parameters and loss function of the deep learning model; inputting the fused feature vector into the deep learning model, and outputting the probabilities corresponding to each emotion category as the emotion probability distribution.

[0127] In this application, based on the foregoing solution, generating a physical conditioning plan according to the emotion probability distribution and sending the physical conditioning plan to the user terminal includes: obtaining the mapping rule between the preset emotion probability and the physical conditioning plan; determining the emotion tendency of the user based on the probability values corresponding to each emotion in the emotion probability distribution; and matching the emotion tendency and the mapping rule to generate a physical conditioning plan.

[0128] The technical solution of this application is to obtain the physiological information and tongue image of the user; preprocess the physiological information respectively according to the type of the physiological information to generate a first feature vector; monitor the tongue state in the tongue image and generate a second feature vector according to the tongue state; extract the cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition technology, and perform fusion to generate a fused feature vector; determine the emotion probability distribution based on the fused feature vector, generate a physical conditioning plan according to the emotion probability distribution, and send the physical conditioning plan to the user terminal. By fusing physiological and tongue image information through multi-modal feature extraction, mining cross-modal correlation features to accurately evaluate the emotional state, generating a personalized physical management report and sending the physical conditioning plan to the user terminal, the accuracy and comprehensiveness of emotion recognition and management are improved, and a scientific, accurate and convenient emotion health management plan is provided for users.

[0129] Figure 4 The structural schematic diagram of the computer system of the electronic device suitable for implementing the embodiments of this application is shown.

[0130] It should be noted that the computer system of the electronic device in this embodiment is only an example, and should not bring any limitation to the functions and usage scope of the embodiments of this application.

[0131] The computer system in this embodiment includes a central processing unit 401, which can perform various appropriate actions and processes according to the program stored in the read-only memory 402 or the program loaded from the storage part 408 to the random access memory 403, such as executing the method for recommending a physical health management plan based on tongue image monitoring described in the above embodiment. In the random access memory 403, various programs and data required for system operation are also stored. The central processing unit 401, the read-only memory 402, and the random access memory 403 are connected to each other through a bus 404. The input / output interface 405 is also connected to the bus 404.

[0132] The following components are connected to the input / output interface 405: an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc. and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the input / output interface 405 as needed. A removable medium 411, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is mounted on the drive 410 as needed so that a computer program read from it can be installed into the storage section 408 as needed.

[0133] In particular, according to an embodiment of the present application, the processes described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product that includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication section 409, and / or installed from the removable medium 411. When the computer program is executed by the central processing unit 401, various functions defined in the system of the present application are executed.

[0134] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium, a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of the computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device. In the present application, the computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable computer program. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium, and this computer-readable medium can send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of the present application. Among them, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that marked in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0136] The units involved in the embodiments described in this application can be implemented in software or in hardware, and the described units can also be provided in a processor. Among them, the names of these units do not, in some cases, constitute a limitation on the units themselves.

[0137] According to one aspect of the present application, there is provided a computer program product or a computer program, the computer program product or the computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above various alternative implementation manners.

[0138] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or may exist separately without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the one or more programs are executed by an electronic device, the electronic device implements the method for recommending a physical health management solution based on tongue image monitoring described in the above embodiments.

[0139] It should be noted that although several modules or units of a device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of the two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0140] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented in software or in a manner of software combined with necessary hardware. Therefore, the technical solutions according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (such as a personal computer, a server, a touch terminal, or a network device, etc.) to execute the methods according to the embodiments of the present application.

[0141] After considering the specification and practicing the disclosed embodiments herein, those skilled in the art will readily conceive of other embodiments of the present application. The present application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include common general knowledge or conventional technical means in the technical field not disclosed in the present application.

[0142] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the present application is only limited by the appended claims.

Claims

1. A method for recommending a physical health management plan based on tongue image monitoring, characterized in that, including: Obtaining the physiological information and tongue image of the user; Preprocessing the physiological information respectively according to the type of the physiological information to generate a first feature vector; Monitoring the tongue state in the tongue image and generating a second feature vector according to the tongue state; Extracting the cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition technology and fusing them to generate a fused feature vector; Determining an emotion probability distribution based on the fused feature vector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to the user terminal.

2. The method for recommending a physical health management plan based on tongue image monitoring according to claim 1, wherein The preprocessing the physiological information respectively according to the type of the physiological information to generate a first feature vector includes: Determining an interphase parameter according to the inter-beat interval in the physiological information; Determining a frequency-domain power ratio according to the power spectral density of the heart rate data in the physiological information; Determining the skin conductance response frequency according to the time interval between adjacent events of the skin conductance response in the physiological information; Normalizing and combining the interphase parameter, the frequency-domain power ratio, and the skin conductance response frequency to generate the first feature vector.

3. The method for recommending a physical health management plan based on tongue image monitoring according to claim 1, wherein The monitoring the tongue state in the tongue image and generating a second feature vector according to the tongue state includes: Performing image recognition and segmentation on the tongue image to obtain a tongue region and a crack region; Extracting features from the tongue region to obtain tongue color features and tongue shape features; Extracting features from the crack region to obtain crack features; Combining the tongue color features, the tongue shape features, and the crack features to generate a second feature vector.

4. The method for recommending a physical health management plan based on tongue image monitoring according to claim 3, wherein The extracting features from the crack region to obtain crack features includes: Identifying cracks in the tongue image; Calculating the weighted sum of the pixel areas of the connected regions at the cracks and calculating the pixel area of the entire tongue image; Dividing the weighted sum of the pixel areas of the connected regions at the cracks by the pixel area of the entire tongue image, and taking the result as the crack density of the tongue image; Generating a binary image of the tongue image, covering a grid on the binary image, and determining the morphological complexity of the cracks according to the number of grids containing cracks; Determining a first spatial distribution at the cracks according to the pixel area of the connected regions at the cracks, and determining a second spatial distribution of all cracks in the entire tongue image according to the first spatial distribution; Combining the crack density, the morphological complexity, and the second spatial distribution to generate crack features.

5. The method for recommending a physical health management plan based on tongue image monitoring according to claim 1, wherein The extracting the cross-modal correlation features in the first feature vector and the second feature vector through tensor decomposition technology and fusing them to generate a fused feature vector includes: Constructing a three-dimensional tensor from the first feature vector and the second feature vector collected at the same time according to the physiological, tongue image, and time dimensions; Decomposing the three-dimensional tensor and generating a core tensor based on the interaction information between the physiological, tongue image, and time dimensions as the cross-modal correlation features; Combining the core tensor according to the physiological, tongue image, and time dimensions to generate a fused feature vector.

6. The method for recommending a physical health management solution based on tongue image monitoring according to claim 5, wherein Decomposing the three-dimensional tensor to generate a core tensor based on the interaction information among the physiological, tongue image, and time dimensions as the cross-modal correlation feature includes: Performing singular value decomposition on the three-dimensional tensor in the physiological, tongue image, and time dimensions respectively to obtain orthogonal basis vectors and corresponding singular values; Screening out the core tensor from the orthogonal basis vectors as the cross-modal correlation feature according to the magnitudes of the singular values.

7. The method for recommending a physical health management plan based on tongue image monitoring according to claim 1, wherein Determining the emotion probability distribution based on the fused feature vector includes: Initializing the parameters and loss function of the deep learning model; Inputting the fused feature vector into the deep learning model to output the probabilities corresponding to each emotion category as the emotion probability distribution.

8. The method for recommending a physical health management plan based on tongue image monitoring according to claim 1, wherein Generating a physical conditioning plan according to the emotion probability distribution and sending the physical conditioning plan to the user terminal includes: Obtaining the mapping rule between the preset emotion probability and the physical conditioning plan; Determining the emotion tendency of the user based on the probability values corresponding to each emotion in the emotion probability distribution; Matching the emotion tendency with the mapping rule to generate a physical conditioning plan.

9. A health management plan recommendation system based on tongue image monitoring. The system includes a cloud device composed of a cloud server and a local terminal device, and is characterized in that, Includes: An acquisition unit for acquiring the physiological information and tongue image of the user; A first feature unit for preprocessing the physiological information respectively according to the type of the physiological information to generate a first feature vector; A second feature unit for monitoring the tongue state in the tongue image and generating a second feature vector according to the tongue state; An association unit for extracting the cross-modal correlation feature from the first feature vector and the second feature vector through tensor decomposition technology and fusing them to generate a fused feature vector; A management unit for determining the emotion probability distribution based on the fused feature vector, generating a physical conditioning plan according to the emotion probability distribution, and sending the physical conditioning plan to the user terminal.

10. A terminal device, characterized in that, Includes: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method for recommending a physical health management plan based on tongue image monitoring as described in any one of claims 1 to 8.

Images