Intelligent AI glasses health monitoring method based on multi-modal fusion

Abstract

The invention discloses an intelligent AI glasses health monitoring method based on multi-modal fusion, and belongs to the technical field of medical health monitoring. Comprising the following steps that a plurality of sensors on an AI glasses body are used for collecting data, and the data comprise heart rate information, blood pressure data, body temperature data, eye images, facial expression images and voice data of a user; a processing module in the AI glasses main body performs feature extraction on the acquired data, including heart rate feature calculation, blood pressure feature calculation, body temperature feature calculation, eye image feature calculation, facial expression image feature calculation and voice feature calculation; the processing module in the AI glasses main body also adopts a fusion model based on a convolutional neural network and a recurrent neural network to perform feature mapping, feature splicing and fusion network processing on the feature vectors of different modals to obtain a fused feature vector; based on a large amount of health data and clinical research, a processing module in the AI glasses body adopts a support vector machine algorithm to classify and predict fused feature vectors.

Description

Technical Field

[0001] The present invention relates to the technical field of medical and health monitoring, and specifically to an intelligent AI glasses health monitoring method based on multimodal fusion. Background Art

[0002] With the continuous improvement of people's health awareness and the rapid development of technology, wearable health monitoring devices have gradually become a research and application hotspot. Traditional health monitoring methods, such as regular physical examinations in hospitals, have limitations in time and space and cannot achieve real-time and dynamic monitoring of human health.

[0003] Most of the existing wearable health monitoring devices can only monitor one or a few physiological parameters. For example, smart bracelets mainly monitor heart rate, steps, etc. These devices have deficiencies in the comprehensiveness of data collection and are difficult to accurately evaluate the complex health status of the human body. At the same time, single-modal data is easily affected by environmental factors and individual differences, resulting in limited accuracy and reliability of monitoring results.

[0004] In addition, different types of health monitoring data are often scattered on multiple devices or platforms, lacking effective integration and comprehensive analysis, and cannot provide users with comprehensive, accurate health assessments and personalized health advice.

[0005] Therefore, developing a technology that can fuse multiple data modalities and achieve comprehensive and accurate health monitoring has important practical significance. The intelligent AI glasses health monitoring method based on multimodal fusion is expected to break through the limitations of the existing technology and provide users with more convenient, efficient, and comprehensive health monitoring services. Summary of the Invention

[0006] The purpose of the present invention is to provide an intelligent AI glasses health monitoring method based on multimodal fusion to solve the problems raised in the above background art.

[0007] In view of the above problems, the technical solution proposed by the present invention is:

[0008] An intelligent AI glasses health monitoring method based on multimodal fusion, comprising the following steps:

[0009] S1. Use a number of sensors on the main body of the AI glasses to collect data, including the heart rate information, blood pressure data, body temperature data, eye images, facial expression images, and voice data of the user;

[0010] As a preferred technical solution of the present invention, the sensor includes a heart rate sensor installed at the nose pad of the AI glasses main body, a blood pressure sensor installed at the temple of the AI glasses main body near the ear, a body temperature sensor installed at the temple of the AI glasses main body near the temple, a high-definition camera installed on both sides of the front end of the AI glasses main body frame, and a microphone installed at the position of the AI glasses main body temple near the mouth;

[0011] The heart rate sensor adopts the photoplethysmography technology. By emitting light of a specific wavelength to irradiate the skin, hemoglobin in the blood will absorb part of the light, and the remaining light will be reflected back to the sensor. As the heart beats, the blood volume changes, and the intensity of the reflected light also changes accordingly. The sensor converts this change in light intensity into an electrical signal to obtain heart rate information;

[0012] The blood pressure sensor uses the oscillometric principle. By applying pressure to the artery through an airbag, when the pressure changes, the vibration of the arterial wall will cause fluctuations in the pressure signal. The sensor detects these fluctuation signals and calculates the systolic blood pressure and diastolic blood pressure through a specific algorithm;

[0013] The body temperature sensor uses temperature-sensitive elements such as thermistors or thermocouples to convert temperature changes into electrical signals, and obtains accurate body temperature values after calibration and processing;

[0014] A power supply module is also installed inside the temple of the AI glasses main body to supply power to each electrical component. At the same time, a charging interface is also embedded outside the AI glasses main body to charge the power supply module.

[0015] S2. The processing module inside the AI glasses main body respectively extracts features from the collected data, including calculating the features of heart rate, blood pressure, body temperature, eye images, facial expression images, and voice;

[0016] As a preferred technical solution of the present invention, the features of the heart rate include time-domain features and frequency-domain features. The time-domain features of the heart rate include average heart rate and standard deviation of heart rate variability. The formula for calculating the average heart rate is: where N is the number of sampling points, and HR i is the heart rate value at the i-th sampling point;

[0017] The formula for calculating the standard deviation of heart rate variability is:

[0018] The frequency-domain features of the heart rate convert the heart rate signal to the frequency domain through fast Fourier transform, and calculate the low-frequency power LF, high-frequency power HF, and the ratio of low-frequency to high-frequency power LF / HF.

[0019] As a preferred technical solution of the present invention, the characteristics of the blood pressure signal include systolic blood pressure SBP, diastolic blood pressure DBP, and mean arterial pressure MAP characteristics. The formula for calculating the mean arterial pressure is:

[0020]

[0021] As a preferred technical solution of the present invention, the characteristics of the body temperature signal include the average body temperature and the body temperature fluctuation range. The formula for calculating the average body temperature is where N is the number of sampling points, and T i is the body temperature value at the i-th sampling point;

[0022] The formula for calculating the body temperature fluctuation range is: ΔT = T max -T min , where T max and T min are the maximum and minimum values of the body temperature respectively.

[0023] As a preferred technical solution of the present invention, the extraction of the eye image features includes the following steps:

[0024] S21. Perform preprocessing on the eye image, including grayscale conversion, filtering, and histogram equalization operations;

[0025] S22. Use an edge detection algorithm to detect the eye contour and blood vessel edges, and extract features such as the eye contour perimeter and the number of blood vessel bifurcation points;

[0026] The extraction of the facial expression image features includes the following steps:

[0027] S23. Input the preprocessed facial image into a pre-trained convolutional neural network model to extract the deep feature vectors of the image.

[0028] As a preferred technical solution of the present invention, the extraction of the speech signal features includes the following steps:

[0029] S24. Perform speech preprocessing, including frame segmentation and windowing operations;

[0030] S25. Extract acoustic features such as fundamental frequency and formant frequency. The fundamental frequency is calculated by the autocorrelation method, and the formant frequency is extracted by the linear predictive cepstral coefficient method.

[0031] S3. The processing module inside the main body of the AI glasses also adopts a fusion model based on convolutional neural network and recurrent neural network, and performs feature mapping, feature splicing, and fusion network processing on the feature vectors of different modalities to obtain the fused feature vectors;

[0032] As a preferred technical solution of the present invention, in the multi-modal fusion model, the feature mapping is to map the physiological signal feature vector F physio the visual image feature vector F image and the speech feature vector F voice respectively perform linear transformation through a fully connected layer to obtain the mapped feature vectors

[0033] The feature concatenation is to concatenate the mapped feature vectors in the feature dimension to obtain the concatenated feature vector

[0034] The fusion network adopts a structure combining a convolutional neural network and a recurrent neural network. The formula for the convolutional neural network layer is: X cnn = ReLU(W cnn *F concat + b cnn ), where W cnn is the weight matrix of the CNN layer, b cnn is the bias vector, * represents the convolution operation, and ReLU is the activation function;

[0035] The formula for the recurrent neural network layer is: h t = tanh(W h h t-1 + W x X cnn,t + b h ), where h t is the hidden state of the recurrent neural network at time t, W h is the weight matrix of the hidden state, W x is the weight matrix of the input feature, b h is the bias vector, and X cnn,t is the output of the CNN layer at time t;

[0036] The formula for the output layer is: F fusion : F fusion = W out h final + b out , where W out is the weight matrix of the output layer, b out is the bias vector, h final is the final hidden state of the recurrent neural network layer, and finally the fused feature vector F fusion is obtained.

[0037] S4. Based on a large amount of health data and clinical research, the processing module inside the AI glasses main body uses a support vector machine algorithm to classify and predict the fused feature vector to evaluate the user's health status;

[0038] As a preferred technical solution of the present invention, a health assessment model is constructed based on a large amount of health data and clinical research, specifically:

[0039] The support vector machine algorithm is used to classify and predict the fused feature vector F fusion The goal of the support vector machine algorithm is to find an optimal hyperplane to separate samples of different classes. The specific formula is as follows:

[0040] For a given training sample set where F fusion is the fused feature vector of the i-th sample, and y i ∈{-1, 1} is the class label of the sample. The optimization problem of the support vector machine algorithm can be expressed as:

[0041]

[0042] where w is the normal vector of the hyperplane, b is the bias, ξ i is the slack variable, C is the penalty parameter,

[0043] is the function that maps the feature vector to a high-dimensional space.

[0044] By solving the above optimization problem, the optimal w and b are obtained. Then, for a new sample using the classification decision function: to determine the health status category to which it belongs. For example, if the calculation result is greater than 0, that is then the decision function determines that the new sample belongs to a certain health status category; if the calculation result is less than 0, that is then it is determined that the sample belongs to another health status category.

[0045] S5. Set the threshold range of the health index. When the monitored data exceeds the normal range, the processing module inside the AI glasses main body issues a warning through the alarm module;

[0046] The alarm module is a vibration motor installed inside the temple of the AI glasses main body.

[0047] For example, the normal range of heart rate is 60 - 100 beats per minute. When the monitored heart rate HR exceeds this range, that is, HR < 60 or HR > 100, the AI glasses main body issues a warning, and the warning method is to remind by vibrating the vibration motor.

[0048] S6. The AI glasses main body sends the processed health monitoring data to the user's smart device through the wireless communication module;

[0049] The wireless communication module is a Bluetooth transceiver and an analog-to-digital converter installed inside the temple of the AI glasses main body.

[0050] S7. The application program on the intelligent device generates line charts, bar charts, and pie charts based on the received data to display the user's health status.

[0051] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0052] First, by integrating multiple data modalities, it overcomes the limitations of single data modalities, can monitor the human health status more comprehensively and accurately, and improves the accuracy and reliability of monitoring.

[0053] Second, the intelligent AI glasses can collect data in real time, continuously track the user's health status, timely detect potential health problems, and provide a basis for early intervention.

[0054] Third, as a wearable device, the intelligent AI glasses are convenient to wear, not restricted by time and space, and users can perform health monitoring anytime and anywhere, improving the convenience and accessibility of health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is a schematic flowchart of the health monitoring method of the intelligent AI glasses based on multi-modal fusion disclosed in the embodiment of the present invention;

[0056] Figure 2 It is a schematic three-dimensional structure diagram of the health monitoring method of the intelligent AI glasses based on multi-modal fusion disclosed in the embodiment of the present invention.

[0057] In the figure: 100, AI glasses main body; 200, heart rate sensor; 300, blood pressure sensor; 400, body temperature sensor; 500, high-definition camera; 600, voice sensor. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0058] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0059] Please refer to Figure 1 - Figure 2 , the present invention provides a technical solution: a health monitoring method of intelligent AI glasses based on multi-modal fusion, including the following steps:

[0060] S1. Collect data using several sensors on the AI glasses main body 100. The data includes the user's heart rate information, blood pressure data, body temperature data, eye images, facial expression images, and voice data.

[0061] As a preferred technical solution of the present invention, the sensors include a heart rate sensor 200 installed at the nose pad part of the AI glasses main body 100, a blood pressure sensor 300 installed at the temple part of the AI glasses main body 100 near the ear, a body temperature sensor 400 installed at the temple part of the AI glasses main body 100 near the temple, high-definition cameras 500 installed on both left and right sides of the front end of the frame of the AI glasses main body 100, and a microphone 600 installed at the position near the mouth of the AI glasses main body 100.

[0062] The heart rate sensor 200 uses the photoplethysmography technology. By emitting light with a specific wavelength to irradiate the skin, hemoglobin in the blood will absorb part of the light, and the remaining light is reflected back to the sensor. As the heart beats, the blood volume changes, and the intensity of the reflected light also changes accordingly. The sensor converts this change in light intensity into an electrical signal to obtain the heart rate information.

[0063] The blood pressure sensor 300 uses the oscillometric principle. By applying pressure to the artery through an airbag, when the pressure changes, the vibration of the artery wall will cause fluctuations in the pressure signal. The sensor detects these fluctuation signals and calculates the systolic blood pressure and diastolic blood pressure through a specific algorithm.

[0064] The body temperature sensor 400 uses temperature-sensitive elements such as thermistors or thermocouples to convert temperature changes into electrical signals, and obtains accurate body temperature values after calibration and processing.

[0065] A power supply module is also installed inside the temple of the AI glasses main body 100 to supply power to each electrical component. At the same time, a charging interface is embedded outside the AI glasses main body 100 to charge the power supply module.

[0066] S2. The processing module inside the AI glasses main body 100 extracts features from the collected data respectively, including calculating the features of heart rate, blood pressure, body temperature, eye images, facial expression images, and voice.

[0067] As a preferred technical solution of the present invention, the features of heart rate include time-domain features and frequency-domain features. The time-domain features of heart rate include average heart rate and standard deviation of heart rate variability. The formula for calculating the average heart rate is: where N is the number of sampling points, and HR i is the heart rate value at the i-th sampling point;

[0068] The formula for calculating the standard deviation of heart rate variability is:

[0069] The frequency domain characteristics of the heart rate are obtained by converting the heart rate signal to the frequency domain through fast Fourier transform, and calculating the low-frequency power LF, high-frequency power HF, and the ratio of low-frequency to high-frequency power LF / HF.

[0070] As a preferred technical solution of the present invention, the characteristics of the blood pressure signal include systolic blood pressure SBP, diastolic blood pressure DBP, and mean arterial pressure MAP characteristics. The formula for calculating the mean arterial pressure is:

[0071]

[0072] As a preferred technical solution of the present invention, the characteristics of the body temperature signal include the average body temperature and the body temperature fluctuation range. The formula for calculating the average body temperature is where N is the number of sampling points, and T i is the body temperature value at the i-th sampling point;

[0073] The formula for calculating the body temperature fluctuation range is: ΔT = T max -T min where T max and T min are the maximum and minimum values of the body temperature, respectively.

[0074] As a preferred technical solution of the present invention, the extraction of eye image features includes the following steps:

[0075] S21. Perform preprocessing on the eye image, including grayscale conversion, filtering, and histogram equalization operations;

[0076] S22. Use an edge detection algorithm to detect the eye contour and blood vessel edges, and extract features such as the eye contour perimeter and the number of blood vessel bifurcation points;

[0077] The extraction of facial expression image features includes the following steps:

[0078] S23. Input the preprocessed facial image into a pre-trained convolutional neural network model to extract the deep feature vector of the image.

[0079] As a preferred technical solution of the present invention, the extraction of speech signal features includes the following steps:

[0080] S24. Perform speech preprocessing, including frame segmentation and windowing operations;

[0081] S25. Extract acoustic features such as fundamental frequency and formant frequency. The fundamental frequency is calculated by the autocorrelation method, and the formant frequency is extracted by the linear predictive cepstral coefficient method.

[0082] S3. The processing module inside the AI glasses main body 100 also adopts a fusion model based on a convolutional neural network and a recurrent neural network, performs feature mapping, feature splicing, and fusion network processing on feature vectors of different modalities to obtain a fused feature vector;

[0083] As a preferred technical solution of the present invention, in the multi-modal fusion model, feature mapping is to linearly transform the physiological signal feature vector F physio the visual image feature vector F image and the voice feature vector F voice respectively through a fully connected layer to obtain the mapped feature vectors

[0084] Feature splicing is to splice the mapped feature vectors in the feature dimension to obtain the spliced feature vector

[0085] The fusion network adopts a structure combining a convolutional neural network and a recurrent neural network. The formula for the convolutional neural network layer is: X cnn = ReLU(W cnn *F concat + b cnn ), where W cnn is the weight matrix of the CNN layer, b cnn is the bias vector, * represents the convolution operation, and ReLU is the activation function;

[0086] The formula for the recurrent neural network layer is: h t = tanh(W h h t-1 + W x X cnn,t + b h ), where h t is the hidden state of the recurrent neural network at time t, W h is the weight matrix of the hidden state, W x is the weight matrix of the input feature, b h is the bias vector, and X cnn,t is the output of the CNN layer at time t;

[0087] The formula for the output layer is: F fusion : F fusion = W out h final + b out , where W out is the weight matrix of the output layer, b out is the bias vector, h final is the final hidden state of the recurrent neural network layer, and finally the fused feature vector F fusion .

[0088] S4. Based on a large amount of health data and clinical research, the processing module inside the AI glasses main body 100 uses the support vector machine algorithm to classify and predict the fused feature vectors, and evaluate the user's health status;

[0089] As a preferred technical solution of the present invention, based on a large amount of health data and clinical research, a health assessment model is constructed, specifically;

[0090] The support vector machine algorithm is used to classify and predict the fused feature vector F fusion The goal of the support vector machine algorithm is to find an optimal hyperplane to separate samples of different categories. The specific formula is as follows:

[0091] For a given training sample set where F fusion is the fused feature vector of the i-th sample, and y i ∈{-1,1} is the class label of the sample. The optimization problem of the support vector machine algorithm can be expressed as:

[0092]

[0093] where w is the normal vector of the hyperplane, b is the bias, ξ i is the slack variable, C is the penalty parameter,

[0094] is the function that maps the feature vector to a high-dimensional space.

[0095] By solving the above optimization problem, the optimal w and b are obtained. Then, for a new sample Using the classification decision function: Judge the health status category to which it belongs. For example, if the calculation result is greater than 0, that is then the decision function determines that the new sample belongs to a certain health status category; if the calculation result is less than 0, that is then it is determined that the sample belongs to another health status category.

[0096] S5. Set the threshold range of the health index. When the monitored data exceeds the normal range, the processing module inside the AI glasses main body 100 issues a warning through the alarm module;

[0097] The alarm module is a vibration motor installed inside the temple of the AI glasses main body 100.

[0098] For example, the normal range of the heart rate is 60 - 100 beats per minute. When the monitored heart rate HR exceeds this range, that is, HR < 60 or HR > 100, the AI glasses main body 100 issues a warning, and the warning method is to vibrate and remind through the vibration motor.

[0099] S6. The AI glasses main body 100 sends the processed health monitoring data to the user's smart device through the wireless communication module;

[0100] The wireless communication module is a Bluetooth transceiver and an analog-to-digital converter installed inside the temple of the AI glasses main body 100.

[0101] S7. The application program on the smart device generates line charts, bar charts, and pie charts based on the received data to display the user's health status.

[0102] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0103] Please refer to Figure 1 - Figure 2 , the present invention provides a technical solution: an intelligent AI glasses health monitoring method based on multimodal fusion, including the following steps:

[0104] In a daily health monitoring scenario, Mr. Li wears smart AI glasses equipped with the health monitoring method of the present invention. The sensors of this pair of glasses are distributed in different positions. The heart rate sensor 200 on the nose pad uses the photoplethysmography technology, emits green light to irradiate the skin, and obtains heart rate information based on the change in the intensity of the reflected light caused by the change in blood volume; the blood pressure sensor 300 near the ear on the temple measures blood pressure using the oscillometric method principle; the body temperature sensor 400 near the temple on the temple uses a thermistor to collect body temperature; the high-definition cameras 500 on the left and right sides of the front end of the frame are responsible for collecting eye images and facial expression images; the microphone 600 near the mouth on the temple is used to collect voice data. The power supply module inside the temple of the AI glasses main body 100 powers each component, and the external charging interface facilitates charging at any time.

[0105] During a day's work, the AI glasses continuously collect multimodal data of Mr. Li. In the morning, Mr. Li worked at his desk in the office for a long time. The heart rate sensor 200 of the glasses collected heart rate data at a frequency of 1 time per second, the blood pressure sensor 300 measured blood pressure every 5 minutes, the body temperature sensor 400 recorded body temperature every 10 minutes, the high-definition camera 500 captured eye and facial images every minute, and the microphone 600 monitored voice information in real time.

[0106] The collected data is transmitted to the processing module inside the AI glasses main body 100. When calculating the average heart rate in the time domain for the heart rate data, a statistical period of 1 minute is used, that is, N = 60. According to the formula the average heart rate within this 1 minute is obtained; at the same time, the standard deviation of heart rate variability is calculated to reflect the fluctuation of the heart rate. In the frequency domain, the heart rate signal is converted through fast Fourier transform, and the low-frequency power LF, high-frequency power HF, and the ratio of low-frequency to high-frequency power LF / HF are calculated. For blood pressure data, systolic blood pressure, diastolic blood pressure, and mean arterial pressure are extracted, and the mean arterial pressure is calculated according to the formula For body temperature data, the average body temperature and the body temperature fluctuation range are calculated. The average body temperature is obtained according to the formula and the body temperature fluctuation range is the difference between the maximum and minimum body temperatures within a period of time.

[0107] The eye images are first preprocessed by grayscale conversion, filtering, and histogram equalization, and then the edge detection algorithm is used to extract features such as the perimeter of the eye contour and the number of blood vessel bifurcation points. The facial expression images are input into a pre-trained convolutional neural network model to obtain deep feature vectors. The speech data is first preprocessed by frame segmentation and windowing, and then the fundamental frequency is calculated by the autocorrelation method, and the formant frequencies are extracted using the linear predictive cepstral coefficient method.

[0108] The processing module uses a fusion model based on convolutional neural network and recurrent neural network to process the feature vectors of each modality. The physiological signal, visual image, and speech feature vectors are linearly transformed through the fully connected layer respectively to complete feature mapping; then they are concatenated in the feature dimension to obtain the concatenated feature vector; further features are extracted and fused through the convolutional neural network layer and the recurrent neural network layer, and finally the fused feature vector is obtained at the output layer.

[0109] The health assessment model constructed based on a large amount of health data and clinical research uses the support vector machine algorithm to classify and predict the fused feature vectors. In this process, the AI glasses will compare and analyze Mr. Li's real-time data with the data in the training sample set. In the training sample set, the health status is divided into three categories: "healthy", "sub-healthy", and "high disease risk", corresponding to the class labels 1, 0, and -1 respectively. When Mr. Li's fused feature vector is input into the classification decision function After calculation, if the result is 1, it is determined that Mr. Li is currently in a healthy state; if it is 0, it is a sub-healthy state; if it is -1, it means a high disease risk.

[0110] The system sets a threshold range for health indicators. The normal range of heart rate is 60 - 100 beats per minute. When Mr. Li's work pressure suddenly increases and he becomes emotionally excited, the heart rate sensor 200 monitors that his heart rate reaches 110 beats per minute, exceeding the normal range. The internal processing module of the AI glasses main body 100 controls the vibration motor inside the temple to send a vibration reminder, informing Mr. Li that his heart rate is abnormal.

[0111] The AI glasses main body 100 sends the processed health monitoring data, such as heart rate, blood pressure, body temperature change data in the past 1 hour, as well as eye images and facial expression analysis results, etc., to Mr. Li's smartphone through a wireless communication module composed of a Bluetooth transceiver and an analog-to-digital converter inside the temple. After a dedicated application on the phone receives the data, it generates line charts to show the changing trends of heart rate, blood pressure, and body temperature over time, uses bar charts to compare the average blood pressure values at different times of the day, and presents the proportion of health indicators such as cardiovascular health in the overall health assessment in a pie chart. By viewing these charts, Mr. Li can intuitively understand his health status and adjust his work rhythm and lifestyle in a timely manner.

Claims

1. An intelligent AI glasses health monitoring method based on multimodal fusion, characterized in that, It includes the following steps: S1. Use several sensors on the AI glasses main body (100) to collect data, including the user's heart rate information, blood pressure data, body temperature data, eye images, facial expression images, and voice data; S2. The processing module inside the AI glasses main body (100) extracts features from the collected data respectively, including calculating the features of heart rate, blood pressure, body temperature, eye images, facial expression images, and voice; S3. The processing module inside the AI glasses main body (100) also uses a fusion model based on convolutional neural network and recurrent neural network to perform feature mapping, feature splicing, and fusion network processing on feature vectors of different modalities to obtain a fused feature vector; S4. Based on a large amount of health data and clinical research, the processing module inside the AI glasses main body (100) uses a support vector machine algorithm to classify and predict the fused feature vector to evaluate the user's health status; S5. Set the threshold range of health indicators. When the monitored data exceeds the normal range, the processing module inside the AI glasses main body (100) issues a warning through the alarm module; S6. The AI glasses main body (100) sends the processed health monitoring data to the user's smart device through the wireless communication module; S7. The application program on the smart device generates line charts, bar charts, and pie charts based on the received data to display the user's health status.

2. The method for health monitoring of an intelligent AI glasses based on multimodal fusion according to claim 1, wherein, The sensors include a heart rate sensor (200) installed at the nose pad of the AI glasses main body (100), a blood pressure sensor (300) installed at the temple of the temple of the AI glasses main body (100), a body temperature sensor (400) installed at the temple of the temple of the AI glasses main body (100), high-definition cameras (500) installed on both sides of the front end of the frame of the AI glasses main body (100), and a microphone (600) installed at the position near the mouth of the temple of the AI glasses main body (100); The alarm module is a vibration motor installed inside the temple of the AI glasses main body (100); The wireless communication module is a Bluetooth transceiver and an analog-to-digital converter installed inside the temple of the AI glasses main body (100).

3. The method for health monitoring of an intelligent AI glasses based on multimodal fusion according to claim 1, characterized in that, The characteristics of the heart rate include time-domain characteristics and frequency-domain characteristics. The time-domain characteristics of the heart rate include the average heart rate and the standard deviation of heart rate variability. The formula for calculating the average heart rate is: where N is the number of sampling points, and HR i is the heart rate value at the i-th sampling point; The standard deviation calculation formula for heart rate variability is as follows: The frequency domain feature of the heart rate converts the heart rate signal to the frequency domain through fast Fourier transform, and calculates the low-frequency power LF, high-frequency power HF, and the ratio of low-frequency to high-frequency power LF / HF.

4. A health monitoring method for an intelligent AI glasses based on multimodal fusion according to claim 1, characterized in that, The characteristics of the blood pressure signal include systolic blood pressure SBP, diastolic blood pressure DBP, and mean arterial pressure MAP characteristics. The formula for calculating the mean arterial pressure is:

5. A health monitoring method for an intelligent AI glasses based on multimodal fusion according to claim 1, characterized in that The characteristics of the body temperature signal include the average body temperature and the body temperature fluctuation range, and the formula for calculating the average body temperature is where N is the number of sampling points, and T i is the body temperature value at the i-th sampling point; The calculation formula for the body temperature fluctuation range is: ΔT = T max - T min , where T max and T min are the maximum and minimum values of the body temperature respectively.

6. The method for health monitoring of an intelligent AI glasses based on multimodal fusion according to claim 1, characterized in that, The extraction of the eye image features includes the following steps: S21. Perform preprocessing on the eye image, including grayscale conversion, filtering, and histogram equalization operations; S22. Use an edge detection algorithm to detect the eye contour and blood vessel edges, and extract the features of the eye contour perimeter and the number of blood vessel bifurcation points; The extraction of the facial expression image features includes the following steps: S23. Input the preprocessed facial image into a pre-trained convolutional neural network model to extract the deep feature vector of the image.

7. A health monitoring method for an intelligent AI glasses based on multi-modal fusion according to claim 1, characterized in that, The extraction of the voice signal features includes the following steps: S24. Perform voice preprocessing, including frame segmentation and windowing operations; S25. Extract the acoustic features of fundamental frequency and formant frequency. The fundamental frequency is calculated by the autocorrelation method, and the formant frequency is extracted by the linear predictive cepstral coefficient method.

8. A health monitoring method for an intelligent AI glasses based on multimodal fusion according to claim 1, characterized in that, In the multi-modal fusion model, the feature mapping is to transform the physiological signal feature vector F physio the visual image feature vector F image and the speech feature vector F voice respectively through the fully connected layer for linear transformation to obtain the mapped feature vector The feature concatenation is to concatenate the mapped feature vectors in the feature dimension to obtain the concatenated feature vector The fusion network adopts a structure that combines a convolutional neural network and a recurrent neural network. The formula for the convolutional neural network layer is: X cnn = ReLU(W cnn * F concat + b cnn ), where W cnn is the weight matrix of the CNN layer, b cnn is the bias vector, * represents the convolution operation, and ReLU is the activation function; The formula for the recurrent neural network layer is: h t = tanh(W h h t-1 + W x X cnn,t + b h ), where h t is the hidden state of the recurrent neural network at time t, W h is the weight matrix of the hidden state, W x is the weight matrix of the input features, b h is the bias vector, and X cnn,t is the output of the CNN layer at time t; The formula for the output layer is: F fusion : F fusion = W out h final + b out , where W out is the weight matrix of the output layer, b out is the bias vector, h final is the final hidden state of the recurrent neural network layer, and finally the fused feature vector F fusion is obtained.

9. The method for health monitoring of an intelligent AI glasses based on multimodal fusion according to claim 1, wherein, In the support vector machine algorithm, for a given training sample set where F fusion is the fusion feature vector of the i-th sample, and y i ∈{-1, 1} is the class label of the sample, the optimization problem of the support vector machine is: where \(w\) is the normal vector of the hyperplane, \(b\) is the bias, and \(\xi\) i is the slack variable, \(C\) is the penalty parameter, and \(\varphi\) is the function that maps the feature vector into a high-dimensional space; By solving the above optimization problem, the optimal \(w\) and \(b\) are obtained. Then, for a new sample its classification decision function is:

Images