Ear-nose-throat department symptom monitoring method and system based on multi-modal data fusion

Abstract

The invention discloses an ear-nose-throat department symptom monitoring method and system based on multi-modal data fusion, and relates to the technical field of medical monitoring. Comprising the steps of monitoring physiological data in real time, calling historical medical record data, collecting environment data, and preprocessing the monitored, called and collected data; carrying out feature fusion on the preprocessed data, extracting potential associated features, and carrying out multilevel and sequential feature learning; on the basis of the extracted features, an improved long-short-term memory network is used for conducting symptom prediction, in combination with historical trends of the blood sugar and the blood pressure of the patient, correlation between the symptoms and ear-nose-throat symptoms is evaluated in real time, and personalized symptom early warning is output; and sending a health prompt to the patient through the intelligent equipment or the mobile terminal application in real time. According to the invention, conventional symptom early warning can be carried out, and personalized health intervention measures can be provided according to individual health conditions.

Description

Technical Field

[0001] The present invention relates to the field of medical monitoring technology, and in particular to an otolaryngology symptom monitoring method and system based on multimodal data fusion. Background Art

[0002] In recent years, the emergence of multimodal data fusion technology has enabled the effective integration of information from multiple sources such as physiological data, environmental data, and historical medical records, thereby providing more comprehensive information support for precision medicine. With the popularization of wearable devices, real-time monitoring of physiological data such as heart rate, blood pressure, body temperature, and blood sugar has become a routine application, and data collection methods are becoming increasingly accurate. In the field of otolaryngology, patients' symptoms are often closely related to environmental factors and physiological conditions. Traditional symptom monitoring methods rely on the doctor's clinical diagnosis and the patient's subjective feelings, making it difficult to achieve efficient and accurate personalized health management.

[0003] However, although the application of multimodal data fusion technology in many fields has achieved initial success, there are still many shortcomings in otolaryngology symptom monitoring. First, most of the existing health monitoring methods focus on the collection of a single data source, such as focusing on the monitoring of physiological data or the collection of environmental data, and fail to effectively integrate multimodal data for comprehensive analysis. Existing technologies usually ignore the mutual influence between environmental factors and individual physiological data, and cannot fully identify and predict the potential risks of specific symptoms. For example, changes in air quality and climate fluctuations may affect the occurrence of respiratory symptoms, while physiological changes in patients (such as increased blood sugar or blood pressure fluctuations) may further aggravate the severity of symptoms. Most of the existing symptom prediction models are based on relatively simple rules or linear models, which cannot effectively capture complex time series characteristics and multi-level correlations, resulting in limited prediction accuracy and a lack of personalized and real-time health intervention methods. Summary of the invention

[0004] In view of the problems existing in the above-mentioned background technology, the present invention proposes an otolaryngology symptom monitoring method and system based on multimodal data fusion.

[0005] Therefore, the problem to be solved by the present invention is how to effectively improve the accuracy of otolaryngology symptom monitoring and the real-time nature of patient health management.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions: In a first aspect, the present invention provides an otolaryngology symptom monitoring method based on multimodal data fusion, which includes: real-time monitoring of physiological data through wearable devices, retrieving historical medical record data from a patient database, collecting environmental data at the same time, and preprocessing the monitored, retrieved and collected data; feature fusion of the preprocessed data, extraction of potential correlation features, and multi-level, time-series feature learning through an optimized model architecture combining a convolutional neural network and a recurrent neural network; based on the extracted features, symptom prediction is performed using an improved long short-term memory network, combined with the historical trends of the patient's individual blood sugar and blood pressure, real-time evaluation of the correlation with otolaryngology symptoms, and output of personalized symptom warnings; when it is predicted that the patient develops otolaryngology symptoms due to changes in physiological data, health reminders are issued to the patient in real time through smart devices or mobile applications, and personalized measures for adjusting diet, drug use or medical advice are provided.

[0007] As a preferred solution of the otolaryngology symptom monitoring method based on multimodal data fusion described in the present invention, the preprocessing includes data cleaning, data format standardization and data synchronization.

[0008] As a preferred solution of the otolaryngology symptom monitoring method based on multimodal data fusion described in the present invention, the feature fusion of the pre-processed data to extract potential correlation features includes: performing time series modeling on each data stream through a recurrent neural network structure to capture the law of change over time, and combining with an LSTM unit to process the dependency of data within a long time span; the formula of the LSTM unit is: ; in, For the The LSTM hidden state at time , , is the corresponding weight matrix, For the -1 time LSTM hidden state, is the input feature at the current moment, is the bias term, is the activation function; by Perform maximum pooling or average pooling to obtain time series features that are highly correlated with symptom changes.

[0009] As a preferred solution of the otolaryngology symptom monitoring method based on multimodal data fusion described in the present invention, the multi-level, temporal feature learning is performed by combining the optimized convolutional neural network and the recurrent neural network model architecture, including: performing cross-modal association learning on the temporal features extracted from different modalities through the fusion layer; when fusing cross-modal data, dynamically adjusting the influence of different modal data on the prediction by using the self-attention mechanism; assuming that the feature vectors of each modality are ,in, is the number of modalities; the self-attention mechanism formula is: ; ; ; in, For the Modal pair The attention weights of modality features, For the Modal pair Modal feature similarity scoring function, is the fused multimodal feature vector, For the The modal eigenvectors, For the The modal eigenvectors, No. Modal eigenvectors.

[0010] As a preferred solution of the otolaryngology symptom monitoring method based on multimodal data fusion described in the present invention, the symptom prediction using the improved long short-term memory network includes: performing local polynomial fitting on physiological data to extract short-term and long-term trends as new features; combining the short-term and long-term trend parameters with the extracted feature vectors Fusion is performed to calibrate the short-term and long-term trend of key physiological data, and to form short-term and long-term trend parameter sets respectively. and ; Combine the trend parameter set extracted by local fitting with the fused feature vector Splice to form sequence input , constructing input sequences for symptom prediction, capturing the joint effects of short-term fluctuations and long-term trends.

[0011] As a preferred embodiment of the method for monitoring otolaryngology symptoms based on multimodal data fusion of the present invention, the method of combining the historical trends of individual blood sugar and blood pressure of the patient, evaluating the correlation with otolaryngology symptoms in real time, and outputting personalized symptom warnings comprises: inputting the sequence into Input long short-term memory network to extract time series features , and generate symptom prediction output based on individual patient characteristics , indicating the probability of occurrence of each ENT symptom; the symptom prediction output The formula is: ; in, For the The LSTM output state at the moment, and are the weight matrix and bias term used to calculate symptom prediction, respectively. is the activation function; predict the credibility According to the sequence input The abnormal distribution detection result is adjusted; according to the prediction credibility and symptom prediction output , output personalized symptom warning parameters.

[0012] As a preferred solution of the otolaryngology symptom monitoring method based on multimodal data fusion of the present invention, the adjustment formula of the prediction credibility is: ; in, To predict the reliability, is an adjustable weight, For the The abnormal detection value at the time; the calculation formula of the symptom warning parameter is: ,in, Provide personalized symptom warning output.

[0013] In a second aspect, the present invention provides an otolaryngology symptom monitoring system based on multimodal data fusion, which includes: a data acquisition module for monitoring physiological data in real time through a wearable device, retrieving historical medical record data from a patient database, simultaneously collecting environmental data, and preprocessing the monitored, retrieved and collected data; The feature fusion module is used to fuse the preprocessed data, extract potential related features, and perform multi-level and time-series feature learning through an optimized model architecture combining convolutional neural networks and recurrent neural networks; The symptom prediction module is used to predict symptoms based on the extracted features using an improved long short-term memory network, combined with the historical trends of individual blood sugar and blood pressure of patients, to evaluate the correlation with ENT symptoms in real time and output personalized symptom warnings; The reminder and intervention module is used to send health reminders to patients in real time through smart devices or mobile applications when it is predicted that the patient will develop ENT symptoms due to changes in physiological data, and provide personalized measures such as adjusting diet, medication use or medical advice.

[0014] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein the computer program instructions, when executed by the processor, implement the steps of the otolaryngology symptom monitoring method based on multimodal data fusion as described in the first aspect of the present invention.

[0015] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program instructions, when executed by a processor, implement the steps of the otolaryngology symptom monitoring method based on multimodal data fusion as described in the first aspect of the present invention.

[0016] The beneficial effects of the present invention are as follows: Based on multimodal data fusion, the present invention combines physiological data monitored by wearable devices, historical medical data and environmental data to achieve deep data fusion and multi-level feature learning, and can accurately capture various potential symptom causes. Different from the limitations of the prior art, the present invention uses optimized convolutional neural network and recurrent neural network architectures to learn temporal features, thereby improving the accuracy and sensitivity of the model in otolaryngology symptom prediction.

[0017] In addition, through the improved long short-term memory network (LSTM) combined with the patient's individual health characteristics, such as the historical change trends of blood sugar and blood pressure, the present invention can not only perform routine symptom warnings, but also provide personalized health intervention measures according to the individual's health status, such as diet adjustment, drug use recommendations or timely medical reminders, thereby realizing truly intelligent and personalized health management. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 Flow chart of the otolaryngology symptom monitoring method based on multimodal data fusion.

[0020] Figure 2 This is the structural diagram of the ENT symptom monitoring system based on multimodal data fusion. DETAILED DESCRIPTION

[0021] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention may also be implemented in other ways different from those described herein, and those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The term "in one embodiment" that appears in different places in this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0024] Example 1, reference Figure 1~Figure 2 , which is the first embodiment of the present invention, provides an otolaryngology symptom monitoring method based on multimodal data fusion, comprising: S1: Use wearable devices such as smart bracelets to monitor physiological data in real time, including but not limited to blood sugar data and blood pressure data, and retrieve historical medical records from the patient database; at the same time, smart home devices collect home environmental data, environmental temperature and humidity and other parameters in real time. All collected data are uniformly named as comprehensive monitoring data sets.

[0025] Specifically, all physiological data and environmental data collected as described above are aggregated to a central processing platform via a data transmission protocol (such as Bluetooth, Wi-Fi, etc.).

[0026] Furthermore, the central processing platform pre-processes the collected data, including data cleaning: removing invalid data (such as sensor errors, data loss, etc.); data format standardization: unifying the data format to ensure that the data collected by each device can be seamlessly integrated; data synchronization: ensuring that the data of wearable devices and home devices are collected and uploaded synchronously to avoid timing inconsistencies, and the pre-processed data will be stored in the cloud or local database in real time to facilitate subsequent data analysis and processing.

[0027] S2: Perform feature fusion on the preprocessed data, combine blood sugar fluctuations, blood pressure changes, environmental factors, etc., extract potential correlation features, and perform multi-level, time-series feature learning through an optimized model architecture that combines convolutional neural networks (CNNs) and recurrent neural networks (RNNs).

[0028] S2.1: First, blood glucose fluctuations, blood pressure changes, environmental factors, and other physiological data are encoded into high-dimensional vectors, and a dedicated embedding layer is used to map each data type, transforming different data sources into a unified feature representation that can be used in deep learning models.

[0029] S2.2: Use the recurrent neural network (RNN) structure to perform time series modeling on each data stream (such as blood sugar fluctuations, blood pressure changes, and environmental parameters) to capture the law of its changes over time. Combined with the LSTM (long short-term memory) unit, it processes the dependencies of data over a long time span and extracts time series features related to changes in ENT symptoms. Specifically, the LSTM unit formula is: ; in, For the The LSTM hidden state at the moment is used to represent the characteristics of the current moment , is the corresponding weight matrix, For the -1 time LSTM hidden state, is the input feature at the current moment (including blood sugar, blood pressure, environmental factors, etc.), is the bias term, is the activation function.

[0030] After the LSTM model is trained, the output of the network will be the hidden state corresponding to each time window, which contains the patterns and trends of multimodal data (blood sugar, blood pressure, environmental parameters, etc.) changing over time. These hidden states can not only be used as feature inputs for prediction, but also can be used to extract time series features related to changes in ENT symptoms.

[0031] By further analyzing and aggregating these hidden states (such as maximum pooling, average pooling, etc.), we can obtain temporal features that are highly correlated with symptom changes: By observing the pattern of blood sugar data over time, the frequency of blood sugar fluctuations can affect the frequency of symptoms. For example, frequent rapid increases or decreases in blood sugar may increase the probability of dry throat or sore throat; By capturing the impact of blood pressure fluctuations on symptoms, high or low pressure conditions that may cause ENT discomfort can be extracted. For example, sudden rises and falls in blood pressure may affect the patient's physical condition and thus affect the occurrence of ENT symptoms; frequent fluctuations in blood pressure (for example, frequent ups and downs in blood pressure values ​​within a short period of time) may lead to aggravation of ENT symptoms.

[0032] By monitoring the changes in environmental data (such as temperature, humidity, etc.) over time, we can extract how these factors affect the patient's ENT health in a specific time period. Among them, high and low temperatures, dry and humid environmental changes will have an impact on ENT symptoms. For example, a high temperature and low humidity environment can easily lead to symptoms such as dry throat and cough, while high humidity may cause nasal congestion and other problems.

[0033] In addition, air quality (such as increased PM2.5 concentration) can cause respiratory discomfort and directly affect the health of the ears, nose and throat. By monitoring the concentration of air pollutants in the environment, the relationship between the concentration of these pollutants and symptoms (such as coughing, sputum, etc.) can be determined.

[0034] S2.3: Through the fusion layer, the temporal features extracted from different modalities (such as blood sugar, blood pressure, environmental data, etc.) are cross-modal association learning, and the multimodal fusion network (such as the multimodal fusion layer based on the self-attention mechanism) is used to further strengthen the mutual influence relationship between different types of data.

[0035] Specifically, when fusing cross-modal data, the self-attention mechanism is used to dynamically adjust the influence of different modal data on prediction, thereby strengthening the mutual influence between different data sources. Assume that the feature vectors of each modality are ,in, is the number of modes.

[0036] Furthermore, the self-attention mechanism formula is: ; ; ; in, For the Modal pair The attention weights of modality features, For the Modal pair Modal feature similarity scoring function; is the number of modes, is the fused multimodal feature vector, For the The modal eigenvectors, For the The modal eigenvectors, No. Modal eigenvectors.

[0037] S2.4: Taking into account the possible noise or outliers in the data, use a data-adaptive recalibration mechanism to recalibrate the extracted features to improve adaptability to the individual differences of various patients, especially in the case of large blood sugar fluctuations, to maintain model stability and accuracy.

[0038] Specifically, the scales of different modal features are dynamically adjusted through the Bayesian optimization method to adapt to individual differences. The Bayesian optimization formula is: ; in, is the adjusted eigenvector, is the feature vector, is the learning rate, which is used to control the adjustment amplitude. For the The data noise estimate at a certain moment reflects the uncertainty of the data at the current moment.

[0039] S3: Based on the features extracted in step S2, an improved long short-term memory (LSTM) network is used for symptom prediction. Combined with individual differences of patients (such as history of diabetes, hypertension, etc.) and historical trends of blood sugar and blood pressure, the correlation between these factors and ENT symptoms (such as dry throat, cough, etc.) is evaluated in real time, and personalized symptom warnings are output.

[0040] S3.1: Perform local polynomial fitting on physiological data such as blood sugar and blood pressure, extract short-term and long-term trends as new features, and compare these trend parameters with the feature vectors extracted in step S2 Fusion, calibration of short-term and long-term trend of key physiological data (such as blood sugar, blood pressure), forming short-term and long-term trend parameter sets respectively and .

[0041] Specifically, the local polynomial regression method is used to perform trend fitting on blood glucose and blood pressure data to obtain short-term and long-term trend calibration. The formula is as follows: ; in, For the Short-term trend of blood sugar at any time. is the window size, For the Blood sugar data at all times.

[0042] ; in, For the Long-term trend of blood pressure at each moment, is the window size for longer time periods (such as days or weeks), For the Blood pressure data at all times.

[0043] S3.2: Combine the trend parameter set extracted by local fitting with the fused feature vector Splice to form sequence input , constructing input sequences for symptom prediction to capture the joint effects of short-term fluctuations and long-term trends.

[0044] S3.3: Input the sequence The long short-term memory (LSTM) network is input, and the adaptive gating mechanism is used to enhance the model's sensitivity to key data (such as abnormal blood sugar fluctuations) to extract time series features. , and generate symptom prediction output based on individual patient characteristics , represents the probability of occurrence of each ENT symptom, where the symptom prediction output The formula is: ; in, For the The LSTM output state at the moment, and are the weight matrix and bias term used to calculate symptom prediction, respectively. is the activation function.

[0045] S3.4: Prediction confidence based on sequence input The abnormal distribution detection results are adjusted to reduce the impact of noise on the prediction results. The adjustment formula for prediction credibility is: ; in, To predict the reliability, is an adjustable weight, For the The anomaly detection value at a certain moment reflects the degree of abnormal fluctuation in the input sequence.

[0046] S3.5: Based on the confidence level of the prediction and symptom prediction output , output personalized symptom warning parameters: ,in, Provide personalized symptom warning output.

[0047] Provide real-time symptom warnings and assist decision-making based on the patient's individual characteristics.

[0048] S4: When it is predicted that a patient may develop ENT symptoms due to changes in physiological data (such as abnormally high blood sugar or blood pressure fluctuations), health reminders are sent to the patient in real time through smart devices or mobile applications, providing personalized measures for adjusting diet, medication use or medical advice.

[0049] Specifically, based on multimodal data analysis and personalized health records, it is predicted that patients may develop ENT symptoms (such as dry throat, cough, etc.) due to changes in physiological data (such as abnormally high blood sugar, blood pressure fluctuations, etc.). Once it is detected that the changes in these physiological indicators exceed the set threshold, the health reminder mechanism is immediately triggered.

[0050] Health reminders are delivered through smart devices that patients wear daily (such as smart bracelets, smart watches) or mobile health management applications installed on their phones. The devices and applications are synchronized with the patient's health records in real time to ensure that the reminders issued are real-time and personalized.

[0051] Based on individual differences of patients (such as history of diabetes, hypertension, etc.) and current physiological data changes, health reminders are automatically generated based on historical records and personalized algorithms. For example, if a patient's blood sugar rises and is predicted to cause symptoms of dry throat, the reminder may include: It is recommended to reduce sugar intake and increase water intake; Remind to take blood sugar regulating medicine on time; If symptoms do not improve, it is recommended to seek medical attention or undergo further examination as soon as possible.

[0052] After receiving a health reminder, patients can use smart devices or apps to provide feedback on their responses (such as whether they have adjusted their diet or medication), and these feedbacks will serve as the basis for the next prediction and reminder adjustment.

[0053] If the patient does not respond after the reminder, intervene by providing additional reminders or by simplifying the suggestions to ensure that the patient takes the appropriate action in a timely manner.

[0054] Furthermore, this embodiment also provides an otolaryngology symptom monitoring system based on multimodal data fusion, including: The data acquisition module is used to monitor physiological data in real time through wearable devices, retrieve historical medical data from the patient database, collect environmental data, and pre-process the monitored, retrieved and collected data; The feature fusion module is used to fuse the preprocessed data, extract potential related features, and perform multi-level and time-series feature learning through an optimized model architecture combining convolutional neural networks and recurrent neural networks; The symptom prediction module is used to predict symptoms based on the extracted features using an improved long short-term memory network, combined with the historical trends of individual blood sugar and blood pressure of patients, to evaluate the correlation with ENT symptoms in real time and output personalized symptom warnings; The reminder and intervention module is used to send health reminders to patients in real time through smart devices or mobile applications when it is predicted that the patient will develop ENT symptoms due to changes in physiological data, and provide personalized measures such as adjusting diet, medication use or medical advice.

[0055] This embodiment also provides a computer device, which is suitable for the case of an otolaryngology symptom monitoring method based on multimodal data fusion, including a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the otolaryngology symptom monitoring method based on multimodal data fusion as proposed in the above embodiment.

[0056] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covered on the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0057] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for monitoring otolaryngology symptoms based on multimodal data fusion as proposed in the above embodiment is implemented.

[0058] In summary, the present invention is based on multimodal data fusion, and by combining physiological data monitored by wearable devices, historical medical data, and environmental data, it realizes deep data fusion and multi-level feature learning, and can accurately capture various potential symptom causes. Different from the limitations of the prior art, the present invention uses optimized convolutional neural network and recurrent neural network architectures to learn temporal features, thereby improving the accuracy and sensitivity of the model in otolaryngology symptom prediction.

[0059] In addition, through the improved long short-term memory network (LSTM) combined with the patient's individual health characteristics, such as the historical change trends of blood sugar and blood pressure, the present invention can not only perform routine symptom warnings, but also provide personalized health intervention measures according to the individual's health status, such as diet adjustment, drug use recommendations or timely medical reminders, thereby realizing truly intelligent and personalized health management.

[0060] Example 2, referring to Table 1, is the second example of the present invention. This example provides an otolaryngology symptom monitoring method based on multimodal data fusion. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through simulation experiments.

[0061] The experiment selected 10 ENT patients, covering different age groups and disease histories (including patients with hypertension and diabetes), and used smart bracelets and smart home devices to collect data. Specific equipment includes blood sugar monitoring bracelets, blood pressure monitoring bracelets, temperature and humidity sensors, air quality monitoring equipment, etc. The patient's physiological data (such as blood sugar, blood pressure) and environmental data (such as temperature, humidity, air quality) are transmitted to the central processing platform in real time for processing and analysis.

[0062] First, wearable devices such as smart bracelets monitor patients' physiological data (such as blood sugar and blood pressure) in real time, and smart home devices (such as temperature and humidity sensors and air quality monitoring devices) collect home environment parameters. These data are aggregated to the central platform via Bluetooth or Wi-Fi protocols.

[0063] Next, all data are preprocessed, including data cleaning, format standardization, and data synchronization, to ensure data accuracy and time series consistency. Then, the model architecture combining the optimized convolutional neural network (CNN) and the recurrent neural network (RNN) is used to learn the features of the preprocessed data to capture the time series characteristics of multimodal data such as blood sugar fluctuations, blood pressure changes, and environmental parameters.

[0064] In the feature extraction stage, the time series data is processed through LSTM (Long Short-Term Memory Network) to extract the time series features related to the changes in ENT symptoms. Finally, based on the extracted features, the improved LSTM network is used for symptom prediction, and the correlation between these factors and ENT symptoms is evaluated in real time in combination with the patient's individual blood sugar and blood pressure historical trends. When the symptoms are predicted to occur, health reminders will be issued to patients through smart devices or mobile applications, and personalized health advice will be provided.

[0065] In the experiment, the patient's physiological data such as blood sugar and blood pressure and home environment data were regularly collected and transmitted, and the system performed multimodal fusion and prediction on these data. The experimental period was 30 days, aiming to verify the application effect of the system in real-time symptom prediction and personalized health management. Some experimental data are as follows: Table 1 Experimental data Patient ID Blood sugar (unit: mmol / L) Blood pressure (unit: mmHg) Temperature (unit: °C) Humidity (unit: %) Air quality (PM2.5 concentration: μg / m³) Predicted symptoms (dry throat probability: %) 1 6.2 120 / 80 22 60 35 45 2 7.5 130 / 85 24 55 40 50 3 5.8 110 / 75 21 65 30 30 4 6.5 140 / 90 23 70 50 55 5 8.2 125 / 85 25 60 60 65 6 5.9 118 / 78 26 75 38 35 7 7.0 135 / 88 22 80 42 60 8 6.3 120 / 80 19 65 55 40 9 8.0 145 / 92 24 68 48 70 10 6.8 130 / 85 22 72 38 52 From the data in the table above, we can see that the otolaryngology symptom monitoring method based on multimodal data fusion can monitor and evaluate the patient's physiological data and environmental factors (such as blood sugar, blood pressure, temperature, humidity, air quality, etc.) in real time, and predict symptoms based on multidimensional data. The "Predicted Symptoms (Probability of Dry Throat)" column in the table is the prediction result generated by the system through the LSTM network, reflecting the probability of otolaryngology symptoms (such as dry throat) that may occur in each patient due to physiological fluctuations and environmental changes.

[0066] In addition, by analyzing the experimental data, it can be found that environmental factors such as temperature, humidity and air quality have an important impact on the occurrence of ENT symptoms. For example, the probability of dry throat in patient 9 (blood sugar 8.0, blood pressure 145 / 92, air quality PM2.5 concentration 48μg / m³) is 70%. In an environment with high PM2.5 concentration, the predicted results are highly consistent with the actual symptoms.

[0067] From the overall experimental results, the symptom monitoring method based on multimodal data can take into account the interactive effects of multiple factors such as blood sugar, blood pressure, temperature and humidity, and air quality, while traditional methods are often limited to a single physiological data and lack sensitivity to environmental changes. Therefore, the multimodal data fusion method based on the present invention can identify potential symptom changes at an earlier stage and provide personalized health reminders, greatly improving the accuracy and personalization of predictions.

[0068] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An otolaryngology symptom monitoring method based on multimodal data fusion, characterized in that: include: Monitor physiological data in real time through wearable devices, retrieve historical medical data from the patient database, collect environmental data, and pre-process the monitored, retrieved and collected data; Perform feature fusion on the preprocessed data to extract potential correlation features, and perform multi-level and time-series feature learning through an optimized model architecture combining convolutional neural networks and recurrent neural networks; Based on the extracted features, an improved long short-term memory network is used to predict symptoms. Combined with the historical trends of individual blood sugar and blood pressure, the correlation with ENT symptoms is evaluated in real time to output personalized symptom warnings. When it is predicted that a patient develops ENT symptoms due to changes in physiological data, health reminders will be sent to the patient in real time through smart devices or mobile applications, and personalized measures will be provided to adjust diet, medication use or medical advice.

2. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 1, characterized in that: The preprocessing includes data cleaning, data format standardization and data synchronization.

3. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 2, characterized in that: The feature fusion of the preprocessed data to extract potential related features includes: The recurrent neural network structure is used to model the time series of each data stream, capturing the laws of change over time, and combined with LSTM units to handle the dependencies of data over a long time span; The formula of the LSTM unit is: ； in, For the The LSTM hidden state at time , , is the corresponding weight matrix, For the -1 time LSTM hidden state, is the input feature at the current moment, is the bias term, is the activation function; By hiding the state Perform maximum pooling or average pooling to obtain time series features that are highly correlated with symptom changes.

4. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 1, characterized in that: The multi-level, temporal feature learning by combining an optimized convolutional neural network and a recurrent neural network includes: The temporal features extracted from different modalities are fused to perform cross-modal association learning; When fusing cross-modal data, the self-attention mechanism is used to dynamically adjust the influence of different modal data on prediction; Assume that each mode eigenvector is ,in, is the number of modes; The formula of the self-attention mechanism is: ； ； ； in, For the Modal pair The attention weights of modality features, For the Modal pair Modal feature similarity scoring function, is the fused multimodal feature vector, For the The modal eigenvectors, For the The modal eigenvectors, No. Modal eigenvectors.

5. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 4, characterized in that: The method of using an improved long short-term memory network to predict symptoms includes: Perform local polynomial fitting on physiological data to extract short-term and long-term trends as new features; The short-term and long-term trend parameters are compared with the extracted feature vector Fusion is performed to calibrate the short-term and long-term trend of key physiological data, and to form short-term and long-term trend parameter sets respectively. and ; Combine the trend parameter set extracted by local fitting with the fused feature vector Splice to form sequence input , constructing input sequences for symptom prediction, capturing the joint effects of short-term fluctuations and long-term trends.

6. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 5, characterized in that: The method combines the historical trends of individual blood sugar and blood pressure of patients, evaluates the correlation with ENT symptoms in real time, and outputs personalized symptom warnings, including: Enter the sequence into Input long short-term memory network to extract time series features , and generate symptom prediction output based on individual patient characteristics , represents the probability of occurrence of each ENT symptom; The symptom prediction output The formula is: ； in, For the The LSTM output state at the moment, and are the weight matrix and bias term used to calculate symptom prediction, respectively. is the activation function; Prediction Confidence According to the sequence input Adjust the abnormal distribution detection results; According to the prediction reliability and symptom prediction output , output personalized symptom warning parameters.

7. The method for monitoring otolaryngology symptoms based on multimodal data fusion according to claim 6, characterized in that: The adjustment formula for the prediction credibility is: ； in, To predict credibility, is an adjustable weight, For the The abnormal detection value at the moment; The calculation formula of the symptom warning parameter is: ； in, Provide personalized symptom warning output.

8. An otolaryngology symptom monitoring system based on multimodal data fusion, based on the otolaryngology symptom monitoring method based on multimodal data fusion according to any one of claims 1 to 7, characterized in that: include: The data acquisition module is used to monitor physiological data in real time through wearable devices, retrieve historical medical data from the patient database, collect environmental data, and pre-process the monitored, retrieved and collected data; The feature fusion module is used to fuse the preprocessed data, extract potential related features, and perform multi-level and time-series feature learning through an optimized model architecture combining convolutional neural networks and recurrent neural networks; The symptom prediction module is used to predict symptoms based on the extracted features using an improved long short-term memory network, combined with the historical trends of individual blood sugar and blood pressure of patients, to evaluate the correlation with ENT symptoms in real time and output personalized symptom warnings; The reminder and intervention module is used to send health reminders to patients in real time through smart devices or mobile applications when it is predicted that the patient will develop ENT symptoms due to changes in physiological data, and provide personalized measures such as adjusting diet, medication use or medical advice.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the otolaryngology symptom monitoring method based on multimodal data fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the otolaryngology symptom monitoring method based on multimodal data fusion according to any one of claims 1 to 7 are implemented.

Images