Health data analysis method and system based on multi-modal fusion and deep learning

Abstract

The invention discloses a health data analysis method and system based on multi-modal fusion and deep learning. The method comprises the following steps: collecting multi-modal health data of a user through a sensor and camera equipment; performing feature fusion on the image and text data by adopting a progressive fusion method, and inputting an improved Hyper-YOLO model for preliminary classification; if the classification fails, calling a special medical instrument for detection; based on a classification result, through an iTrans-GAHNet model and in combination with cloud drug data, recommending applicable drugs, and displaying a drug use confirmation interface; if the user selects'medication guidance ', dose and taking time guidance is provided; if the known medication method is selected, the process is ended; according to the method, the abnormal detection value is matched with the related disease through the binary classification method, so that disease diagnosis is completed, a large amount of data can be quickly processed, the diagnosis speed and accuracy are improved, medical resources are saved, and the medical service efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of intelligent health prediction technology, and in particular relates to a health data analysis method and system based on multimodal fusion and deep learning. Background Art

[0002] In recent years, with the rapid advancement of medical technology and the widespread application of big data, intelligent health prediction, diagnosis, and drug recommendation systems have gradually become important development directions in the medical industry. The accumulation of multimodal medical data, including high-resolution medical imaging, electronic health records, genomic data, and real-time physiological indicators collected by wearable devices, provides a rich source of information for health assessment. Furthermore, the maturity of artificial intelligence technology has provided strong support for the deep mining and intelligent application of medical data, promoting the development of precision medicine and personalized treatment. Intelligent health systems have shown great potential in early disease prediction, auxiliary diagnosis, and personalized drug recommendations, becoming an important component of the modern medical system.

[0003] Currently, intelligent health prediction, diagnosis, and drug recommendation systems primarily rely on artificial intelligence technologies, particularly the combination of deep learning and knowledge graphs. In terms of data processing, convolutional neural networks (CNNs) are widely used in medical image analysis, efficiently identifying lesion characteristics. Natural language processing (NLP) technology is used to parse unstructured text in electronic medical records and extract key diagnostic information. For drug recommendations, knowledge graph technology combines medical guidelines, drug interactions, and patient history data to generate personalized medication plans. The coordinated application of these technologies enables the system to achieve intelligent integration throughout the entire process, from disease prediction to precise recommendations.

[0004] Despite progress, existing technologies still face numerous challenges. First, the heterogeneity of multimodal data makes fusion difficult. Differences in format, scale, and semantics between different data sources make effective integration challenging. Second, the causes and symptoms of complex diseases are often interrelated or overlapping, and existing algorithms are inadequate in modeling this complexity, potentially leading to misdiagnosis or missed diagnoses. Ultimately, this significantly impacts the accuracy, reliability, and clinical effectiveness of intelligent health prediction and recommendation systems. Summary of the Invention

[0005] Purpose of the invention: The purpose of the present invention is to provide a health data analysis method based on multimodal fusion and deep learning that can improve the accuracy, reliability and clinical application effect of health prediction; on the other hand, to provide a health data analysis system based on multimodal fusion and deep learning.

[0006] Technical solution: The health data analysis method of the present invention comprises the following steps:

[0007] (1) Collecting multimodal health data of users through sensors and cameras, including skin images, electrocardiograms, heart rate, blood oxygen, pulse, and body temperature data, breaks through the limitations of a single data source and builds a multi-dimensional health data system, providing a more comprehensive and accurate data foundation for subsequent analysis;

[0008] (2) A multimodal progressive fusion method is used to fuse the features of the image and text data in the health data, and the fused data is input into the improved Hyper-YOLO model for preliminary classification. The feature extraction strategy can be adaptively adjusted, and the recognition accuracy of common health problems is greatly improved while maintaining real-time response capabilities. This technology effectively reduces the workload of manual review and significantly improves the efficiency of health risk assessment.

[0009] (3) If the initial classification fails, a dedicated medical device is called for testing: in response to the user selecting the "guidance mode", the device usage tutorial in the database is retrieved through the iTrans-GAHNet model; in response to the user selecting the "autonomous mode", the test data of the user's operation is received and input into the Hyper-YOLO model for final classification. The interactive guidance mode provides visual operation guidance through augmented reality technology, which greatly reduces the operating threshold of professional medical equipment; the autonomous detection mode realizes the standardized access of professional data. This hierarchical processing mechanism significantly expands the diagnostic capability range of the system, improves the recognition rate of complex health problems, and provides reliable protection for precision medicine;

[0010] (4) Based on the results of preliminary or final classification, the iTrans-GAHNet model is combined with cloud-based drug data to recommend suitable drugs to users, and a drug taking confirmation interface is displayed, achieving scientific and safe medication recommendations. Intelligent recommendation technology greatly optimizes the medication decision-making process and effectively reduces the risk of irrational medication use;

[0011] (5) In response to the user selecting "Medication guidance required" on the drug taking confirmation interface, the iTrans-GAHNet model provides guidance information on the dosage and time of taking the drug; in response to the user selecting "Known medication method" on the drug taking confirmation interface, the process ends, and medication compliance is ensured through a multimodal reminder mechanism. In combination with feedback data, the guidance strategy is continuously optimized, which significantly improves the patient's correct medication rate, forms a complete health management closed loop, and provides continuous protection for treatment effects.

[0012] Preferably, the multimodal progressive fusion described in step 2 includes:

[0013] Build a basic model to receive multimodal input x=(x1,x2,...,x k ), where x iRepresents the input of the i-th modality, the basic model includes: single modal feature generator Among them, X i represents the i-th input; Represents X i dimensional space, d i is the characteristic dimension of the i-th mode, which is used to process the input data x of each mode separately i And generate the corresponding feature representation G i (x i ); fusion layer Used to fuse the feature representations of each modality into a common representation F(G1(x1),G2(x2),...,G K (x K ));

[0014] Introduce the context vector c∈R on the base model d , forming an enhanced model Where X represents the original input; Y represents the output. When c = 0, the output of the enhanced model is the same as that of the basic model, that is,

[0015] The output F of the fusion layer is connected through the weight matrix or function W i Projected back to the network as the context vector c t Feedback to the unimodal feature generator G i , where the enhanced unimodal feature generator The input is Among them, c i-1 The output of the fusion layer F is passed through the matrix W i The projected vector; the original input x i and the transformed context vector W i (c t-1 ) are added as an enhanced unimodal feature generator Input;

[0016] Set the initial context vector c0 = 0, perform R iterations to gradually refine the representation, and at the tth iteration, calculate as follows:

[0017]

[0018] Where K is the total number of modes, x K is the original input of each modality;

[0019] The context vector c obtained by the final iteration R Passed as a fusion representation to the improved Hyper-YOLO model for output

[0020] By constructing a collaborative architecture of basic models and enhanced models, efficient fusion of multi-source health data is achieved. This method innovatively introduces an iteratively optimized context vector mechanism, and dynamically feeds back the feature representation of the fusion layer to each single-modal feature extractor through reverse connection to form a closed-loop optimization system. It adopts a multi-round iterative progressive fusion strategy to continuously refine the feature representation of different modalities in the process of hierarchical interaction, and finally generates a unified feature representation rich in cross-modal correlation information. It effectively solves the problems of inaccurate feature matching and large information loss in traditional multimodal fusion, significantly improves the integrity and discriminability of feature expression, and lays a solid foundation for the subsequent accurate classification of the Hyper-YOLO model while maintaining the advantage of computational efficiency.

[0021] Preferably, the improved Hyper-YOLO model described in step 2 includes:

[0022] The backbone network uses a hybrid aggregation network MANet as the core computing module. The MANet includes: a 1×1 bypass convolution module for channel-level feature recalibration; a depthwise separable convolution DSConv module for spatial feature processing; and a C2f module for feature level integration. The formula is as follows:

[0023]

[0024] Among them, X mid is the intermediate feature map, X in is the input feature map, Conv is the convolution layer, DSConv is the depth-separable convolution, Split is the segmentation, ConvNeck is the convolution neck module, X mid The number of channels is 2c, X1, X2, ..., X 4+n The number of channels is c, and the three modules of MANet are fused through splicing operation and compressed through 1×11×1 convolution to generate an output X with a channel number of 2c out ;

[0025] The neck network includes a hypergraph-based cross-level and cross-position representation network HyperC2Net, which is used to capture high-order correlations between features;

[0026] The hypergraph convolutional layer introduces the ECA attention mechanism to dynamically allocate channel weights by adaptively selecting the kernel size of channel attention.

[0027] Through the innovative hybrid aggregation network (MANet) architecture, efficient extraction and fusion of multi-scale features are achieved. The model adopts the collaborative design of 1×1 bypass convolution, depthwise separable convolution and C2f module, which significantly improves the feature expression ability while maintaining computational efficiency; the innovative HyperC2Net neck network models the high-order correlation between features through a hypergraph structure, combined with the adaptive channel weighting strategy of the ECA attention mechanism, enables the model to dynamically focus on key feature areas. This composite architecture design effectively enhances the model's ability to represent multimodal health data, greatly improves the classification accuracy while ensuring real-time performance, and provides a more reliable decision-making basis for health risk assessment.

[0028] Preferably, the hypergraph construction process of the HyperC2Net network includes:

[0029] The backbone network is divided into five discrete stages, and the feature maps {B1, B2, B3, B4, B5} corresponding to the five discrete stages are concatenated at the channel level to form cross-level visual features;

[0030] Define a hypergraph G = (V, E) by the vertex set V and the hyperedge set E, and use the concatenated feature point set as the hypergraph vertex set V;

[0031] For each vertex v∈V, an ε-ball is constructed as a hyperedge ε∈E with a preset distance threshold, so that ε contains all feature points within the distance threshold from the vertex. The construction of the overall hyperedge set can be defined as:

[0032] E={ball(v,ε)|v∈V},ball(v,ε)={u|||x u -x v || d <ε,u∈V}

[0033] Among them, E represents the neighbor vertex set of the specified vertex v, ||xy|| d is the distance function, x u and x v The eigenvectors representing vertices u and v;

[0034] The hypergraph convolution process of the HyperC2Net network includes:

[0035] Utilize spatial domain hypergraph convolution and add additional residual connections to perform high-order learning of vertex features. The formula is as follows:

[0036]

[0037] Among them, N v (e) and N e (v) are two neighbor exponential functions, N v(e) = {v|v∈e,v∈V}, N e (v) = {e|v∈e,e∈E}, Θ is a learnable weight matrix used to perform linear transformation on features;

[0038] Construct a matrix for hypergraph information passing:

[0039]

[0040] Among them, D v and D e Represent the diagonal matrices of vertices and hyperedges respectively, X is the vertex feature matrix, H is the hypergraph association matrix; HyperConv is the hypergraph convolution.

[0041] Through innovative hypergraph construction and convolution mechanisms, efficient modeling and information transmission of cross-level visual features are achieved, the spatial correlation between feature points is effectively captured, the network's modeling ability of complex feature relationships is significantly enhanced, and the optimal propagation of feature information in the hypergraph structure is achieved, enabling the model to fully explore the deep correlation between multi-scale features, greatly improving the discriminability and robustness of feature representation, and providing more discriminative feature representation for subsequent classification tasks.

[0042] Preferably, the overall definition of the HyperC2Net network is as follows:

[0043]

[0044] Among them, Φ is the fusion function, X mixed is the fusion of backbone features from multiple levels, including cross-level information, || represents splicing; X hyper To use hypergraph convolution to fused X mixed The result after processing; H is the correlation matrix of the hypergraph; the feature X after the hypergraph convolution hyper r is fused with feature maps B3, B4, and B5 at different levels to obtain results N3, N4, and N5.

[0045] Through the hypergraph feature fusion architecture, efficient integration and enhancement of cross-level visual features are achieved. This hierarchical progressive processing method not only retains the detailed information of features at different scales, but also captures deep semantic associations through hypergraph learning, so that the final output feature representation has both local accuracy and global consistency, significantly improving the model's ability to represent complex health data and providing a more discriminative feature basis for accurate classification.

[0046] Preferably, the implementation of the ECA attention mechanism includes the following steps:

[0047] Perform global average pooling on the feature map, compress the feature map of each channel into a scalar, and obtain a feature vector F with a shape of 1×1×C gap , where C is the number of channels;

[0048] The compressed feature vector F is then converted to gap Perform one-dimensional convolution operation F conv =Conv1D(F gap ,,k), where the convolution kernel size k is adaptively determined according to the number of channels C:

[0049]

[0050] Among them, γ is the scaling parameter and δ is the offset;

[0051] The convolution result F conv Generate weight vector F through Sigmoid activation function eca =σ(F conv );

[0052] The generated weight vector F eca Multiply F by channel with the original feature map F out =F⊙F eca , complete feature recalibration.

[0053] Through the adaptive convolution kernel strategy, efficient dynamic weighting of channel features is achieved, avoiding the limitations of manually setting a fixed convolution kernel size. By introducing learnable scaling parameters and offsets, the network can intelligently adjust the receptive field range according to the complexity of the features. Finally, Sigmoid activation is used to generate discriminative channel weights. This design not only retains the computational efficiency advantages of the traditional attention mechanism, but also significantly improves the model's ability to focus on important channel features, making the feature recalibration process more accurate and effectively enhancing the network's feature selectivity and representation capabilities.

[0054] Preferably, the step 3 of retrieving the instrument usage tutorial from the database through the iTrans-GAHNet model includes:

[0055] Encode the patient's electronic health record (EHR) data based on the Transformer architecture to generate a comprehensive state representation vector for the patient;

[0056] Construct medical device association diagrams, including medical device usage frequency diagrams and medical device compatibility diagrams;

[0057] Performing matching calculations on the comprehensive state representation vector and the medical device association graph to determine the type of medical device suitable for the patient;

[0058] The usage tutorial of the corresponding medical device is retrieved from the database according to the result of the matching calculation, and the tutorial includes a video demonstration of the operation steps and instructions for precautions.

[0059] Through the fusion architecture of medical knowledge graph and Transformer, intelligent recommendation and guidance on the use of medical devices are realized. This method breaks through the static mode of traditional medical equipment operation guidance and realizes personalized recommendations based on individual differences of patients. It significantly improves the safety and convenience of medical device use and effectively reduces medical risks caused by improper operation.

[0060] Preferably, the learning module of the iTrans-GAHNet model described in step 4 includes:

[0061] Use the Transformer network to encode the patient's electronic health record (EHR) data and generate a patient representation vector;

[0062] Convert the patient's diagnosis and procedure information into embedding vectors using the embedding table:

[0063] d i,t =x i,t W d

[0064] p i,t =y i,t W p

[0065] Among them, d i,t and p i,t Represents the embedding vector at each moment, x i,t and y i,t are the multi-hot encoded vectors of the diagnosis and procedure of the i-th patient at the t-th visit, W d and W p is the embedding matrix;

[0066] The Transformer encoder captures different position information in the collected data sequence through a multi-head self-attention mechanism:

[0067]

[0068] MHA(Q,K,V)=Concat(head1,...,head h )W O

[0069] Among them, Q represents query, K represents key, and V represents value; Softmax function is used for normalization, MHA is multi-head attention, head i =Attention(QW i Q,KW i K ,VW i V ), h is the number of heads.

[0070] Each encoder layer consists of a feed-forward network:

[0071] FFN(X)=ReLU(XW1+b1)W2+b2

[0072]

[0073] Among them, FFN is a feedforward network; W is the weight matrix, b is the bias term, and are the embedding vectors of diagnosis and program respectively, and LayerNorm is layer normalization;

[0074] Combine the diagnosis and procedure representation vectors into the final patient representation vector:

[0075]

[0076] Among them, NN is a feedforward neural network.

[0077] Through the innovative multimodal Transformer architecture, deep representation learning of patient health data is achieved. The module adopts a dual-path embedding mechanism of diagnosis and procedure, and converts discrete medical events into continuous semantic vectors through multi-hot encoding conversion and embedding matrix mapping; it uses a multi-head self-attention mechanism to dynamically capture the temporal dependencies and clinical context information in medical records; it combines feedforward networks and layer normalization for feature enhancement and stable training, and finally generates a semantically rich patient comprehensive state vector by fusing the dual-path representation of diagnosis and procedure through a neural network. This vector not only retains the fine-grained information of the original medical data, but also extracts high-level clinical features, providing an accurate patient portrait basis for subsequent drug recommendations and medical decision-making, and significantly improving the personalization and precision of medical services.

[0078] Preferably, the drug generation module of the iTrans-GAHNet model described in step 4 includes:

[0079] Construct EHR graphs and DDI graphs to capture drug co-occurrence and drug interactions, respectively:

[0080]

[0081] in, and are the adjacency matrices of the EHR graph and the DDI graph, and is the corresponding degree matrix;

[0082] A two-layer GCN model was used to build a drug co-occurrence model based on the EHR graph. In addition, another two-layer GCN was used based on the DDI graph to model the DDIs that should be prevented in drug prescriptions.

[0083] Construct a graph convolutional network and use two layers of GCN to encode the EHR graph and DDI graph:

[0084]

[0085] Among them, X e and X d is the initial feature matrix of the drug;

[0086] Retrieve the most relevant drug combinations from dynamic memory via the attention mechanism:

[0087]

[0088] in, and It is the drug combination retrieved from the EHR map and DDI map;

[0089] Construct a bipartite graph of drugs and molecules to control potential drug interactions:

[0090] m i,t =Sigmoid(W1h i,t )

[0091]

[0092] Among them, m i,t is a value between 0 and 1 generated by the Sigmoid activation function. MIT and Hm are multiplied element by element and then linearly changed through the learnable parameter matrix to obtain H m is the characteristic matrix of drug molecules, W1 and W2 are learnable parameter matrices;

[0093] Based on the bipartite graph of drugs and molecules, combined with patient representation and drug representation, the final drug recommendation is generated:

[0094]

[0095] in, is the predicted drug combination.

[0096] Through the multi-graph neural network architecture, safe and effective personalized drug recommendations are achieved. This module innovatively constructs a dual-graph structure of EHR drug co-occurrence graph and DDI drug interaction graph, and models drug usage patterns and potential risks respectively through a two-layer GCN; a dynamic memory attention mechanism is used to intelligently retrieve the optimal drug combination; a specially designed drug-molecule bipartite graph mechanism uses a learnable parameter matrix and Sigmoid gating to accurately control the molecular-level interactions between drugs, and ultimately intelligently integrates the deep representation of patients with multi-dimensional drug characteristics to generate personalized prescription plans that meet individual patient needs and avoid adverse drug reactions, significantly improving medication safety and treatment effects, and providing reliable intelligent support for clinical decision-making.

[0097] The health data analysis system of the present invention includes:

[0098] Multimodal data acquisition module, used to collect the user's multimodal health data through sensors and cameras, including skin images, electrocardiogram, heart rate, blood oxygen, pulse and body temperature data;

[0099] A data fusion and classification module uses a multimodal progressive fusion method to fuse the features of the image and text data in the health data, and inputs the fused data into an improved Hyper-YOLO model for preliminary classification;

[0100] The medical device interaction module is used to call a dedicated medical device for testing when the initial classification fails: in response to the user selecting "guided mode", the device usage tutorial in the database is called through the iTrans-GAHNet model; in response to the user selecting "autonomous mode", the test data operated by the user is received and input into the Hyper-YOLO model for final classification;

[0101] The drug recommendation module recommends suitable drugs to users based on the results of preliminary or final classification through the iTrans-GAHNet model combined with cloud-based drug data, and displays a drug intake confirmation interface;

[0102] The medication guidance module responds to the user selecting "Need medication guidance" on the medication confirmation interface and provides guidance information on the dosage and time of administration of the drug through the iTrans-GAHNet model; responds to the user selecting "Known medication method" on the medication confirmation interface and ends the process.

[0103] Beneficial effects: Compared with the existing technology, the present invention has the following significant advantages: 1. Through the binary classification method, the abnormal detection values ​​are matched with related diseases to complete the disease diagnosis, which can quickly process large amounts of data, improve the diagnosis speed and accuracy, save medical resources, and improve the efficiency of medical services; 2. Through hierarchical feature fusion, it effectively avoids information loss and feature inconsistency problems, and improves model performance; 3. The iTrans-GAHNet model combines Transformer and graph attention mechanisms to accurately mine patient data associations and improve the accuracy and personalization of drug recommendations; 4. The system provides guidance on the use of equipment and instructions for medication to help patients operate and take medication correctly, enhance compliance, and reduce medication risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] Figure 1 is a flow chart of the present invention;

[0105] Figure 2 It is a model flow chart of the present invention;

[0106] Figure 3 This is a structural diagram of the Hyper-yolo model of the present invention;

[0107] Figure 4 is a logic diagram of the present invention;

[0108] Figure 5 This is the structural diagram of the iTrans-GAHNet model of the present invention. DETAILED DESCRIPTION

[0109] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0110] like Figure 1 As shown, the present invention includes the following steps:

[0111] Step 1: Use a high-definition camera to collect human skin images and facial emotion maps; use an electrocardiogram (ECG) sensor to collect electrocardiogram and heart rate; use a PPG sensor to collect blood oxygen and pulse; and use a temperature sensor to collect the patient's body temperature.

[0112] Step 2: Multimodal progressive fusion is a model-independent technique for multimodal fusion that aims to improve the expressive power of representations by transferring late fusion representations to early layers through reverse connections through iterative representation refinement. The detailed process is as follows: Figure 2 The following are the main steps of the progressive fusion process:

[0113] First, define the basic model F to receive multimodal input x=(x1,x2,...,x k ), where x iRepresents the input of the i-th modality. The basic model consists of a single-modal feature generator G i :X i →Rd i and fusion layer Each unimodal feature generator G i Responsible for processing the input x of the corresponding modality i , and generate feature representation G i (x i ). The fusion layer F fuses the feature representations of all modalities into a common representation F(G1(x1),G2(x2),...,G K (x K )).

[0114] Introducing reverse connection: Enhanced model, based on the basic model, introduces an additional context vector c∈R d As input, form the enhanced model When c = 0, the output of the enhanced model is the same as that of the basic model, that is,

[0115] The output F of the fusion layer is passed through the matrix or function W i Projected back to the network as the context vector c t Provided to the unimodal feature generator G i Specifically, the enhanced unimodal feature generator The input is

[0116] Starting from the initial context vector c0=0, the representation is gradually refined through multiple iterations. In the tth iteration, the calculation is as follows:

[0117]

[0118] After R iterations, the final context vector c R As a fusion representation, it is passed to the prediction model P for output:

[0119]

[0120] Step 3: The overall framework of the Hyper-yolo model includes a backbone network and a neck network. Given an input image, the backbone network of the model uses MANet as the core computing module, thereby enhancing the feature recognition ability of the traditional C2f module in YOLOv8. The model structure is as follows Figure 3 shown.

[0121] In the backbone network of Hyper-yolo, a hybrid aggregation network (MANet) is designed to enhance the feature extraction capability of the basic network. This architecture organically combines three typical convolution variants: 1×1 bypass convolution for recalibration of channel-level features; depthwise separable convolution (DSConv) for efficient spatial feature processing; and C2f module for enhanced feature level integration.

[0122] This fusion produces richer and more diverse gradient flows during training, significantly enhancing the semantic depth contained in the basic features within the five key stages.

[0123] The hybrid aggregation network (MANet) is the core module of the Hyper-yolo backbone network, which is used to enhance the feature extraction capability. Its formula is as follows:

[0124]

[0125] Among them, X mid The number of channels is 2c, and X1,X2,...,X 4+n The number of channels is c.

[0126] Finally, the three types of features are fused through the splicing operation and compressed through 1×11×1 convolution to generate an output X with a channel number of 2c. out .

[0127] To fully integrate cross-level and cross-position information from the backbone network, we introduce a hypergraph-based cross-level and cross-position representation network (HyperC2Net). HyperC2Net is an implementation of the proposed HGC-SCS framework that can capture potential high-order relationships in the semantic space.

[0128] Hypergraph construction:

[0129] In the Hyper-yolo framework, the backbone network is divided into five discrete stages. The feature graphs of these stages are represented as {B1, B2, B3, B4, B5}. In order to use hypergraph computing to clarify the complex high-order relationships between basic features, cross-level visual features are synthesized by channel-level splicing of the five basic features. The hypergraph G = (V, E) is usually defined by its vertex set V and hyperedge set E. We deconstruct the grid-based visual features to form the vertex set V of the hypergraph. In order to model the neighborhood relationship in the semantic space, an ε ball is constructed from each feature point using a distance threshold. The ε ball will serve as a hyperedge, containing all feature points within a certain threshold from the center point and the feature point. The construction of the overall hyperedge set can be defined as:

[0130] E={ball(v,ε)|v∈V},ball(v,ε)={u|||x u -xv || d <ε,u∈V}

[0131] E represents the set of neighbor vertices of the specified vertex v. ||xy|| d is the distance function. In computation, the hypergraph G is usually represented by its incidence matrix H.

[0132] Hypergraph Convolution:

[0133] In order to promote high-order information transmission on the hypergraph structure, the typical spatial domain hypergraph convolution is used and additional residual connections are added to perform high-order learning of vertex features, as follows:

[0134]

[0135] where N v (e) and N e (v) are two neighbor exponential functions, as defined in the HGNN+ definition: N v (e) = {v|v∈e,v∈V} and N e (v) = {e|v∈e,e∈E}. For ease of calculation, the matrix expression of the two-stage hypergraph information transfer can be defined as:

[0136]

[0137] Among them, D v and D e Represent the diagonal matrices of vertices and hyperedges respectively.

[0138] The ECA attention mechanism is introduced. The ECA attention mechanism is an efficient channel attention mechanism. Its core idea is to dynamically assign weights to each channel by adaptively selecting the kernel size of the channel attention. The main steps of the ECA module include:

[0139] Global average pooling: Perform global average pooling on the feature map to compress the feature map of each channel into a scalar, obtaining a feature vector with a shape of (1×1×C), where C is the number of channels.

[0140] Adaptive convolution kernel: The compressed feature vector F is processed through an adaptive convolution kernel. gap The compressed feature vector is processed through an adaptive convolution kernel. The core of the ECA module is to adaptively select the size k of the convolution kernel:

[0141]

[0142] Among them, C is the number of channels, γ is a scaling parameter, usually takes a value of 2, and δ is an offset, usually takes a value of 1.

[0143] F gap Perform a one-dimensional convolution operation with a convolution kernel size of k:

[0144] F conv =Conv1D(F gap ,k)

[0145] Sigmoid activation function: The feature vector F after the convolution operation conv Through the Sigmoid activation function, a weight vector F is generated in the range [0,1]. eca :

[0146] F eca =σ(F conv )

[0147] Among them, σ is the Sigmoid activation function,

[0148] Feature recalibration: The generated weight vector F eca Multiply the original feature map F channel by channel to complete the feature recalibration:

[0149] F out =F⊙F eca

[0150] By combining the hypergraph construction and convolution strategies defined above, we introduce a simplified instance of the HGC-SCS framework, called Hypergraph-based Cross-Level and Cross-Position Representation Network (HyperC2Net), which is generally defined as follows:

[0151]

[0152] Among them, Φ is the fusion function, in HyperC2Net, X mixed It inherently contains cross-level information because it is a fusion of backbone features from multiple levels.

[0153] Step 4: If the condition cannot be diagnosed in step 2, a dedicated medical device is used for further examination, such as blood sugar, blood pressure, cholesterol, etc. For patients who are not familiar with the use of the device, the system calls the iTrans-GAHNet model and provides a tutorial, such as Figure 4 As shown;

[0154] The Transformer architecture is used to model the patient's EHR data, specifically the data from the initial test in Step 1, to generate a comprehensive representation of the patient's status. This is used to determine whether the patient is suitable for a specific medical device. Medical device-related graphs, such as device usage frequency and compatibility graphs, are constructed to enhance the model's understanding of medical device usage scenarios. Based on the patient's individual characteristics and historical records, appropriate medical device usage tutorials are recommended.

[0155] If the patient is familiar with the detection function and usage of the device, the device inspection data will be input into the Hyper-yolo model for disease diagnosis.

[0156] Step 5: iTrans-GAHNet is a novel drug recommendation model that combines the Transformer architecture and multi-graph enhancement strategy to generate safe and personalized drug combinations. The model structure is as follows: Figure 5 shown.

[0157] Transformer incorporates an inverted dimension design: Patient electronic health records (EHRs) contain multiple variables, each of which changes over time, such as heart rate monitoring, electrocardiogram (ECG), and electroencephalogram (EEG). iTransformer independently embeds the time series of each variable into a "variate token," then performs self-attention on these variable tokens and applies a self-attention mechanism to capture correlations between multiple variables. Each variable token integrates information from other variables through the self-attention mechanism.

[0158] The dimensionality-inverting approach avoids potential delay events and confounding of different physical measurements caused by fusing multiple variables in traditional methods.

[0159] The model is mainly divided into two modules:

[0160] Patient representation learning module: Use the Transformer network to encode the patient's electronic health record (EHR) data and generate a patient representation vector. Each patient's visit record contains diagnosis and procedure information, which is converted into an embedding vector using the embedding table:

[0161] d i,t =x i,t W d

[0162] p i,t =y i,t W p

[0163] Among them, x i,t and y i,tare the multi-hot encoded vectors of the diagnosis and procedure of the i-th patient at the t-th visit, W d and W p is the embedding matrix.

[0164] The Transformer encoder captures information at different positions in the sequence through a multi-head self-attention mechanism:

[0165]

[0166] MHA(Q,K,V)=Concat(head1,...,head h )W O

[0167] Among them, head i =Attention(QW i Q ,KW i K ,VW i V ), h is the number of heads.

[0168] Each encoder layer also consists of a feed-forward network:

[0169] FFN(X)=ReLU(XW1+b1)W2+b2

[0170]

[0171] Combine the diagnosis and procedure representation vectors into the final patient representation vector:

[0172]

[0173] Among them, NN is a feedforward neural network.

[0174] The second is the drug generation module. Some drugs may interact with each other when taken together, leading to worsening health or even death. In particular, drug-drug interactions (DDIs) are more difficult to prevent than adverse reactions of a single drug.

[0175] Therefore, for patients with complex health conditions such as multiple uncontrolled chronic diseases, it is crucial for doctors to consider medication safety in prescription planning. Multiple graphs (electronic health record (EHR) graph, drug-drug interaction (DDI) graph, and drug molecule bipartite graph) are constructed to enhance the representation of drug information and generate safe drug combinations.

[0176] Construct EHR graphs and DDI graphs to capture drug co-occurrence and drug interactions, respectively:

[0177]

[0178] in, and are the adjacency matrices of the EHR graph and the DDI graph, and is the corresponding degree matrix.

[0179] A two-layer GCN model was used to build a drug co-occurrence model based on the EHR graph. In addition, another two-layer GCN was used based on the DDI graph to model the DDIs that should be prevented in drug prescriptions.

[0180] Construct a graph convolutional network and use two layers of GCN to encode the EHR graph and DDI graph:

[0181]

[0182] Among them, X e and X d is the initial feature matrix of the drug.

[0183] Retrieve the most relevant drug combinations from dynamic memory via the attention mechanism:

[0184]

[0185] in, and are drug combinations retrieved from the EHR graph and the DDI graph.

[0186] In order to fully capture the impact of explicit and implicit DDIs on safe drug recommendations, a bipartite graph of drugs and molecules is constructed to control potential drug interactions:

[0187] m i,t =Sigmoid(W1h i,t )

[0188]

[0189] Among them, H m is the feature matrix of the drug molecule, W1 and W2 are the learnable parameter matrices.

[0190] By generating patient representations and memory outputs that match the patient, personalized drug combinations can be achieved. Thanks to the bipartite graph built on drugs and their molecules, the patient representation and drug representation are combined to generate the final drug recommendation:

[0191]

[0192] in, is the predicted drug combination.

[0193] Step 6: If the patient is not taking medication correctly, detailed guidance is provided using the iTrans-GAHNet model. The model accurately determines the appropriate dosage and schedule for each patient's specific condition, ensuring safe and effective medication use. If the patient is taking medication correctly, the diagnostic process ends here.

Claims

1. A health data analysis method based on multimodal fusion and deep learning, characterized in that: The following steps are involved: (1) Collect multimodal health data of users through sensors and cameras, including skin images, electrocardiogram, heart rate, blood oxygen, pulse and body temperature data; (2) using a multimodal progressive fusion method to fuse features of the image and text data in the health data, and inputting the fused data into an improved Hyper-YOLO model for preliminary classification; (3) If the preliminary classification fails, a dedicated medical device is called for testing: in response to the user selecting the "guidance mode", the device usage tutorial in the database is called through the iTrans-GAHNet model; in response to the user selecting the "autonomous mode", the test data operated by the user is received and input into the Hyper-YOLO model for final classification; (4) Based on the results of the preliminary or final classification, the iTrans-GAHNet model is combined with cloud-based drug data to recommend suitable drugs to users, and a drug administration confirmation interface is displayed; (5) In response to the user selecting "Medication guidance required" on the drug taking confirmation interface, the iTrans-GAHNet model provides guidance information on the dosage and taking time of the drug; in response to the user selecting "Known medication method" on the drug taking confirmation interface, the process ends.

2. The health data analysis method according to claim 1, characterized in that: The multimodal progressive fusion described in step 2 includes: Build a basic model to receive multimodal input x=(x1,x2,...,x k ), where x i Represents the input of the i-th modality, the basic model includes: single modal feature generator Among them, X i represents the i-th input; Represents X i dimensional space, d i is the characteristic dimension of the i-th mode, which is used to process the input data x of each mode separately i , and generate the corresponding feature representation G i (x i ); fusion layer Used to fuse the feature representations of each modality into a common representation F(G1(x1),G2(x2),...,G K (x K )); Introduce the context vector c∈R on the base model d , forming an enhanced model Where X represents the original input; Y represents the output. When c = 0, the output of the enhanced model is the same as that of the basic model, that is, The output F of the fusion layer is connected through the weight matrix or function W i Projected back to the network as the context vector c t Feedback to the unimodal feature generator G i , where the enhanced unimodal feature generator The input is Among them, c i-1 The output of the fusion layer F is passed through the matrix W i The projected vector; the original input x i and the transformed context vector W i (c t-1 ) are added as an enhanced unimodal feature generator Input; Set the initial context vector c0 = 0, perform R iterations to gradually refine the representation, and at the tth iteration, calculate as follows: Where K is the total number of modes, x K is the original input of each modality; The context vector c obtained by the final iteration R Passed as a fusion representation to the improved Hyper-YOLO model for output 3. The health data analysis method according to claim 1, characterized in that: The improved Hyper-YOLO model described in step 2 includes: The backbone network uses a hybrid aggregation network MANet as the core computing module. The MANet includes: a 1×1 bypass convolution module for channel-level feature recalibration; a depthwise separable convolution DSConv module for spatial feature processing; and a C2f module for feature level integration. The formula is as follows: Among them, X mid is the intermediate feature map, X in is the input feature map, Conv is the convolution layer, DSConv is the depth-separable convolution, Split is the segmentation, ConvNeck is the convolution neck module, X mid The number of channels is 2c, X1, X2, ..., X 4+n The number of channels is c, and the three modules of MANet are fused through splicing operation and compressed through 1×11×1 convolution to generate an output X with a channel number of 2c out ; The neck network includes a hypergraph-based cross-level and cross-position representation network HyperC2Net, which is used to capture high-order correlations between features; The hypergraph convolutional layer introduces the ECA attention mechanism to dynamically allocate channel weights by adaptively selecting the kernel size of channel attention.

4. The health data analysis method according to claim 3, characterized in that: The hypergraph construction process of the HyperC2Net network includes: The backbone network is divided into five discrete stages, and the feature maps {B1, B2, B3, B4, B5} corresponding to the five discrete stages are concatenated at the channel level to form cross-level visual features; Define a hypergraph G = (V, E) by the vertex set V and the hyperedge set E, and use the concatenated feature point set as the hypergraph vertex set V; For each vertex v∈V, an ε-ball is constructed as a hyperedge ε∈E with a preset distance threshold, so that ε contains all feature points within the distance threshold from the vertex. The construction of the overall hyperedge set can be defined as: E={ball(v,ε)|v∈V},ball(v,ε)={u|||x u -x v || d <ε,u∈V} Among them, E represents the neighbor vertex set of the specified vertex v, ||xy|| d is the distance function, x u and x v The eigenvectors representing vertices u and v; The hypergraph convolution process of the HyperC2Net network includes: Utilize spatial domain hypergraph convolution and add additional residual connections to perform high-order learning of vertex features. The formula is as follows: Among them, N v (e) and N e (v) are two neighbor exponential functions, N v (e) = {v|v∈e,v∈V}, N e (v) = {e|v∈e,e∈E}, Θ is a learnable weight matrix used to perform linear transformation on features; Construct a matrix for hypergraph information passing: Among them, D v and D e Represent the diagonal matrices of vertices and hyperedges respectively, X is the vertex feature matrix, H is the hypergraph association matrix; HyperConv is the hypergraph convolution.

5. The health data analysis method according to claim 4, characterized in that: The overall definition of the HyperC2Net network is as follows: Among them, Φ is the fusion function, X mixed is the fusion of backbone features from multiple levels, including cross-level information, || represents splicing; X hyper To use hypergraph convolution to fused X mixed The result after processing; H is the correlation matrix of the hypergraph; the feature X after the hypergraph convolution hyper r is fused with feature maps B3, B4, and B5 at different levels to obtain results N3, N4, and N5.

6. The health data analysis method according to claim 3, characterized in that: The implementation of the ECA attention mechanism includes the following steps: Perform global average pooling on the feature map, compress the feature map of each channel into a scalar, and obtain a feature vector F with a shape of 1×1×C gap , where C is the number of channels; The compressed feature vector F is then converted to gap Perform one-dimensional convolution operation F conv =Conv1D(F gap ,,k), where the convolution kernel size k is adaptively determined according to the number of channels C: Among them, γ is the scaling parameter and δ is the offset; The convolution result F conv Generate weight vector F through Sigmoid activation function eca =σ(F conv ); The generated weight vector F eca Multiply F by channel with the original feature map F out =F⊙F eca , complete feature recalibration.

7. The health data analysis method according to claim 1, characterized in that: The instrument usage tutorials retrieved from the database using the iTrans-GAHNet model described in step 3 include: Encode the patient's electronic health record (EHR) data based on the Transformer architecture to generate a comprehensive state representation vector for the patient; Construct medical device association diagrams, including medical device usage frequency diagrams and medical device compatibility diagrams; Performing matching calculations on the comprehensive state representation vector and the medical device association graph to determine the type of medical device suitable for the patient; The usage tutorial of the corresponding medical device is retrieved from the database according to the result of the matching calculation, and the tutorial includes a video demonstration of the operation steps and instructions for precautions.

8. The health data analysis method according to claim 1, characterized in that: The learning module of the iTrans-GAHNet model described in step 4 includes: Use the Transformer network to encode the patient's electronic health record (EHR) data and generate a patient representation vector; Convert the patient's diagnosis and procedure information into embedding vectors using the embedding table: d i,t =x i,t W d p i,t =y i,t W p Among them, d i,t and p i,t Represents the embedding vector at each moment, x i,t and y i,t are the multi-hot encoded vectors of the diagnosis and procedure of the i-th patient at the t-th visit, W d and W p is the embedding matrix; The Transformer encoder captures different position information in the collected data sequence through a multi-head self-attention mechanism: MHA(Q,K,V)=Concat(head1,...,head h )W O Among them, Q represents query, K represents key, and V represents value; Softmax function is used for normalization, MHA is multi-head attention, head i =Attention(QW i Q ,KW i K ,VW i V ), h is the number of heads. Each encoder layer consists of a feed-forward network: FFN(X)=ReLU(XW1+b1)W2+b2 Among them, FFN is a feedforward network; W is the weight matrix, b is the bias term, and are the embedding vectors of diagnosis and program respectively, and LayerNorm is layer normalization; Combine the diagnosis and procedure representation vectors into the final patient representation vector: Among them, NN is a feedforward neural network.

9. The health data analysis method according to claim 1, characterized in that: The drug generation module of the iTrans-GAHNet model described in step 4 includes: Construct EHR graphs and DDI graphs to capture drug co-occurrence and drug interactions, respectively: in, and are the adjacency matrices of the EHR graph and the DDI graph, and is the corresponding degree matrix; A two-layer GCN model was used to build a drug co-occurrence model based on the EHR graph. In addition, another two-layer GCN was used based on the DDI graph to model the DDIs that should be prevented in drug prescriptions. Construct a graph convolutional network and use two layers of GCN to encode the EHR graph and DDI graph: Among them, X e and X d is the initial feature matrix of the drug; Retrieve the most relevant drug combinations from dynamic memory via the attention mechanism: in, and It is the drug combination retrieved from the EHR map and DDI map; Construct a bipartite graph of drugs and molecules to control potential drug interactions: m i,t =Sigmoid(W1h i,t ) Among them, m i,t is a value between 0 and 1 generated by the Sigmoid activation function. MIT and Hm are multiplied element by element and then linearly changed through the learnable parameter matrix to obtain H m is the characteristic matrix of drug molecules, W1 and W2 are learnable parameter matrices; Based on the bipartite graph of drugs and molecules, combined with patient representation and drug representation, the final drug recommendation is generated: in, is the predicted drug combination.

10. A health data analysis system based on multimodal fusion and deep learning, characterized in that: include: Multimodal data acquisition module, used to collect the user's multimodal health data through sensors and cameras, including skin images, electrocardiogram, heart rate, blood oxygen, pulse and body temperature data; A data fusion and classification module uses a multimodal progressive fusion method to fuse the features of the image and text data in the health data, and inputs the fused data into an improved Hyper-YOLO model for preliminary classification; The medical device interaction module is used to call a dedicated medical device for testing when the initial classification fails. In response to the user selecting "guided mode", the iTrans-GAHNet model is used to call the device usage tutorial in the database; in response to the user selecting "autonomous mode", the test data generated by the user is received and input into the Hyper-YOLO model for final classification. The drug recommendation module recommends suitable drugs to users based on the results of preliminary or final classification through the iTrans-GAHNet model combined with cloud-based drug data, and displays a drug intake confirmation interface; The medication guidance module responds to the user selecting "Need medication guidance" on the medication confirmation interface and provides guidance information on medication dosage and administration time through the iTrans-GAHNet model; responds to the user selecting "Known medication method" on the medication confirmation interface and ends the process.

Images