Health monitoring system and device based on multi-branch adaptive deep fusion

Abstract

The present invention discloses a health monitoring system and device based on multi-branch adaptive deep fusion, which relates to the field of health management auxiliary technology. The system includes: a data acquisition module to acquire image, text and audio data of the target population; a health monitoring model construction module to construct a multimodal health monitoring model, the model extracts features from each modal data through a feature extraction submodule, the supervised autoencoder alignment submodule obtains a low-dimensional potential representation vector through self-supervised learning and label-based supervised learning, the multi-branch adaptive fusion submodule takes one modality as the leading modality and adaptively fuses other modalities and adjusts the weights through a gating mechanism, the dynamic gated deep fusion decision submodule adopts a dual-branch architecture to generate multimodal prediction results; the health assessment module generates health status assessment results accordingly. The present invention solves the problems of modal fusion difficulty of traditional methods through a multi-branch adaptive deep fusion architecture, and improves the accuracy, robustness and intelligence level of health monitoring.

Description

Technical Field

[0001] The present invention relates to the field of health management assistance technology, and in particular to a health monitoring system and device based on multi-branch adaptive deep fusion. Background Art

[0002] Against the backdrop of an accelerating global aging population and rising prevalence of chronic non-communicable diseases, health monitoring technology is becoming increasingly important in scenarios such as smart healthcare, home-based elderly care, and personal health management. Traditional health monitoring technologies often employ a single-modal data acquisition and analysis paradigm, encompassing physiological parameter monitoring (such as electrocardiogram (ECG) signals and blood pressure waveforms), medical imaging diagnostics (CT and MRI imaging), and behavioral data collection from wearable devices (exercise steps and sleep cycles). However, human health is inherently a multidimensional dynamic system encompassing physiological indicators, psychological states, and behavioral patterns. Single-modal data, due to its single information dimension and limited representational capabilities, struggles to accurately characterize and dynamically assess health status.

[0003] With breakthroughs in artificial intelligence, microelectromechanical systems (MEMS) sensors, and big data analytics, health monitoring methods based on multimodal data fusion have become a cutting-edge research direction. By integrating heterogeneous information sources such as images, speech, and text, multimodal data fusion can capture physiological and pathological characteristics (such as facial expressions and electromyographic signals), cognitive function indicators (prosodic characteristics of speech), and subjective health perceptions (natural language symptom descriptions) from multiple dimensions, achieving greater information complementarity and robustness. This fusion mechanism can effectively overcome the limitations of single-modal data and provide more comprehensive and in-depth information support for health monitoring.

[0004] Despite the enormous potential of multimodal health monitoring, existing technologies still face numerous technical bottlenecks in practical applications. First, multimodal data such as images, speech, and text exhibit significant heterogeneity and high-dimensional feature space differences, making it difficult for traditional data fusion algorithms (such as early fusion and late fusion) to effectively mine the deep, nonlinear relationships between modalities. Second, multimodal data acquisition often suffers from temporal synchronization errors, information redundancy, and modal conflicts, leading to unstable model training and limited generalization performance. Third, existing fusion strategies often rely on manually designed feature engineering and fixed fusion rules, lacking the ability to adaptively model the dynamic interactions between modalities, making it difficult to meet the personalized monitoring needs in complex health scenarios.

[0005] In summary, the development of multimodal health monitoring methods with the ability to automatically identify the importance of modal features and dynamically adaptively fuse them has become a key technical path to break through existing technical bottlenecks and improve the accuracy and intelligence level of health status assessment. Summary of the Invention

[0006] To this end, the technical problem to be solved by the present invention is to overcome the problems of immaturity of multimodal data fusion and insufficient alignment methods in the prior art.

[0007] To solve the above technical problems, the present invention provides a health monitoring system and device based on multi-branch adaptive deep fusion, which includes: a data acquisition module, a health monitoring model construction module and a health assessment module; wherein the data acquisition module is configured to: acquire image data, text data and audio data related to the target population;

[0008] The health monitoring model construction module is configured to: construct a multimodal health monitoring model, wherein the multimodal health monitoring model includes a feature extraction submodule, a supervised autoencoder alignment submodule, a multi-branch adaptive fusion submodule, and a dynamic gated deep fusion decision submodule;

[0009] The feature extraction submodule is configured to: perform feature extraction on the image data, the text data, and the audio data respectively to obtain image modality features, text modality features, and audio modality features;

[0010] The supervised autoencoder alignment submodule is configured to: reconstruct and optimize each modality feature through self-supervised learning and label-based supervised learning to obtain a low-dimensional potential representation vector;

[0011] The multi-branch adaptive fusion submodule is configured to: based on the low-dimensional potential representation vector, take one of the modal features as the dominant modal feature, adaptively fuse the other two modal features, and dynamically adjust the fusion weight through a gating mechanism to obtain image fusion features, text fusion features, and audio fusion features;

[0012] The dynamic gated deep fusion decision submodule is configured to: adopt a dual-branch architecture to perform high-dimensional mapping, feature extraction, and gate selection on modal features such as the image fusion feature, the text fusion feature, and the audio fusion feature, and finally perform weighted fusion to generate a multimodal prediction result;

[0013] The health assessment module is configured to generate a health status assessment result based on the multimodal prediction result, wherein the health status assessment result includes a classification, score or risk prediction of the health status of the target population.

[0014] In one embodiment of the present invention, obtaining a low-dimensional potential representation vector includes:

[0015] For the unimodal features of the input , perform dimensionality reduction mapping through the encoder network to generate a low-dimensional potential representation vector ;

[0016] The low-dimensional latent representation vector Input to the decoder network, restore the original feature distribution, and generate reconstructed features ;

[0017] Based on the low-dimensional latent representation vector , introduce sample labels Perform supervised training using Perform label prediction and calculate the unimodal classification loss value based on the predicted label and its original label error;

[0018] By the reconstruction feature and the original unimodal features The error calculation reconstruction loss value is performed, and a loss function is constructed based on the unimodal classification loss value and the reconstruction loss value, with the goal of minimizing the loss function value to optimize the low-dimensional potential representation vector ;

[0019] in, , Indicates the audio mode, Indicates text mode, Indicates image modality.

[0020] In one embodiment of the present invention, the method for calculating the unimodal classification loss value is as follows:

[0021] ,

[0022] in, represents the true label of the i-th sample in single modality j, represents the predicted label of the i-th sample using the potential representation of modality j, and N is the number of samples.

[0023] In one embodiment of the present invention, the calculation method of the reconstruction loss value is as follows:

[0024] ,

[0025] Where N is the number of samples, for The i-th input feature in , For The corresponding reconstruction features.

[0026] In one embodiment of the present invention, obtaining image fusion features, text fusion features, and audio fusion features includes:

[0027] A modal feature in the low-dimensional latent representation vector As the dominant modal feature, the other two modal features serve as auxiliary modal features;

[0028] For any dominant mode , its fusion characteristics The generation process of is expressed as:

[0029] Perform feature transformation on the auxiliary modal features to obtain the auxiliary modal features after feature transformation : , ;

[0030] Calculate the gating weight of the auxiliary modality feature : , ;

[0031] Based on the gating weight , perform weighted summation on the two auxiliary modal features after feature transformation to obtain the auxiliary modal fusion feature : ;

[0032] The dominant modal features and the auxiliary modal fusion features after linear transformation are Perform residual fusion to obtain the fusion features of each modality : ,

[0033] in, represents the weight parameter of the linear transformation, represents the bias parameter of the linear transformation, Indicates the audio mode, Indicates text mode, Indicates image modality.

[0034] In one embodiment of the present invention, the dynamic gated deep fusion decision submodule is composed of an input unit, a gated feature selection unit, a convolutional feature extraction unit, and a multimodal decision output generation unit;

[0035] The input unit is configured to: fuse the image features , text fusion features and audio fusion features Splicing into a multimodal joint feature vector ;

[0036] The gated feature selection unit is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the first high-dimensional multimodal feature vector : ;

[0037] Based on the first high-dimensional multimodal feature vector , using the sigmoid function to generate dynamic gating weights : ;

[0038] The dynamic gating weight and the high-dimensional multimodal feature vector Multiply to get the adaptive screening result ;

[0039] The convolution feature extraction unit is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the second high-dimensional multimodal feature vector : ;

[0040] The second high-dimensional multimodal feature vector is transformed into Perform deep feature extraction, and then use the sigmoid function to gate the feature vector obtained by convolution calculation to obtain the adaptive screening result ;

[0041] The multimodal decision output generating unit is configured to: and the adaptive screening results Add them together and map them back to the original dimension through a linear layer to generate the final fusion weight vector ;

[0042] Based on the fusion weight vector The image fusion features , text fusion features and audio fusion features Perform weighted summation to obtain multimodal fusion features ;

[0043] The multimodal fusion features Input the multi-layer perceptron for classification, and optimize the error between the multimodal label prediction result and the true label through the constructed multimodal loss function to obtain the multimodal prediction result , is the probability value of the healthy state.

[0044] In one embodiment of the present invention, the multimodal loss function is as follows:

[0045] ,

[0046] in, is the true label of the i-th multimodal fusion sample, is the label prediction result of the i-th multimodal fusion sample, and N is the number of samples.

[0047] In one embodiment of the present invention, the system also includes a back-propagation optimization module, which is configured to: construct a total loss function, and use the total loss function to perform end-to-end optimization on the feature reconstruction and label-based supervised learning of the supervised autoencoder alignment submodule, the cross-modal feature fusion of the multi-branch adaptive fusion submodule, and the final classification decision of the dynamic gated deep fusion decision submodule to obtain the optimal training parameters of the multimodal health monitoring model.

[0048] In one embodiment of the present invention, the total loss function is as follows:

[0049] ,

[0050] in, and It is a hyperparameter used to balance the contribution of classification loss and reconstruction loss; is the multimodal classification loss term, is the classification loss term of single modality j, is the reconstruction loss term of single modality j, , Indicates the audio mode, Indicates text mode, Indicates image modality.

[0051] The present invention also provides a health monitoring device, comprising the health monitoring system.

[0052] The above technical solution of the present invention has the following advantages over the prior art:

[0053] The present invention uses a supervised autoencoder alignment module to achieve self-alignment of intra-modal features, dynamically fuses cross-modal features in a dominant modality-first manner through a multi-branch adaptive fusion module, and then generates health prediction results through a dynamic gated deep fusion decision module. Combined with end-to-end optimization of a composite loss function including single-modal reconstruction loss, classification loss, and multi-modal classification loss, it effectively solves the problems of difficult multimodal data fusion, poor alignment, and frequent manual intervention. It can comprehensively mine the physiological, behavioral, and cognitive information in multimodal data, realize feature adaptive fusion and intelligent decision-making, and significantly improve the accuracy, robustness, and personalization of health monitoring, showing excellent performance in scenarios such as Alzheimer's disease prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein:

[0055] Figure 1This is a schematic diagram of the structure of a health monitoring system based on multi-branch adaptive deep fusion provided by an embodiment of the present invention;

[0056] Figure 2 is a partial structural diagram of a multimodal health monitoring model provided by an embodiment of the present invention;

[0057] Figure 3 1 is a schematic diagram of the data processing flow of the supervised autoencoder alignment submodule provided in an embodiment of the present invention;

[0058] Figure 4 This is a flow chart of an embodiment of the present invention, which provides a method for adaptively fusing two other auxiliary modal features to obtain an audio fusion feature using an audio modal feature as a dominant modal feature.

[0059] Figure 5 Schematic diagram of the structure of the dynamic gated deep fusion decision submodule provided by an embodiment of the present invention;

[0060] Figure 6 1 is a flow chart of the dynamic gated deep fusion decision submodule provided in an embodiment of the present invention for calculating a multimodal prediction result;

[0061] Figure 7 1 is a diagram showing the working principle of the back propagation optimization module provided in an embodiment of the present invention;

[0062] Explanation of the figure marks in the specification: 10. Data acquisition module; 20. Health monitoring model construction module; 201. Feature extraction submodule; 202. Supervised autoencoder alignment submodule; 203. Multi-branch adaptive fusion submodule; 204. Dynamic gated deep fusion decision submodule; 2041. Input unit; 2042. Gated feature selection unit; 2043. Convolutional feature extraction unit; 2044. Multimodal decision output generation unit; 30. Health assessment module; 40. Back propagation optimization module. DETAILED DESCRIPTION

[0063] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0064] Example 1:

[0065] like Figure 1 and Figure 2 As shown, the present invention provides a health monitoring system based on multi-branch adaptive deep fusion, which realizes the whole process from data input to health assessment through multi-module collaboration. The system includes: a data acquisition module 10, a health monitoring model construction module 20 and a health assessment module 30;

[0066] The data acquisition module 10 is configured to: acquire image data, text data, and audio data related to the target population; the image data includes but is not limited to facial videos and daily activity images of the target population; the text data includes but is not limited to medical consultation records and electronic medical records; the audio data includes but is not limited to conversational voice and voice question and answer data;

[0067] The health monitoring model construction module 20 is configured to: construct a multimodal health monitoring model, the multimodal health monitoring model including a feature extraction submodule 201, a supervised autoencoder alignment submodule 202, a multi-branch adaptive fusion submodule 203 and a dynamic gated deep fusion decision submodule 204;

[0068] The feature extraction submodule 201 is configured to: perform feature extraction on the image data, the text data and the audio data respectively to obtain image modality features, text modality features and audio modality features;

[0069] The supervised autoencoder alignment submodule 202 is configured to: reconstruct and optimize each modality feature through self-supervised learning and label-based supervised learning to obtain a low-dimensional potential representation vector;

[0070] The multi-branch adaptive fusion submodule 203 is configured to: based on the low-dimensional latent representation vector, take one of the modal features as the dominant modal feature, adaptively fuse the other two modal features, and dynamically adjust the fusion weight through a gating mechanism to obtain image fusion features, text fusion features, and audio fusion features;

[0071] The dynamic gated deep fusion decision submodule 204 is configured to: adopt a dual-branch architecture to perform high-dimensional mapping, feature extraction, and gate selection on modal features such as the image fusion feature, the text fusion feature, and the audio fusion feature, and finally perform weighted fusion to generate a multimodal prediction result;

[0072] The health assessment module 30 is configured to generate a health status assessment result based on the multimodal prediction result, wherein the health status assessment result includes a classification, a score or a risk prediction of the health status of the target population.

[0073] Furthermore, the feature extraction submodule 201 is configured to extract semantic features from the text data using the Bert model to obtain text modal features. ; Use the Hubert model to extract acoustic features from the audio data to obtain audio modal features ; Use the CLIP model to extract visual features from the image data to obtain image modal features .

[0074] Further, if Figure 3 As shown, the supervised autoencoder alignment submodule 202 is configured to reconstruct and optimize the features of each modality through self-supervised learning and label-based supervised learning, extract valuable information representation from single-modal data, and enhance the expressiveness of features through reconstruction tasks to obtain a low-dimensional potential representation vector, including:

[0075] For the unimodal features of the input ,pass The encoder network performs dimensionality reduction mapping to generate a low-dimensional potential representation vector : ;

[0076] The low-dimensional latent representation vector Input to The decoder network performs decoding operations, restores the original feature distribution, and generates reconstructed features : ;

[0077] Based on the low-dimensional latent representation vector , introduce sample labels Conduct supervised task training to further improve the alignment of features within the same modality, using Perform label prediction and calculate the unimodal classification loss value based on the predicted label and its original label error:

[0078] ,

[0079] in, represents the true label of the i-th sample in single modality j, represents the predicted label of the i-th sample using the potential representation of modality j, and N is the number of samples;

[0080] By the reconstruction feature and the original unimodal features The error calculation reconstruction loss value is:

[0081] ,

[0082] Where N is the number of samples, for The i-th input feature in , For Corresponding reconstruction features;

[0083] In order to capture the key information within the modality and promote the self-alignment of features within the modality, a loss function is constructed based on the unimodal classification loss value and the reconstruction loss value, and the low-dimensional potential representation vector is optimized with the goal of minimizing the loss function value. ;

[0084] in, , Indicates the audio mode, Indicates text mode, Indicates image modality.

[0085] Furthermore, in this embodiment, the image fusion feature, text fusion feature, and audio fusion feature are obtained, including:

[0086] A modal feature in the low-dimensional latent representation vector As the dominant modal feature, the other two modal features serve as auxiliary modal features;

[0087] For any dominant mode , its fusion characteristics The generation process of is expressed as:

[0088] Perform feature transformation on the auxiliary modal features to obtain the auxiliary modal features after feature transformation : , ;

[0089] Calculate the gating weight of the auxiliary modality feature : , ;

[0090] Based on the gating weight , perform weighted summation on the two auxiliary modal features after feature transformation to obtain the auxiliary modal fusion feature : ;

[0091] The dominant modal features and the auxiliary modal fusion features after linear transformation are Perform residual fusion to obtain the fusion features of each modality : ,in, represents the weight parameter of the linear transformation, Represents the bias parameter of the linear transformation.

[0092] The above technical solution realizes the dominant role of each modality in the corresponding branch through the three-layer operation of "auxiliary modal feature transformation → gated dynamic fusion → dominant modal residual superposition", and adaptively fuses the complementary information of the other two modalities, avoiding manual intervention in fusion weights and improving the expressive ability of multimodal features.

[0093] Specifically, if Figure 4 As shown, based on the above technical solution, the audio fusion feature is obtained ,include:

[0094] The low-dimensional image potential representation vector and low-dimensional text latent representation vector As an auxiliary modal feature, feature transformation is performed on the auxiliary modal feature, including:

[0095] respectively and Perform linear transformation and compress it into nonlinear space to obtain the potential representation vector and : , , and is the weight parameter, and is the bias term;

[0096] The and stated After splicing, pass the weight matrix Mapping, processed by sigmoid function to generate gate weights : ;

[0097] Based on the gating weight right and Weighted summation to obtain auxiliary modality fusion features : ;

[0098] The low-dimensional audio latent representation vector As the dominant modal feature, the dominant modal feature after linear transformation is and the auxiliary modality fusion feature Perform residual fusion to obtain audio fusion features : ,in, is the weight parameter, is the bias parameter.

[0099] Specifically, fusion features with audio The acquisition method is similar to that of ,include:

[0100] The low-dimensional text latent representation vector and the low-dimensional audio latent representation vector As an auxiliary modal feature, feature transformation is performed on the auxiliary modal feature, including:

[0101] respectively and Perform linear transformation and compress it into nonlinear space to obtain the potential representation vector and : , , and is the weight parameter, and is the bias term;

[0102] The and stated After splicing, pass the weight matrix Mapping, through the sigmoid function Processing to generate gating weights : ;

[0103] Based on the gating weight right and Weighted summation to obtain auxiliary modality fusion features : ;

[0104] The low-dimensional image potential representation vector As the dominant modal feature, the dominant modal feature after linear transformation is and the auxiliary modality fusion feature Perform residual fusion to obtain image fusion features : ,in, is the weight parameter, is the bias parameter.

[0105] Specifically, fusion features with audio The acquisition method is similar to that of ,include:

[0106] The low-dimensional image potential representation vector and the low-dimensional audio latent representation vector As an auxiliary modal feature, feature transformation is performed on the auxiliary modal feature, including:

[0107] Respectively and Perform linear transformation and then compress it into nonlinear space to obtain the potential representation vector and : , , and is the weight parameter, and is the bias term;

[0108] The and stated After splicing, pass the weight matrix Mapping, processed by sigmoid function to generate gate weights : ;

[0109] Based on the gating weight right and Weighted summation to obtain auxiliary modality fusion features : ;

[0110] The low-dimensional text latent representation vector As the dominant modal feature, the dominant modal feature after linear transformation is and the auxiliary modality fusion feature Perform residual fusion to obtain image fusion features : ,in, is the weight parameter, is the bias parameter.

[0111] Furthermore, if Figure 5 and Figure 6 As shown, the dynamic gated deep fusion decision submodule 204 is composed of an input unit 2041, a gated feature selection unit 2042, a convolutional feature extraction unit 2043 and a multimodal decision output generation unit 2044;

[0112] The input unit 2041 is configured to: fuse the image features , text fusion features and audio fusion features Splicing into a multimodal joint feature vector ;

[0113] The gated feature selection unit 2042 is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the first high-dimensional multimodal feature vector : ;

[0114] Based on the first high-dimensional multimodal feature vector , using the sigmoid function Generating dynamic gating weights : ;

[0115] The dynamic gating weight and the high-dimensional multimodal feature vector Multiply to get the adaptive screening result ;

[0116] The convolution feature extraction unit 2043 is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the second high-dimensional multimodal feature vector : ;

[0117] The second high-dimensional multimodal feature vector is transformed into Perform deep feature extraction and then use the sigmoid function Perform gate activation on the feature vector obtained by convolution calculation to obtain adaptive screening results ;

[0118] The multimodal decision output generating unit 2044 is configured to: and the adaptive screening results Add them together and map them back to the original dimension through a linear layer to generate the final fusion weight vector ;

[0119] Based on the fusion weight vector The image fusion features , text fusion features and audio fusion features Perform weighted summation to obtain multimodal fusion features ;

[0120] The multimodal fusion features Input the multi-layer perceptron for classification, and optimize the multimodal label prediction results through the constructed multimodal loss function and the true label The error between them is used to obtain the multimodal prediction result. , is the probability value of the healthy state.

[0121] Furthermore, in this embodiment, the multimodal loss function is as follows:

[0122] , is the true label of the i-th multimodal fusion sample, is the label prediction result of the i-th multimodal fusion sample, and N is the number of samples.

[0123] Furthermore, if Figure 7 As shown, the system further includes a back propagation optimization module 40, and the back propagation optimization module 40 is configured to:

[0124] Construct the total loss function, which is expressed as follows:

[0125] ,

[0126] in, and It is a hyperparameter used to balance the contribution of classification loss and reconstruction loss; is the multimodal classification loss term, is the classification loss term of single modality j, is the reconstruction loss term of single modality j, , Indicates the audio mode, Indicates text mode, Indicates image modality;

[0127] The total loss function is used to perform end-to-end optimization on the feature reconstruction and label-based supervised learning of the supervised autoencoder alignment submodule 202, the cross-modal feature fusion of the multi-branch adaptive fusion submodule 203, and the final classification decision of the dynamic gated deep fusion decision submodule 204 to obtain the optimal training parameters of the multimodal health monitoring model.

[0128] In summary, the advantages of the present invention are that, by constructing a multi-branch network structure, feature extraction is performed on multimodal data such as images, text, and speech, and the physiological, behavioral, and cognitive information contained in each modality is fully explored to achieve a comprehensive assessment of human health status; with the help of the multi-branch adaptive fusion submodule 203 and the dynamic gated deep fusion submodule 204, the weights of each branch feature in the fusion process are automatically adjusted according to the importance and relevance of different modal features to achieve deep fusion of multimodal features; the supervised autoencoder alignment submodule 202 is used to perform modal self-alignment, eliminate the differences in features of different modalities, enhance feature correlation and consistency, improve the model's ability to understand and analyze multimodal data, optimize feature representation, and enhance the ability to understand each modality.

[0129] Example 2:

[0130] Based on the same inventive concept as that of the first embodiment, the present invention further provides a health monitoring device, which includes the health monitoring system described in the first embodiment.

[0131] Furthermore, the health monitoring device also includes a multimodal data acquisition terminal, an edge computing processing unit and a communication module.

[0132] Among them, the multimodal data acquisition terminal integrates a high-definition camera module (for capturing facial micro-expressions and body movements), a microphone array (to achieve high-fidelity acquisition of voice and intonation) and a text input component (including a touch interactive screen and a speech-to-text processing module), which can support real-time synchronous acquisition of three types of data: images, audio, and text, and realize the fusion acquisition of multi-source heterogeneous data.

[0133] The edge computing processing unit, equipped with high-performance GPU / TPU computing chips, supports the local deployment and operation of pre-trained models such as Bert and Hubert, effectively reducing data transmission latency to the cloud and improving privacy and security. The unit is equipped with cache and storage media to temporarily store raw data and intermediate calculation results, supporting offline data processing even when the device is disconnected from the network.

[0134] The communication module integrates 5G / WiFi / Bluetooth and other communication protocols to achieve real-time data transmission to the cloud server and support remote health monitoring; it also supports NFC function to facilitate rapid device pairing for the target population.

[0135] The device performs in-depth processing of multimodal data through the health monitoring system, and combines the composite optimization mechanism of single-modal reconstruction loss, single-modal classification loss and multimodal classification loss to generate health status assessment results, thereby realizing risk warning for the target population in dimensions such as cognitive function and emotional state, and supporting multi-form intervention reminders through APP pop-ups, SMS notifications, etc.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0137] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A health monitoring system based on multi-branch adaptive deep fusion, characterized in that: include: The data acquisition module is configured to: acquire image data, text data, and audio data related to the target population; A health monitoring model construction module is configured to: construct a multimodal health monitoring model, wherein the multimodal health monitoring model includes a feature extraction submodule, a supervised autoencoder alignment submodule, a multi-branch adaptive fusion submodule, and a dynamic gated deep fusion decision submodule; The feature extraction submodule is configured to: perform feature extraction on the image data, the text data, and the audio data respectively to obtain image modality features, text modality features, and audio modality features; The supervised autoencoder alignment submodule is configured to: reconstruct and optimize each modality feature through self-supervised learning and label-based supervised learning to obtain a low-dimensional potential representation vector; The multi-branch adaptive fusion submodule is configured to: based on the low-dimensional latent representation vector, take one of the modal features as the dominant modal feature, adaptively fuse the other two modal features, and dynamically adjust the fusion weight through a gating mechanism to obtain image fusion features, text fusion features, and audio fusion features, including: A modal feature in the low-dimensional latent representation vector As the dominant modal feature, the other two modal features serve as auxiliary modal features; For any dominant mode , its fusion characteristics The generation process of is expressed as: Perform feature transformation on the auxiliary modal features to obtain the auxiliary modal features after feature transformation : , ; Calculate the gating weight of the auxiliary modality feature : , ; Based on the gating weight , perform weighted summation on the two auxiliary modal features after feature transformation to obtain the auxiliary modal fusion feature : ; The dominant modal features and the auxiliary modal fusion features after linear transformation are Perform residual fusion to obtain the fusion features of each modality : , in, represents the weight parameter of the linear transformation, represents the bias parameter of the linear transformation, Indicates the audio mode, Indicates text mode, Indicates image modality; The dynamic gated deep fusion decision submodule is configured to: adopt a dual-branch architecture to perform high-dimensional mapping, feature extraction, and gate selection on modal features such as the image fusion feature, the text fusion feature, and the audio fusion feature, and finally perform weighted fusion to generate a multimodal prediction result; wherein the dynamic gated deep fusion decision submodule is composed of an input unit, a gated feature selection unit, a convolutional feature extraction unit, and a multimodal decision output generation unit; The input unit is configured to: fuse the image features , text fusion features and audio fusion features Splicing into a multimodal joint feature vector ; The gated feature selection unit is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the first high-dimensional multimodal feature vector : ; Based on the first high-dimensional multimodal feature vector , using the sigmoid function to generate dynamic gating weights : ; The dynamic gating weight and the high-dimensional multimodal feature vector Multiply to get the adaptive screening result ; The convolution feature extraction unit is configured to: transform the multimodal joint feature vector Mapped into high-dimensional space, we get the second high-dimensional multimodal feature vector : ; The second high-dimensional multimodal feature vector is transformed into Perform deep feature extraction, and then use the sigmoid function to gate the feature vector obtained by convolution calculation to obtain the adaptive screening result ; The multimodal decision output generating unit is configured to: and the adaptive screening results Add them together and map them back to the original dimension through a linear layer to generate the final fusion weight vector ; Based on the fusion weight vector The image fusion features , text fusion features and audio fusion features Perform weighted summation to obtain multimodal fusion features ; The multimodal fusion features Input the multi-layer perceptron for classification, and optimize the error between the multimodal label prediction result and the true label through the constructed multimodal loss function to obtain the multimodal prediction result , is the probability value of the health state; and a health assessment module configured to generate a health status assessment result based on the multimodal prediction result, wherein the health status assessment result includes a classification, a score or a risk prediction of the health status of the target population.

2. The health monitoring system based on multi-branch adaptive deep fusion according to claim 1 is characterized in that: Get a low-dimensional potential representation vector, including: For the unimodal features of the input , perform dimensionality reduction mapping through the encoder network to generate a low-dimensional potential representation vector ; The low-dimensional latent representation vector Input to the decoder network, restore the original feature distribution, and generate reconstructed features ; Based on the low-dimensional latent representation vector , introduce sample labels Perform supervised training using Perform label prediction and calculate the unimodal classification loss value based on the predicted label and its original label error; By the reconstruction feature and the original unimodal features The error calculation reconstruction loss value is performed, and a loss function is constructed based on the unimodal classification loss value and the reconstruction loss value, with the goal of minimizing the loss function value to optimize the low-dimensional potential representation vector ; in, , Indicates the audio mode, Indicates text mode, Indicates image modality.

3. The health monitoring system based on multi-branch adaptive deep fusion according to claim 2, characterized in that: The calculation method of the unimodal classification loss value is as follows: ， in, represents the true label of the i-th sample in single modality j, represents the predicted label of the i-th sample using the potential representation of modality j, and N is the number of samples.

4. The health monitoring system based on multi-branch adaptive deep fusion according to claim 2, characterized in that: The calculation method of the reconstruction loss value is as follows: ， Where N is the number of samples, for The i-th input feature in , For The corresponding reconstruction features.

5. The health monitoring system based on multi-branch adaptive deep fusion according to claim 1, characterized in that: The multimodal loss function is as follows: ， in, is the true label of the i-th multimodal fusion sample, is the label prediction result of the i-th multimodal fusion sample, and N is the number of samples.

6. The health monitoring system based on multi-branch adaptive deep fusion according to claim 1, characterized in that: The system also includes a back-propagation optimization module, which is configured to: construct a total loss function, and use the total loss function to perform end-to-end optimization on the feature reconstruction and label-based supervised learning of the supervised autoencoder alignment submodule, the cross-modal feature fusion of the multi-branch adaptive fusion submodule, and the final classification decision of the dynamic gated deep fusion decision submodule to obtain the optimal training parameters of the multimodal health monitoring model.

7. The health monitoring system based on multi-branch adaptive deep fusion according to claim 6, characterized in that: The total loss function is as follows: ， in, and It is a hyperparameter used to balance the contribution of classification loss and reconstruction loss; is the multimodal classification loss term, is the classification loss term of single modality j, is the reconstruction loss term of single modality j, , Indicates the audio mode, Indicates text mode, Indicates image modality.

8. A health monitoring device, characterized in that: Comprising the health monitoring system according to any one of claims 1 to 7.

Images