A health data multi-modal data processing method based on generative artificial intelligence

Abstract

This invention relates to the fields of digital healthcare and artificial intelligence, and particularly to a method for processing multimodal health data based on generative artificial intelligence. The method includes the following steps: acquiring a multimodal health data stream of a target object within a preset time window, wherein the multimodal health data stream includes at least a first modality data sequence and a second modality data sequence, and the data acquisition frequency of the first modality data sequence is lower than that of the second modality data sequence; inputting the multimodal health data stream into a pre-trained multimodal generative model, wherein the multimodal generative model includes at least a time-aware fusion module, a cross-granularity semantic calibration module, and an adversarial semantic filtering module; and in the time-aware fusion module, constructing an asymmetric time-aware fusion network based on the timestamp information corresponding to the first modality data sequence and the second modality data sequence.

Description

Technical Field

[0001] This invention relates to the fields of digital healthcare and artificial intelligence, and in particular to a method for processing multimodal health data based on generative artificial intelligence. Background Technology

[0002] With the rapid development of IoT, wearable devices, and digital healthcare technologies, the methods for collecting personal health data are becoming increasingly diverse, and data modalities are showing a trend towards diversification. In modern medical and health monitoring systems, multimodal health data mainly includes two categories: one is medical imaging data, such as computed tomography (CT) scans, magnetic resonance imaging (MRI) images, ultrasound images, and X-ray images. This data is characterized by high dimensionality and high information density, and can intuitively reflect the internal structure and pathological changes of the human body. The other is linguistic modal data, such as electronic medical records, doctor's diagnostic texts, patient daily health logs, and status description texts generated by wearable devices. This data is characterized by time-intensive processing and rich semantics, and can reflect the continuous changes in the patient's status. In actual long-term health monitoring scenarios, such as chronic disease management, postoperative rehabilitation monitoring, and home care for the elderly, there are significant structural differences between these two types of data. Specifically, medical imaging data is usually collected by specialized hospital equipment, which is costly and complex to operate, resulting in a low collection frequency, often at weekly, monthly, or even quarterly intervals. Language modality data, especially daily health logs generated by patients through mobile terminals or wearable devices, is low in cost and easy to collect. The collection frequency can reach once a day or even multiple times an hour, forming a high-frequency continuous data stream.

[0003] Existing health multimodal data processing technologies based on generative artificial intelligence mainly suffer from the following technical shortcomings: Existing technologies typically assume that multimodal data is temporally continuous and aligned, or handle missing data using simple interpolation methods. However, during the long intervals between two image acquisitions, patients may generate a large amount of high-frequency text logs reflecting changes in their physical condition. For example, between two CT scans, patients may record daily changes in symptoms such as "worsening cough," "difficulty breathing," and "increased sputum production." Existing technologies cannot effectively utilize this high-frequency text data to extrapolate the missing image states in accordance with pathological evolution patterns, resulting in structural discontinuities in the temporal dimension. When a health status report for this time period needs to be generated, the model either ignores this textual information or fails to cross-modally correlate the textual descriptions with image features, leading to insufficient information utilization and inconsistent generated results.

[0004] Significant semantic granularity differences exist in the descriptions of the same pathological features in health data from different sources. Medical imaging reports and physician diagnostic texts use professional terminology, such as "ground-glass nodules," "interstitial edema," and "calcifications," which have precise medical meanings. In contrast, patients' daily health logs use colloquial expressions, such as "chest tightness," "feeling short of breath," and "a slight cough." Existing multimodal fusion methods struggle to establish an accurate mapping between colloquial expressions and professional pathological features when dealing with this granularity mismatch. This leads to an "understanding gap" in the model during the fusion process, meaning it cannot correctly associate everyday symptom descriptions with corresponding pathological states, thus affecting the accuracy of health status assessments.

[0005] With the widespread application of generative artificial intelligence in the medical field, adversarial attacks targeting multimodal models have emerged as a new security threat. Attackers can exploit the "semantic gaps" during multimodal data fusion to embed invisible sub-visual cues into medical images or inject malicious commands into text logs. This modal masquerading data can bypass traditional security detection mechanisms during multimodal fusion, inducing models to generate incorrect diagnoses, leak patient privacy, or output malicious content. Current technologies lack effective identification and defense mechanisms against these novel attacks that utilize modality transformation for masquerading.

[0006] In long-term health monitoring, the reliability of different data sources changes dynamically over time. For example, while imaging data from three months ago is rich in information, its timeliness is poor and may not reflect the current true state; while yesterday's high-frequency text logs are timely, they are highly subjective and have limited accuracy. Current technologies for multimodal fusion typically use fixed weight allocations or dynamic weights based on content relevance, lacking a fusion confidence assessment mechanism that comprehensively considers data timeliness, inherent modality reliability, and content uncertainty. This results in the inability to quantify the reliability of the fusion results, affecting the reliability of clinical decisions. Summary of the Invention

[0007] To overcome at least one of the defects mentioned in the prior art, the present invention provides a multimodal data processing method for health data based on generative artificial intelligence, in order to solve the problems of modal cliffs in the time dimension, semantic gaps at the semantic level, adversarial attack vulnerabilities at the security level, and lack of credibility of fusion results in multimodal data as proposed in the background art.

[0008] The technical solution of this invention is implemented as follows: A method for processing multimodal health data based on generative artificial intelligence includes the following steps: Acquire multimodal health data stream of target object within a preset time window. The multimodal health data stream includes at least a first modal data sequence and a second modal data sequence. The data acquisition frequency of the first modal data sequence is lower than the data acquisition frequency of the second modal data sequence. The multimodal health data stream is input into a pre-trained multimodal generative model, which includes at least a time-aware fusion module, a cross-granularity semantic calibration module, and an adversarial semantic filtering module. In the time-aware fusion module, an asymmetric time-aware fusion network is constructed based on the timestamp information corresponding to the first modal data sequence and the second modal data sequence, capturing the state compensation features of high-frequency modal data for the missing period of low-frequency modal data, and calculating the confidence weight of the fusion feature according to the data timeliness and modal credibility to generate the first fusion feature; In the adversarial semantic filtering module, adversarial perturbation detection is performed on each modality of the multimodal health data stream to identify and filter out modal spoofing data containing malicious sub-visual cues or text injection attacks, thereby obtaining the purified second fusion feature. In the cross-granularity semantic calibration module, the purified second fusion feature interacts with the preset medical knowledge graph to map non-professional descriptive text to a professional pathological feature space, and aligns the semantic granularity differences between data from different sources to generate a third fusion feature. Based on the third fusion feature, a health data processing result corresponding to the target object is generated by a generative decoder. The health data processing result includes at least image completion data for low-frequency modal missing periods, health status prediction data for the next preset time point, or a multimodal health management plan.

[0009] Based on the above technical solutions, preferably, the construction of the asymmetric time-aware fusion network to capture the state compensation features of high-frequency modal data for periods when low-frequency modal data is missing specifically includes: Feature extraction is performed on each data sample in the first modality data sequence to obtain a set of first modality feature vectors; Feature extraction is performed on each data sample in the second modality data sequence to obtain a set of second modality feature vectors; For any target time point within the missing period between any two adjacent acquisition time points in the low-frequency modal data sequence, the attention weight of each feature vector in the second modal feature vector set relative to the target time point is calculated based on the time decay coefficient. The attention weight is then weighted and summed with the corresponding second modal feature vector to obtain the state compensation feature of the target time point. The first modal feature vector is fused with the state compensation feature to obtain the first fused feature.

[0010] Based on the above technical solutions, preferably, the step of calculating the confidence weight of the fusion features according to data timeliness and modal credibility specifically includes: Obtain the time interval between the target time point and the most recent first modality data acquisition; Obtain the information entropy of the second modality data sequence within the time window adjacent to the target time point; The time interval and the information entropy are input into a preset confidence evaluation network, and the confidence weight corresponding to the target time point is output. The confidence weight is weighted together with the first fusion feature to obtain a fusion feature with a confidence label.

[0011] Based on the above technical solutions, preferably, the adversarial semantic filtering module includes an adversarial discriminator, which is trained and obtained in the following way: Obtain a clean sample dataset and an adversarial sample dataset, wherein the adversarial sample dataset includes at least image samples with embedded subvisual cues and text samples with embedded text injection attacks; The samples in the clean sample dataset are labeled with the first category label, and the samples in the adversarial sample dataset are labeled with the second category label; With the goal of minimizing the classification loss function, the adversarial discriminator is iteratively trained so that it can distinguish between clean samples and adversarial samples. The process of identifying and filtering modal spoofing data containing malicious sub-visual cues or text injection attacks specifically includes: inputting each modal data into a trained adversarial discriminator; when the adversarial probability output by the discriminator exceeds a preset threshold, marking the modal data as modal spoofing data and removing it from the multimodal health data stream.

[0012] Based on the above technical solutions, preferably, the interaction between the purified second fusion feature and the preset medical knowledge graph specifically includes: Multiple pathological entities and their relationships are extracted from the medical knowledge graph. The medical knowledge graph is stored in a graph structure, where nodes represent pathological entities and edges represent semantic relationships between entities. Entity linking is performed on the non-professional description text in the purified second fusion feature, the semantic similarity between the non-professional description text and each pathological entity is calculated, and the pathological entity with the highest semantic similarity is selected as the matching entity. Using the matching entity as the central node, the feature information of the multi-order neighbor nodes of the central node is aggregated through a graph neural network to generate enhanced pathological features; The purified second fusion feature is fused with the enhanced pathological feature to generate the third fusion feature.

[0013] Based on the above technical solutions, preferably, the multimodal generative model is obtained through the following pre-training steps: Construct a multimodal health dataset, wherein each sample in the multimodal health dataset includes synchronously collected first modality data, second modality data, and corresponding annotation information, wherein the annotation information includes at least the true health status label and the true value of the missing modality; The multimodal health dataset is divided into a training set and a validation set; The time-aware fusion module is pre-trained using a self-supervised learning approach, enabling the time-aware fusion module to learn cross-modal temporal correlation representations; The cross-granularity semantic calibration module is pre-trained using supervised learning, enabling it to learn the mapping relationship between non-technical descriptions and technical terms. The adversarial semantic filtering module is pre-trained using an adversarial training method, enabling the adversarial semantic filtering module to learn and recognize various adversarial attack patterns. The pre-trained temporal awareness fusion module, cross-granularity semantic calibration module, and adversarial semantic filtering module are jointly fine-tuned to update all parameters of the multimodal generative model with the goal of minimizing the multi-task loss function.

[0014] Based on the above technical solutions, preferably, the generative decoder is an autoregressive language model based on the Transformer architecture or an image generation network based on a diffusion model. When the health data processing result is image completion data for a period of low-frequency modality missing, the generative decoder takes the third fusion feature as a conditional input and generates the missing medical image by gradually denoising through a diffusion model. When the health data processing result is the health status prediction data for the next preset time point, the generative decoder uses the third fusion feature as a conditional input and generates a structured health report word by word through an autoregressive approach. When the health data processing result is a multimodal health management plan, the generative decoder uses the third fusion feature as a conditional input and combines it with a preset medical knowledge base to generate personalized intervention suggestions.

[0015] A health multimodal data processing system based on generative artificial intelligence includes: A data acquisition unit is used to acquire a multimodal health data stream of a target object within a preset time window. The multimodal health data stream includes at least a first modal data sequence and a second modal data sequence, wherein the data acquisition frequency of the first modal data sequence is lower than the data acquisition frequency of the second modal data sequence. The time-aware fusion module is used to construct an asymmetric time-aware fusion network based on the timestamp information corresponding to the first modal data sequence and the second modal data sequence, capture the state compensation features of high-frequency modal data for the missing period of low-frequency modal data, and calculate the confidence weight of the fusion feature according to the data timeliness and modal credibility to generate the first fusion feature; The adversarial semantic filtering module is used to perform adversarial perturbation detection on each modality of the multimodal health data stream, identify and filter out modal spoofing data containing malicious sub-visual cues or text injection attacks, and obtain the purified second fusion feature. The cross-granularity semantic calibration module is used to interact the purified second fusion feature with the preset medical knowledge graph, map non-professional descriptive text to professional pathological feature space, and align the semantic granularity differences between data from different sources to generate a third fusion feature. A generative decoder is used to generate health data processing results corresponding to the target object based on the third fusion feature.

[0016] Based on the above technical solutions, preferably, the time-aware fusion module includes: The first feature extraction unit is used to extract features from each data sample in the first modality data sequence to obtain a first modality feature vector set; The second feature extraction unit is used to extract features from each data sample in the second modality data sequence to obtain a set of second modality feature vectors; The timing compensation unit is used to calculate the attention weight of each feature vector in the second modal feature vector set relative to the target time point based on the time decay coefficient for any target time point within the missing time period between any two adjacent acquisition time points in the low-frequency modal data sequence, and to perform a weighted summation of the attention weights and the corresponding second modal feature vectors to obtain the state compensation feature of the target time point. The feature fusion unit is used to fuse the first modal feature vector with the state compensation feature to obtain the first fused feature; The confidence assessment unit is used to obtain the time interval between the target time point and the most recent first modality data acquisition, obtain the information entropy of the second modality data sequence within the time window adjacent to the target time point, input the time interval and the information entropy into a preset confidence assessment network, output the confidence weight corresponding to the target time point, and weight the confidence weight with the first fusion feature.

[0017] An electronic device is characterized by comprising a processor, a communication bus, a user interface, a network interface, and a memory, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory to cause the electronic device to perform the above-described health multimodal data processing method based on generative artificial intelligence.

[0018] In summary, the multimodal health data processing method based on generative artificial intelligence provided by this invention has the following advantages over existing technologies: (1) By constructing an asymmetric time-aware fusion network, this invention realizes the dynamic extrapolation of the missing period of the low-frequency visual modality by the high-frequency text modality. It can reasonably allocate the weight of text information at different times according to the time decay coefficient, generate state compensation features that conform to the pathological evolution law, effectively fill the structural gap in the time dimension, and make the generated health report logically coherent in time sequence. (2) By introducing medical knowledge graph and graph neural network technology, this invention establishes a semantic mapping bridge between non-professional oral descriptions and professional pathological terms. The cross-granularity semantic calibration module can accurately associate daily symptom expressions with corresponding pathological entities and their contexts, and enhance the richness of semantic expressions by aggregating neighbor node information, fundamentally solving the problem of understanding gaps caused by semantic granularity mismatch. (3) The adversarial detection mechanism is embedded in the multimodal health data processing flow. Through a specially trained adversarial discriminator, it can effectively detect and eliminate sub-visual cue attacks and text injection attacks that are implanted using semantic gaps. This mechanism significantly improves the anti-attack capability and data security of generative artificial intelligence in clinical applications. (4) A confidence assessment mechanism for fusion features was introduced. By comprehensively considering the timeliness of data, the time interval between the most recent image acquisition and the text information entropy of modal content uncertainty, a quantitative confidence weight is generated for each fusion feature, enabling downstream clinical decision-making to conduct risk assessment based on confidence information, thereby enhancing the interpretability and practicality of the system. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating an embodiment of the present invention. Detailed Implementation

[0021] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0022] See Figure 1 This invention discloses a method for processing multimodal health data based on generative artificial intelligence, comprising the following steps: Acquire multimodal health data stream of target object within a preset time window. The multimodal health data stream includes at least a first modal data sequence and a second modal data sequence. The data acquisition frequency of the first modal data sequence is lower than the data acquisition frequency of the second modal data sequence.

[0023] It should be noted that multimodal health data streams refer to time-series data sets collected from multiple data sources, reflecting the health status of the target object, and possessing different modal characteristics. The first modality data sequence refers to health data sequences collected at a lower frequency, typically medical imaging data. Specifically, first modality data includes, but is not limited to, the following: computed tomography (CT) images, magnetic resonance imaging (MRI) images, positron emission tomography (PET) images, ultrasound images, X-ray images, and pathological slide images. First modality data can be represented as: in, This represents the first modal data sample collected in the k-th iteration. Represents the first modal data sample The collection timestamp, m represents the number of collections within the preset time window.

[0024] Second-modal data sequences refer to health data sequences collected frequently, typically text-based health logs. Second-modal data includes, but is not limited to, the following types: patient health logs, electronic medical record texts, wearable device descriptions, doctor-patient dialogue records, and social media health records. Second-modal data can be represented as: in, This represents the second modal data sample acquired in the j-th acquisition. Represents the second modality data sample The collection timestamp, where n represents the number of collections within the preset time window.

[0025] The data acquisition frequency of the first modality data sequence is significantly lower than that of the second modality data sequence. The data acquisition frequency is defined as: in, The length of the preset time window, This is the data acquisition frequency for the first mode. The data acquisition frequency for the second mode is [value], and it satisfies [condition]. In application, It is usually done every 2-4 weeks. Usually 1-3 times a day.

[0026] To achieve continuous monitoring of health status, a sliding time window mechanism is adopted. This involves sliding the time window forward at regular intervals of Δs, generating new window data. The sliding time window is defined as follows: in, Here, Δs is the window index, and Δs is the sliding step size. The initial window start time, This is the initial window end time.

[0027] This application also provides an adaptive adjustment mechanism for the time window, which dynamically adjusts the window length based on the rate of change in the target object's health status. When the target object's health status is stable, the window length can be appropriately increased to reduce the calculation frequency; when the target object's health status changes rapidly, the window length can be appropriately shortened to improve monitoring sensitivity; when an abnormal event is detected, the window length is automatically shortened for refined analysis. The formula for calculating the adaptive window length is: in, The length of the preset time window is γ, where γ is the adjustment coefficient. For health status Standard deviation of the change in condition This represents the maximum expected change value.

[0028] The first modality data is acquired through DICOM protocol acquisition, API interface calls and file uploads. During the acquisition process, the following information is automatically recorded: acquisition time, device information and quality assessment. The second modality data is acquired through mobile terminal APP, wearable device, voice assistant, WeChat mini program and IoT device. During the acquisition process, the following information is automatically recorded: acquisition time, acquisition method and user identifier.

[0029] In addition to the first and second modalities, this invention can also access auxiliary modal data to enrich the representation of health status. Auxiliary modal data includes, but is not limited to: time-series physiological parameter data: continuous monitoring data of heart rate, blood pressure, blood oxygen saturation, body temperature, and respiratory rate; activity data: steps, distance traveled, calorie consumption, and type of exercise; environmental data: indoor air quality, temperature and humidity, and noise levels; genetic data: gene testing reports and genetic information; and laboratory test data: complete blood count, urinalysis, and biochemical indicators.

[0030] Because the collected raw data may contain missing, abnormal, or duplicate issues, it needs to be cleaned. Missing image data is marked as missing and not imputed. Missing text data is judged based on context; if consecutive missing data exceeds a threshold, it is marked as a data gap. Structured auxiliary data is filled using forward imputation or interpolation methods. Outlier detection and handling employ statistical methods such as the 3σ principle. Data exceeding the medically normal range is marked as suspicious and recorded. Data with obvious errors, such as incorrect dates or formats, is corrected or removed. Duplicate data processing is based on timestamps and content fingerprints to identify duplicate data. The first record is retained, and subsequent duplicate records are removed.

[0031] To ensure the temporal alignment of multimodal data, the timestamps of all data are standardized. Since the first and second modal data were acquired at different frequencies and their acquisition times do not overlap, a time alignment mechanism is needed to associate data from different modalities onto a unified timeline, laying the foundation for subsequent time-series fusion. For any target time point, the nearest neighbor principle is used to obtain the corresponding modal data. Specifically, the first modal data is selected from the image data that is temporally closest to the target time point.

[0032] in, The first modal data at the target time point, For the target time point, For the k-th first modal data sample, This is the timestamp for the acquisition of the kth first modal data.

[0033] in, The second modal data is for the target time point. For the target time point, For the j-th second modality data sample, This is the timestamp for the acquisition of the j-th second modality data.

[0034] To fully utilize multimodal information, this invention employs a time window alignment method for the target time point. Define the time window W(t_target) = [ ], collect all data samples within the window.

[0035] The first modality dataset is as follows: W(t_target) The second modality data set is: W(t_target) in, W(t_target) is the first modal data set at the target time point, and W(t_target) is the time window at the target time point. The second modal data set at the target time point; δ is the window size, which can be dynamically adjusted according to the data frequency.

[0036] After preprocessing and alignment, the multimodal data is organized into a time-series format: First mode data sequence: The timestamp corresponding to the first modality data sequence is:

[0037] Second modal data sequence:

[0038] The timestamp corresponding to the second modality data sequence is:

[0039] v k Let t represent the feature vector of the k-th first modality sample. j The features representing the j-th second modality sample vector.

[0040] The multimodal health data stream is input into a pre-trained multimodal generative model, which includes at least a time-aware fusion module, a cross-granularity semantic calibration module, and an adversarial semantic filtering module. In this multimodal generative model, the input data uses a time window as the basic processing unit, and the input data packet is defined as follows:

[0041] Where X is the input data packet; V is the first mode data sequence; T is the second mode data sequence; and M is the second mode data sequence. meta This is a collection of metadata.

[0042]

[0043] The input data is organized into a sequence in chronological order and includes timestamp information: Among them, V seq For the first modality data sequence, v kFor the k-th first modal data sample, T is the timestamp of the acquisition of the k-th first modal data, where m is the total number of first modal data; seq For the second mode time series, t j For the j-th second modality data sample, Let be the timestamp of the j-th second modal data collection, and n be the total number of second modal data.

[0044] To improve processing efficiency, the model supports batch input. For a batch of input data spanning B time windows, the data is organized as follows:

[0045] Among them, each X i This corresponds to multimodal data within a time window.

[0046] The pre-training of the time-aware fusion module employs a masked time-series prediction task, randomly masking text features at certain time points. The trained model then predicts the masked features based on text and image features from other time points. The loss function is:

[0047] Among them, L mask Let M be the loss function, M be the set of masked time points, and t be the time point. j Features of real text These are the predicted text features.

[0048] Simultaneously, a temporal prediction task is employed, randomly shuffling the order of the text sequence to train the model for accurate predictions. The temporal order. Loss function:

[0049] in, Lorder Here, p(order) is the time sequence prediction loss function, where n is the length of the text sequence and p(order) is the order of the text sequence. i |T shuffled ) represents the conditional probability, T shuffled This is the shuffled text sequence.

[0050] The adversarial semantic filtering module includes an adversarial discriminator, which is trained in the following manner: Obtain a clean sample dataset and an adversarial sample dataset, wherein the adversarial sample dataset includes at least image samples with embedded subvisual cues and text samples with embedded text injection attacks; The samples in the clean sample dataset are labeled with the first category label, and the samples in the adversarial sample dataset are labeled with the second category label; With the goal of minimizing the classification loss function, the adversarial discriminator is iteratively trained so that it can distinguish between clean samples and adversarial samples. The process of identifying and filtering modal spoofing data containing malicious sub-visual cues or text injection attacks specifically includes: inputting each modal data into a trained adversarial discriminator; when the adversarial probability output by the discriminator exceeds a preset threshold, marking the modal data as modal spoofing data and removing it from the multimodal health data stream.

[0051]

[0052] The adversarial semantic filtering module employs an adversarial training strategy during pre-training, generating both adversarial and clean samples to train the discriminator to distinguish between the two classes. The loss function for generating adversarial samples is as follows:

[0053] Among them, L attack Let P be the loss function for adversarial samples, x be a clean sample, and P be the loss function for adversarial samples. clean For a clean sample distribution, For adversarial examples, For counter-perturbation; y target For target labels.

[0054] Among them, L disc Let P be the discriminator loss function, D(x) be the discriminator output, and P be the discriminator output. attack This is an adversarial sample distribution.

[0055] The pre-training of the cross-granularity semantic calibration module employs a contrastive learning strategy, bringing matching text-entity pairs closer together and pushing unmatched text-entity pairs further apart. The contrastive loss function is as follows:

[0056] Among them, L contrast To compare the learning loss function, For positive sample similarity, t s For text feature vectors, For positive sample entity features, To match entities, represents the temperature parameter, and he represents the negative sample entity feature.

[0057] The pre-training data comes from multiple sources, such as publicly available medical datasets, anonymized hospital data, and synthetic data.

[0058] The multimodal generative model is obtained through the following pre-training steps: Construct a multimodal health dataset, wherein each sample in the multimodal health dataset includes synchronously collected first modality data, second modality data, and corresponding annotation information, wherein the annotation information includes at least the true health status label and the true value of the missing modality; The multimodal health dataset is divided into a training set and a validation set; The time-aware fusion module is pre-trained using a self-supervised learning approach, enabling the time-aware fusion module to learn cross-modal temporal correlation representations; The cross-granularity semantic calibration module is pre-trained using supervised learning, enabling it to learn the mapping relationship between non-technical descriptions and technical terms. The adversarial semantic filtering module is pre-trained using an adversarial training method, enabling the adversarial semantic filtering module to learn and recognize various adversarial attack patterns. The pre-trained temporal awareness fusion module, cross-granularity semantic calibration module, and adversarial semantic filtering module are jointly fine-tuned to update all parameters of the multimodal generative model with the goal of minimizing the multi-task loss function.

[0059] In the time-aware fusion module, an asymmetric time-aware fusion network is constructed based on the timestamp information corresponding to the first modal data sequence and the second modal data sequence, capturing the state compensation features of high-frequency modal data for the missing period of low-frequency modal data, and calculating the confidence weight of the fusion feature according to the data timeliness and modal credibility to generate the first fusion feature.

[0060] It should be noted that the construction of the asymmetric time-aware fusion network to capture the state compensation features of high-frequency modal data for periods when low-frequency modal data is missing specifically includes: Feature extraction is performed on each data sample in the first modality data sequence to obtain a set of first modality feature vectors; Feature extraction is performed on each data sample in the second modality data sequence to obtain a set of second modality feature vectors; For any target time point within the missing period between any two adjacent acquisition time points in the low-frequency modal data sequence, the attention weight of each feature vector in the second modal feature vector set relative to the target time point is calculated based on the time decay coefficient. The attention weight is then weighted and summed with the corresponding second modal feature vector to obtain the state compensation feature of the target time point. The first modal feature vector is fused with the state compensation feature to obtain the first fused feature.

[0061] After feature extraction, the first modality feature vector set is obtained:

[0062] Among them, V feat v is the first mode feature vector. m Let be the feature vector of the m-th image sample.

[0063] Specifically, the calculation of the confidence weights of the fused features based on data timeliness and modal credibility includes: Obtain the time interval between the target time point and the most recent first modality data acquisition; Obtain the information entropy of the second modality data sequence within the time window adjacent to the target time point; The time interval and the information entropy are input into a preset confidence evaluation network, and the confidence weight corresponding to the target time point is output. The confidence weight is weighted together with the first fusion feature to obtain a fusion feature with a confidence label.

[0064] In the adversarial semantic filtering module, adversarial perturbation detection is performed on each modality of the multimodal health data stream to identify and filter out modal spoofing data containing malicious sub-visual cues or text injection attacks, thereby obtaining the purified second fusion feature.

[0065] In the cross-granularity semantic calibration module, the purified second fusion feature interacts with a preset medical knowledge graph to map non-professional descriptive text to a professional pathological feature space, and aligns the semantic granularity differences between data from different sources to generate a third fusion feature.

[0066] The step of interacting the purified second fusion feature with a preset medical knowledge graph specifically includes: Multiple pathological entities and their relationships are extracted from the medical knowledge graph. The medical knowledge graph is stored in a graph structure, where nodes represent pathological entities and edges represent semantic relationships between entities. Entity linking is performed on the non-professional description text in the purified second fusion feature, the semantic similarity between the non-professional description text and each pathological entity is calculated, and the pathological entity with the highest semantic similarity is selected as the matching entity. Using the matching entity as the central node, the feature information of the multi-order neighbor nodes of the central node is aggregated through a graph neural network to generate enhanced pathological features; The purified second fusion feature is fused with the enhanced pathological feature to generate the third fusion feature.

[0067] Based on the third fusion feature, a health data processing result corresponding to the target object is generated by a generative decoder. The health data processing result includes at least image completion data for low-frequency modal missing periods, health status prediction data for the next preset time point, or a multimodal health management plan.

[0068] The generative decoder is either an autoregressive language model based on the Transformer architecture or an image generation network based on a diffusion model. When the health data processing result is image completion data for a period of low-frequency modality missing, the generative decoder takes the third fusion feature as a conditional input and generates the missing medical image by gradually denoising through a diffusion model. When the health data processing result is the health status prediction data for the next preset time point, the generative decoder uses the third fusion feature as a conditional input and generates a structured health report word by word through an autoregressive approach. When the health data processing result is a multimodal health management plan, the generative decoder uses the third fusion feature as a conditional input and combines it with a preset medical knowledge base to generate personalized intervention suggestions.

[0069] In some embodiments, an electronic device provided in this application includes a processor and a memory; the memory stores a computer program, wherein the computer program, when executed by the processor, implements the above-described health multimodal data processing method based on generative artificial intelligence.

[0070] Specifically, the processor may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor may also include onboard memory for caching purposes. The processor may be a single processing unit or multiple processing units for performing different actions of the method flow according to embodiments of this application.

[0071] Memory can be any medium capable of containing, storing, transmitting, propagating, or transmitting instructions. For example, memory can include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, instruments, or propagation media. Specific examples of memory include: magnetic storage devices such as magnetic tape or hard disk drives (HDDs); optical storage devices such as optical discs (CD-ROMs); and also random access memory (RAM) or flash memory; and / or wired / wireless communication links.

[0072] This application also provides a computer-readable medium storing a computer program thereon, which, when executed by a processor, implements the aforementioned method for processing health multimodal data based on generative artificial intelligence. This computer-readable medium may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into that device / apparatus / system. The aforementioned computer-readable medium carries one or more programs, which, when executed, implement the method as described in the embodiments of this application.

[0073] According to embodiments of this application, a computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wired, optical fiber, radio frequency signals, etc., or any suitable combination thereof.

[0074] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application. Therefore, the scope of this application should not be limited to the above embodiments. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing health multimodal data based on generative artificial intelligence, characterized in that, Includes the following steps: Acquire multimodal health data stream of target object within a preset time window. The multimodal health data stream includes at least a first modal data sequence and a second modal data sequence. The data acquisition frequency of the first modal data sequence is lower than the data acquisition frequency of the second modal data sequence. The multimodal health data stream is input into a pre-trained multimodal generative model, which includes at least a time-aware fusion module, a cross-granularity semantic calibration module, and an adversarial semantic filtering module. In the time-aware fusion module, an asymmetric time-aware fusion network is constructed based on the timestamp information corresponding to the first modal data sequence and the second modal data sequence, capturing the state compensation features of high-frequency modal data for the missing period of low-frequency modal data, and calculating the confidence weight of the fusion feature according to the data timeliness and modal credibility to generate the first fusion feature; In the adversarial semantic filtering module, adversarial perturbation detection is performed on each modality of the multimodal health data stream to identify and filter out modal spoofing data containing malicious sub-visual cues or text injection attacks, thereby obtaining the purified second fusion feature. In the cross-granularity semantic calibration module, the purified second fusion feature interacts with the preset medical knowledge graph to map non-professional descriptive text to a professional pathological feature space, and aligns the semantic granularity differences between data from different sources to generate a third fusion feature. Based on the third fusion feature, a health data processing result corresponding to the target object is generated by a generative decoder. The health data processing result includes at least image completion data for low-frequency modal missing periods, health status prediction data for the next preset time point, or a multimodal health management plan.

2. The method for processing health multimodal data based on generative artificial intelligence according to claim 1, characterized in that, The construction of the asymmetric time-aware fusion network to capture the state compensation features of high-frequency modal data for periods when low-frequency modal data is missing specifically includes: Feature extraction is performed on each data sample in the first modality data sequence to obtain a set of first modality feature vectors; Feature extraction is performed on each data sample in the second modality data sequence to obtain a set of second modality feature vectors; For any target time point within the missing period between any two adjacent acquisition time points in the low-frequency modal data sequence, the attention weight of each feature vector in the second modal feature vector set relative to the target time point is calculated based on the time decay coefficient. The attention weight is then weighted and summed with the corresponding second modal feature vector to obtain the state compensation feature of the target time point. The first modal feature vector is fused with the state compensation feature to obtain the first fused feature.

3. The method for processing health multimodal data based on generative artificial intelligence according to claim 2, characterized in that, The calculation of confidence weights for fused features based on data timeliness and modal credibility specifically includes: Obtain the time interval between the target time point and the most recent first modality data acquisition; Obtain the information entropy of the second modality data sequence within the time window adjacent to the target time point; The time interval and the information entropy are input into a preset confidence evaluation network, and the confidence weight corresponding to the target time point is output. The confidence weight is weighted together with the first fusion feature to obtain a fusion feature with a confidence label.

4. The method for processing health multimodal data based on generative artificial intelligence according to claim 1, characterized in that, The adversarial semantic filtering module includes an adversarial discriminator, which is trained in the following manner: Obtain a clean sample dataset and an adversarial sample dataset, wherein the adversarial sample dataset includes at least image samples with embedded subvisual cues and text samples with embedded text injection attacks; The samples in the clean sample dataset are labeled with the first category label, and the samples in the adversarial sample dataset are labeled with the second category label; With the goal of minimizing the classification loss function, the adversarial discriminator is iteratively trained so that it can distinguish between clean samples and adversarial samples. The process of identifying and filtering modal spoofing data containing malicious sub-visual cues or text injection attacks specifically includes: inputting each modal data into a trained adversarial discriminator; when the adversarial probability output by the discriminator exceeds a preset threshold, marking the modal data as modal spoofing data and removing it from the multimodal health data stream.

5. The method for processing health multimodal data based on generative artificial intelligence according to claim 1, characterized in that, The step of interacting the purified second fusion feature with a preset medical knowledge graph specifically includes: Multiple pathological entities and their relationships are extracted from the medical knowledge graph. The medical knowledge graph is stored in a graph structure, where nodes represent pathological entities and edges represent semantic relationships between entities. Entity linking is performed on the non-professional description text in the purified second fusion feature, the semantic similarity between the non-professional description text and each pathological entity is calculated, and the pathological entity with the highest semantic similarity is selected as the matching entity. Using the matching entity as the central node, the feature information of the multi-order neighbor nodes of the central node is aggregated through a graph neural network to generate enhanced pathological features; The purified second fusion feature is fused with the enhanced pathological feature to generate the third fusion feature.

6. The method for processing health multimodal data based on generative artificial intelligence according to claim 1, characterized in that, The multimodal generative model is obtained through the following pre-training steps: Construct a multimodal health dataset, wherein each sample in the multimodal health dataset includes synchronously collected first modality data, second modality data, and corresponding annotation information, wherein the annotation information includes at least the true health status label and the true value of the missing modality; The multimodal health dataset is divided into a training set and a validation set; The time-aware fusion module is pre-trained using a self-supervised learning approach, enabling the time-aware fusion module to learn cross-modal temporal correlation representations; The cross-granularity semantic calibration module is pre-trained using supervised learning, enabling it to learn the mapping relationship between non-technical descriptions and technical terms. The adversarial semantic filtering module is pre-trained using an adversarial training method, enabling the adversarial semantic filtering module to learn and recognize various adversarial attack patterns. The pre-trained temporal awareness fusion module, cross-granularity semantic calibration module, and adversarial semantic filtering module are jointly fine-tuned to update all parameters of the multimodal generative model with the goal of minimizing the multi-task loss function.

7. The method for processing health multimodal data based on generative artificial intelligence according to claim 1, characterized in that, The generative decoder is either an autoregressive language model based on the Transformer architecture or an image generation network based on a diffusion model. When the health data processing result is image completion data for a period of low-frequency modality missing, the generative decoder takes the third fusion feature as a conditional input and generates the missing medical image by gradually denoising through a diffusion model. When the health data processing result is the health status prediction data for the next preset time point, the generative decoder uses the third fusion feature as a conditional input and generates a structured health report word by word through an autoregressive approach. When the health data processing result is a multimodal health management plan, the generative decoder uses the third fusion feature as a conditional input and combines it with a preset medical knowledge base to generate personalized intervention suggestions.

8. A health multimodal data processing system based on generative artificial intelligence, characterized in that, include: A data acquisition unit is used to acquire a multimodal health data stream of a target object within a preset time window. The multimodal health data stream includes at least a first modal data sequence and a second modal data sequence, wherein the data acquisition frequency of the first modal data sequence is lower than the data acquisition frequency of the second modal data sequence. The time-aware fusion module is used to construct an asymmetric time-aware fusion network based on the timestamp information corresponding to the first modal data sequence and the second modal data sequence, capture the state compensation features of high-frequency modal data for the missing period of low-frequency modal data, and calculate the confidence weight of the fusion feature according to the data timeliness and modal credibility to generate the first fusion feature; The adversarial semantic filtering module is used to perform adversarial perturbation detection on each modality of the multimodal health data stream, identify and filter out modal spoofing data containing malicious sub-visual cues or text injection attacks, and obtain the purified second fusion feature. The cross-granularity semantic calibration module is used to interact the purified second fusion feature with the preset medical knowledge graph, map non-professional descriptive text to professional pathological feature space, and align the semantic granularity differences between data from different sources to generate a third fusion feature. A generative decoder is used to generate health data processing results corresponding to the target object based on the third fusion feature.

9. A health multimodal data processing system based on generative artificial intelligence according to claim 8, characterized in that, The time-aware fusion module includes: The first feature extraction unit is used to extract features from each data sample in the first modality data sequence to obtain a first modality feature vector set; The second feature extraction unit is used to extract features from each data sample in the second modality data sequence to obtain a set of second modality feature vectors; The timing compensation unit is used to calculate the attention weight of each feature vector in the second modal feature vector set relative to the target time point based on the time decay coefficient for any target time point within the missing time period between any two adjacent acquisition time points in the low-frequency modal data sequence, and to perform a weighted summation of the attention weights and the corresponding second modal feature vectors to obtain the state compensation feature of the target time point. The feature fusion unit is used to fuse the first modal feature vector with the state compensation feature to obtain the first fused feature; The confidence assessment unit is used to obtain the time interval between the target time point and the most recent first modality data acquisition, obtain the information entropy of the second modality data sequence within the time window adjacent to the target time point, input the time interval and the information entropy into a preset confidence assessment network, output the confidence weight corresponding to the target time point, and weight the confidence weight with the first fusion feature.

10. An electronic device, characterized in that, The device includes a processor, a communication bus, a user interface, a network interface, and a memory. The memory is used to store instructions. The user interface and the network interface are used to communicate with other devices. The processor is used to execute the instructions stored in the memory to cause the electronic device to perform the health multimodal data processing method based on generative artificial intelligence as described in any one of claims 1 to 7.

Images