Information generation method and system based on multi-modal data, storage medium and product

Abstract

The application discloses a kind of information generation methods, systems, storage medium and product based on multi-modal data.The system is used to obtain the first modal set of target object and second modal set, first modal set includes the available modal of target object that has been acquired, and second modal set includes the missing modal of target object to be compensated;According to the available modal data of available modal, the first risk probability that target medical event occurs to target object is determined;According to available modal data and first risk probability, similar sample is determined from multi-modal sample library;According to the modal characteristics of similar sample under missing modal, modal compensation is carried out to target object;Modal compensation information, available modal data and first risk probability are decision fusion, and auxiliary reference information is generated.Therefore, the scheme provided in the application can generate accurate and reliable auxiliary reference information under the condition of modal missing, and then improve the accuracy of auxiliary management for patients.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to an information generation method, system, storage medium and product based on multimodal data. Background Technology

[0002] In the medical field, multimodal medical data is often used to predict the probability of a patient experiencing a certain medical event. This probability is then used as supplementary reference information to assist medical personnel in monitoring the patient's condition. For example, it can help medical personnel develop follow-up plans or classify and stratify multiple patients to generate risk stratification reports to support clinical research and management decisions.

[0003] However, in practical applications, multimodal medical data is often incomplete. When modalities are missing, the risk probabilities generated by directly using incomplete multimodal data usually deviate from the patient's actual risk status, reducing the accuracy of auxiliary reference information and consequently decreasing the precision of auxiliary patient management. Summary of the Invention

[0004] This application provides an information generation method, system, storage medium, and product based on multimodal data, which can generate accurate and reliable auxiliary reference information even when modalities are missing, thereby improving the accuracy of auxiliary management of patients.

[0005] In a first aspect, embodiments of this application provide an information generation system based on multimodal data. The system includes a processor and a memory storing computer program instructions. When the processor executes the computer program instructions, it performs the following steps: acquiring a first modality set and a second modality set of a target object, wherein the first modality set includes available modalities already acquired for the target object, and the second modality set includes missing modalities to be compensated for by the target object; determining a first risk probability of the target object occurring a target medical event based on available modality data; determining similar samples corresponding to the target object from a preset multimodal sample library based on the available modality data and the first risk probability; performing modal compensation on the target object based on the modal characteristics of the similar samples in the missing modalities to obtain modal compensation information, wherein the modal compensation information includes at least the compensated modal characteristics of the target object in the second modality, a reference risk probability of the target object occurring a target medical event, and a compensation confidence level for modal compensation of the target object; and performing decision fusion on the modal compensation information, available modality data, and the first risk probability to generate auxiliary reference information, wherein the auxiliary reference information is used to assist medical personnel in monitoring the state of the target object.

[0006] Secondly, embodiments of this application provide an information generation method based on multimodal data, applied in a processor of the information generation system based on multimodal data as described in the first aspect. The method includes: acquiring a first modality set and a second modality set of a target object, wherein the first modality set includes available modalities already acquired for the target object, and the second modality set includes missing modalities to be compensated for by the target object; determining a first risk probability of the target object experiencing a target medical event based on available modality data; determining similar samples corresponding to the target object from a preset multimodal sample library based on the available modality data and the first risk probability; performing modal compensation on the target object based on the modal characteristics of the similar samples under the missing modality to obtain modal compensation information, wherein the modal compensation information includes at least the compensated modal characteristics of the target object under the missing modality, a reference risk probability of the target object experiencing a target medical event, and a compensation confidence level for modal compensation of the target object; and performing decision fusion on the modal compensation information, available modality data, and the first risk probability to generate auxiliary reference information, wherein the auxiliary reference information is used to assist medical personnel in monitoring the state of the target object.

[0007] Thirdly, embodiments of this application provide an electronic device, which includes: a processor and a memory storing computer program instructions; the processor executes the computer program instructions to implement the information generation method based on multimodal data as described in the second aspect.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the information generation method based on multimodal data as described in the second aspect.

[0009] Fifthly, embodiments of this application also provide a computer program product, wherein instructions in the computer program product, when executed by a processor of an electronic device, cause the electronic device to perform the information generation method based on multimodal data as described in the second aspect.

[0010] As described above, in this embodiment, when the medical data of the target object has modal missing features, modal compensation for the missing modality is performed using modal features of similar samples. The similar samples for these modal features are not determined by simple similarity calculations, but rather by searching a multimodal sample library based on the available modal data of the target object and the corresponding risk probabilities. This method ensures that the retrieved similar samples are not only similar to the target object in modal features but also aligned with the target object at the risk semantic level, improving the accuracy of risk probability prediction and thus enhancing the accuracy of auxiliary reference information generation. Furthermore, modal compensation information, including compensating modal features, reference risk probabilities, and compensation confidence, is generated using the modal features of similar samples under missing modalities. This ensures that the modal compensation information includes not only modal feature compensation but also risk reference levels and reliability metrics. Finally, the modal compensation information and the first risk probability are fused together for decision-making. The contribution of the compensation information is constrained by the already reliable first risk probability, avoiding unreliable compensation information from misleading prediction results, thereby generating accurate and reliable auxiliary reference information under modal missing conditions.

[0011] Therefore, the solution provided in this application can generate accurate and reliable auxiliary reference information even when modalities are missing, thereby improving the accuracy of auxiliary management of patients. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a schematic diagram of the structure of an information generation system based on multimodal data provided in one embodiment of this application; Figure 2 This is an embodiment of the present application providing an architecture diagram for generating auxiliary reference information in the case of missing modalities of the target object; Figure 3 This is an embodiment of the present application providing an architecture diagram for generating auxiliary reference information when the modality of the target object is complete; Figure 4 This is a flowchart illustrating an information generation method based on multimodal data provided in another embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in another embodiment of this application. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are intended only to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] To facilitate understanding, before explaining the solution provided in this application, the background of the solution provided in this application will be explained first.

[0017] Sudden cardiac death is characterized by its sudden onset, rapid progression, and low success rate of out-of-hospital treatment, posing a significant technical challenge to cardiovascular disease prevention and stratified management. Especially in seemingly healthy individuals or those not yet diagnosed with severe cardiovascular disease, sudden cardiac death often lacks obvious warning signs. Patients may not have undergone systematic cardiac examinations before the event, making it difficult for traditional clinical pathways to identify high-risk individuals in a timely manner. Compared to patients diagnosed with heart failure, severe coronary artery disease, or cardiomyopathy, sudden cardiac death in this population is more insidious and sudden, with a shorter out-of-hospital treatment window, and existing models and tools for early warning and risk stratification are relatively insufficient.

[0018] From an etiological perspective, sudden cardiac death is usually associated with a variety of underlying cardiac abnormalities, including coronary atherosclerosis and acute myocardial ischemia, malignant ventricular arrhythmias, cardiomyopathy and cardiac structural remodeling, heart failure, valvular heart disease, pulmonary hypertension, hereditary ion channelopathies, as well as electrolyte disturbances, drug effects, and metabolic abnormalities. These factors and pathological changes are not concentrated in a single data source, but are dispersed and hidden within multimodal medical information. For example, electrocardiogram (ECG) signals may contain electrophysiological risk information such as arrhythmias, conduction abnormalities, ST-T changes, QT interval abnormalities, and decreased heart rate variability; echocardiogram images may reflect structural and functional abnormalities such as left ventricular hypertrophy, enlarged heart chambers, decreased ejection fraction, abnormal ventricular wall motion, valvular regurgitation, and elevated pulmonary artery pressure; echocardiogram reports contain both quantitative measurements such as heart chamber size, ventricular wall thickness, ejection fraction, and valvular condition, as well as the physician's written assessment of the cardiac structure, function, and overall degree of abnormality; and structured clinical data and medical records may contain systemic risk factors such as age, gender, blood pressure, blood lipids, blood glucose, renal function, inflammatory markers, past medical history, medication use, and lifestyle. Therefore, early identification of sudden cardiac death risk requires comprehensive analysis of multi-source heterogeneous information, including ECG, cardiac imaging, echocardiogram reports, structured clinical data, and unstructured text.

[0019] In existing technologies, risk assessment methods mainly fall into two categories: traditional statistical scoring models or machine learning models, and artificial intelligence models. Traditional statistical scoring models or machine learning models typically analyze structured clinical variables such as age, gender, blood pressure, blood lipids, diabetes, and smoking. These models have simple scoring methods but relatively low accuracy in predicting risk probabilities. Artificial intelligence models typically analyze data from a single examination modality; for example, identifying arrhythmia-related risks based on electrocardiogram (ECG) signals, or assessing cardiac structural and functional abnormalities based on echocardiography. While these methods can reflect cardiovascular event risk to some extent, their ability to identify occult cardiac abnormalities in seemingly healthy individuals is limited, and they struggle to fully interpret and utilize complementary information between different modalities.

[0020] In electrocardiogram (ECG) related techniques, the 12-lead ECG waveform is typically subjected to filtering, denoising, baseline drift correction, R-peak detection, heartbeat segmentation, fixed window slicing, or time-series encoding. Features are then extracted and risk probabilities are output using one-dimensional convolutional neural networks, recurrent neural networks, Transformers, or basic ECG models. These methods are suitable for capturing electrophysiological abnormalities and rhythm changes, but their ability to reflect information such as structural cardiac lesions, valvular abnormalities, and decreased cardiac function is limited. Furthermore, the stability of the model can be affected by different equipment, sampling rates, lead quality, and noise levels.

[0021] In cardiac ultrasound imaging techniques, DICOM (Digital Imaging and Communications in Medicine) images or videos typically undergo view recognition, frame sampling, image normalization, and segmentation of cardiac chambers or ventricular walls. Then, models such as convolutional neural networks, 3D convolutional networks, and visual Transformers are used to extract image features to identify left ventricular structural abnormalities, decreased ejection fraction, abnormal wall motion, valvular regurgitation, and cardiac chamber enlargement. While these methods can capture structural and functional information in images, they are susceptible to variations in cross-sectional views, image quality, frame rate, equipment source, and operator differences. Furthermore, relying solely on image models fails to fully utilize the comprehensive judgments, quantitative measurements, and clinical semantic information already summarized in physician reports.

[0022] In cardiac ultrasound reporting technologies, reports typically include quantitative or semi-quantitative measurements such as intracardiac chamber diameter, ventricular wall thickness, ejection fraction, valvular regurgitation, and pulmonary artery pressure, along with physician-generated text such as "reduced left ventricular diastolic function," "mild valvular regurgitation," and "abnormal wall motion." In current technologies, this type of report information is usually manually entered or only a few structured variables are extracted and incorporated into clinical table data, rarely being jointly modeled with corresponding ultrasound DICOM image features. Since ultrasound reports contain a high degree of physician summarization and clinical interpretation of the images, the complementary value between detailed image information and semantic information in the report cannot be fully utilized if it cannot be used in conjunction with the original image features.

[0023] In clinical structured data and unstructured medical record text correlation techniques, variables such as demographic data, past medical history, medication, blood pressure, blood glucose, blood lipids, renal function, and inflammatory markers are typically input into models such as logistic regression, random forest, gradient boosting tree, or neural networks to predict cardiovascular events, death, or other outcomes. For medical record texts, examination descriptions, and follow-up records, keyword extraction, rule mapping, natural language processing models, or large language models can also be used for information extraction. However, structured clinical data often only reflects systemic risk factors and past disease burden, making it difficult to directly capture electrocardiographic and cardiac imaging abnormalities; unstructured text suffers from inconsistent expression, numerous abbreviations, inconsistent units, missing information, and significant noise, making its direct use in model training challenging in terms of standardization and quality control.

[0024] In related technologies, multimodal techniques attempt to stitch together or jointly train ECG, imaging, and clinical tabular data. However, these techniques are mostly focused on disease classification, imaging diagnosis, or short-term hospitalization outcomes, and technical solutions for early warning of sudden cardiac death risk in large-scale real-world population cohorts remain insufficient. Particularly in terms of multimodal fusion strategies, existing methods often employ simple feature stitching, probability weighting, or end-to-end joint training, lacking organizational methods tailored to different information levels. For example, ultrasound DICOM images are highly correlated with ultrasound report sources; these should be integrated first at the cardiac structure and function level, and then further fused with electrophysiological information reflected by ECG signals and systemic risk factors reflected by structured clinical data. Existing technologies have insufficient exploration of the aforementioned hierarchical modeling strategy of "fusion of homologous modalities first, then fusion of heterologous modalities," resulting in unclear information relationships, weight allocation, and complementary mechanisms between different modalities.

[0025] Real-world cohort data also exhibits significant heterogeneity and incompleteness. For example, electrocardiogram (ECG) data may be XML (Extensible Markup Language) files exported from different devices; echocardiogram images may be DICOM videos with different frame rates, cross-sections, and resolutions; ultrasound reports may originate from Word, Excel, PDF, or electronic medical record texts; and structured variables suffer from inconsistencies in field naming, units, missing proportions, and time windows. Existing methods typically rely on idealized, modally complete data input. For objects lacking a particular modality, methods such as sample deletion, zero-filling, mean imputation, or simple mask weighting are often used, making it difficult to fully utilize the relevant information between different modalities in historical cases.

[0026] Therefore, it is evident that the existing technology has the following problems: First, existing unimodal models can only utilize one type of information from electrocardiogram, echocardiogram, or clinical tabular data, making it difficult to simultaneously reflect electrophysiological abnormalities, cardiac structural and functional abnormalities, and systemic metabolic and underlying disease risks, thus limiting their predictive ability for complex outcomes such as sudden cardiac death. Second, existing multimodal models employ simple feature splicing, early fusion, or fixed-weight fusion, assuming all modalities are fully present; when a modality is missing, of poor quality, examined at inconsistent times, or unreadable, the multimodal model struggles to output stable results, making it unsuitable for large-scale population screening and real-world applications at the grassroots level. Third, current technologies for handling missing modalities largely rely on sample deletion, mean imputation, zero vector imputation, or binary masking, lacking mechanisms for similarity retrieval using historical complete multimodal cases and for missing modality reference compensation, failing to fully utilize the implicit cross-modal associations in existing cohort data. Furthermore, existing technologies typically use only images or a small number of quantitative report values, neglecting the complementary relationship between cardiac ultrasound images and ultrasound reports. They lack a complete process for jointly processing and analyzing ultrasound DICOM images, structured report indicators, and physician impression text, and for compensating for the absence of one ultrasound submodality through similar cases. Moreover, the raw data formats in multi-center or large-scale cohorts vary significantly; for example, ECG XML formats are inconsistent, ultrasound DICOM video frame counts and views are inconsistent, and ultrasound reports come from diverse sources with inconsistent naming, units, and abbreviations. This lacks an automated cleaning, quality control, standardization, retrieval index construction, and modeling closed loop for heterogeneous real-world data. Sudden cardiac death events have a low incidence and are extremely imbalanced. Directly using existing data to train models can easily bias towards non-event populations, leading to insufficient identification of high-risk individuals. Furthermore, existing models often output single risk probabilities, lacking verifiable similar cases, modal contributions, key features, and risk level interpretations, which hinders physician review, follow-up management, and intervention in high-risk populations.

[0027] To address the problems existing in the prior art, this application provides an information generation method, system, storage medium, and product based on multimodal data. In this application, multimodal data such as electrocardiogram (ECG) signals, echocardiogram images, echocardiogram reports, and structured clinical data of a target object are used as input. Through a technical path involving multi-source data standardization, independent encoding of single modalities, single-modal risk calibration, missing modality retrieval enhancement and compensation, decision fusion, risk stratification, and interpretation output, the risk probability of the target object experiencing a target medical event, and auxiliary reference information generated based on the risk probability, are obtained. Specifically, in this application, ECG signals undergo separate feature extraction; echocardiogram images and echocardiogram reports (including report values ​​and diagnostic text) are internally fused before feature extraction; structured clinical data undergoes separate feature extraction; finally, the extracted features are fused, and the fused features are used to generate auxiliary reference information.

[0028] The solution provided in this application does not simply superimpose multiple medical data, nor does it generate missing real examination results out of thin air. Instead, when a target object has partial modality loss, it references the risk probability and compensated confidence level, and uses its existing modality features to search for similar samples (samples with similar modality features) in a multimodal sample library. Based on the complete or relatively complete modality features, single-modality risk probability, follow-up outcome, and textual summary data of the similar samples, it generates compensated modality features, reference risk probability, and compensated confidence levels for the missing modality used for risk prediction. Then, it performs decision fusion on the generated modality compensation information, the target object's available modality data, and the risk probability corresponding to the available modality data, thereby generating auxiliary reference information to assist medical personnel in monitoring the target object's condition.

[0029] In this embodiment, the information generation system based on multimodal data can serve as the execution entity of the information generation method based on multimodal data. This multimodal data information generation system can be a local server, a high-performance computing platform, a hospital information system plugin, a remote computing platform, a cloud service, an edge computing terminal, or a software-as-a-service platform. Figure 1 As shown, the system includes a processor 10 and a memory 20 storing computer program instructions. The processor 10 can be, but is not limited to, software products running on the aforementioned server or platform, such as multimodal cardiovascular risk prediction software, hospital information system plugins, image / ECG AI analysis platforms, remote computing services, risk stratification report generation tools, RAG (Retrieval-Augmented Generation) reference case database management tools, model training and validation management platforms, etc. In this application, the RAG can be used to retrieve similar historical samples based on the available modal information of the target object, and aggregate the missing modal features, risk probabilities, and quality information of similar samples to form modality compensation information.

[0030] In this embodiment of the application, the processor 10 may include a CPU, a GPU, etc. For example, the processor 10 may be a computing cluster composed of six Nvidia A100 (40G) GPUs.

[0031] For example, the processor may include a data access module, a data quality control and standardization module, an electrocardiogram analysis module, a cardiac ultrasound analysis module, a clinical structured data analysis module, a multimodal reference case library module, a RAG missing modality compensation module, a multimodal fusion module, a results interpretation and display module, and a model management module. The functions of the above modules included in the processor can be implemented by software products within the processor.

[0032] The data access module can receive or read ECG XML, echocardiography DICOM, ultrasound reports, clinical structured data, and follow-up outcome data; the data quality control and standardization module can perform format conversion, field mapping, unit unification, time window matching, outlier handling, and missing value marking on the received or read data; the ECG analysis module can preprocess ECG waveforms, extract features, and output ECG risk probabilities; the echocardiography analysis module can preprocess DICOM images, classify views, segment view regions, extract video features, and output patient-level image risk probabilities; in addition, the echocardiography analysis module can extract report text, generate structured variables, encode impression text, and output report risk probabilities; the clinical structured data analysis module can clean, impute, and standardize clinical variables, and output structured data risk probabilities; multimodal... The reference case database module can store multimodal embeddings, risk probabilities, outcome labels, key anomaly indicators, and textual summaries of historical samples, and supports vector retrieval or hybrid retrieval; the RAG missing modality compensation module can retrieve similar samples based on the available modalities of the target object, and generate compensation features, reference risk probabilities, and compensation confidence for missing modalities; the multimodal fusion module can fuse the true features of available modalities, compensation features of missing modalities, single-modal risk probabilities, reference risk probabilities, and compensation confidence to generate risk probabilities; the result interpretation and display module can generate risk levels, modal contributions, key features, summaries of similar samples, compensation confidence, or follow-up suggestions based on the risk probabilities generated by the multimodal fusion module; the model management module can realize model training, validation, reference case database updates, version management, threshold settings, calibration updates, and log recording.

[0033] In this embodiment, when the processor executes computer program instructions, it first acquires a first modality set and a second modality set of the target object. Then, it determines a first risk probability of the target object experiencing a target medical event based on the available modality data of the available modalities. Based on the available modality data and the first risk probability, it determines similar samples corresponding to the target object from a preset multimodal sample library. Next, it performs modal compensation on the target object based on the modal characteristics of the similar samples in the missing modalities to obtain modal compensation information. Finally, it performs decision fusion on the modal compensation information, available modality data, and the first risk probability to generate auxiliary reference information.

[0034] In the above embodiments, the target group can be individuals who need to predict or monitor the risk of sudden cardiac death. For example, in community population screening, the target group can be residents participating in community health checkups; in hospital outpatient clinics or health checkup centers, the target group can be patients or those undergoing health checkups; in chronic disease management or follow-up scenarios, the target group can be patients diagnosed with cardiovascular disease who need regular follow-up assessments; in wearable device outpatient monitoring scenarios, the target group can be individuals outside the hospital wearing electrocardiogram monitoring devices; and in clinical research scenarios, the target group can be subjects participating in prospective cohort studies of cardiovascular disease.

[0035] In the embodiments of this application, the medical data of the target object typically comes from various examination methods. The data types collected by different examination methods vary, constituting different data modalities, such as electrocardiogram modal, echocardiogram modal, and structured clinical data modal. The above three data modalities are described below.

[0036] For ECG modalities, the corresponding modal data includes ECG signal data generated by ECG examination. The data access and standardization module receives the raw XML ECG file and performs preprocessing operations such as lead integrity checks, sampling rate unification, and low-pass filtering to prevent aliasing and downsampling. At the same time, it standardizes and segments the signal to obtain the modal data corresponding to the ECG modality.

[0037] For the cardiac ultrasound modality, the corresponding modality data includes ultrasound image data and ultrasound report data generated from the cardiac ultrasound examination. The ultrasound report data includes report numerical variables and diagnostic text. The data access and standardization module receives heterogeneous ultrasound data such as DICOM, AVI, and JPG. It extracts the effective imaging region through irrelevant image filtering and ultrasound sector segmentation, removing invalid background and low-quality content to obtain the modality data corresponding to the image modality. After the above processing, the processor can achieve unified multimodal data format, quality control, and standardized representation, providing high-quality input for subsequent feature extraction and multimodal fusion.

[0038] For example, for cardiac ultrasound DICOM videos, the processor can also preprocess the videos. For instance, the processor extracts the effective sector region of each video, crops it, scales it uniformly to 224×224, and samples 16 frames at a preset step size; if the number of video frames is insufficient, it is repeated or padded. Then, the view classifier outputs the section category, the image features output by the image encoder are merged with the section category information, and multiple videos of the same subject are weighted and aggregated.

[0039] For example, the processor reads text from Word, Excel, and PDF files, and extracts indicators such as left atrial diameter, left ventricular end-diastolic diameter, interventricular septal thickness, left ventricular posterior wall thickness, ejection fraction, degree of valvular regurgitation, pressure gradient, and TAPSE based on keywords, regular expressions, and a terminology dictionary. It then levels descriptions such as "mild," "small amount," "+," "++," and specific ml values, and standardizes units such as mm, cm, m / s, and mmHg. Furthermore, the processor can preprocess the ultrasound impression text in the ultrasound report and translate the preprocessed ultrasound impression text into English.

[0040] For structured clinical data modalities, the corresponding modal data includes structured clinical data generated from blood biochemistry tests and medical record entry. For example, the processor can read demographic data through the Internet, public databases of medical institutions, etc., and obtain data such as the target's past medical history, medication, vital signs, and laboratory tests from authorized institutions.

[0041] In this embodiment of the application, the processor can associate the data of the above-mentioned multiple modalities, and then classify the associated multimodal data to predict the risk probability of a certain medical event (e.g., sudden cardiac death) occurring in the target object, and then generate auxiliary reference information for monitoring the status of the target object based on the risk probability.

[0042] In one example, the processor acquires the target subject's ECG XML file, echocardiogram DICOM images or videos, echocardiogram report text / tables, demographic data, past medical history, medications, vital signs, and laboratory tests. It then associates these data with unique subject identifiers, examination dates, item numbers, and follow-up time windows to form a multimodal sample based on the subject. Next, based on the information generation task, it sets an index date and prediction window, for example, using the baseline date, physical examination date, ECG date, or echocardiogram date as the index date, and filters available modal data within a preset time range before and after the index date. It determines sudden cardiac death or related cardiovascular outcomes based on death registration, medical record summary, follow-up records, or diagnostic codes, and sets a follow-up termination time for those who did not experience the event.

[0043] In this embodiment, the modality set corresponding to the target object includes a first modality set and a second modality set. The first modality set includes the available modalities already acquired by the target object, and the second modality set includes the missing modalities of the target object to be compensated. If the target object has complete modality data, the second modality set is empty, meaning the modality of the target object is complete and no modality compensation is needed; conversely, if the data of all modalities of the target object is incomplete, the first modality set is empty. The union of the first and second modality sets is a preset multimodal set, which includes, but is not limited to, electrocardiogram modalities, echocardiogram modalities, and structured clinical data modalities.

[0044] It should be noted that scenarios where the target object modality is missing include the complete absence of modal data for the entire cardiac ultrasound modality of the target object, or the absence of partial modal data for the cardiac ultrasound modality. For example, missing a portion of the cardiac ultrasound images, reported numerical variables, or diagnostic text, rather than all of them.

[0045] Furthermore, it should be noted that when the first modality set is empty, similar samples cannot be retrieved, and thus compensation for missing modalities cannot be performed. Therefore, in this embodiment, when the first modality set is empty, the system can generate a prompt message to indicate insufficient data.

[0046] For example, if a target subject has completed an electrocardiogram (ECG) and blood tests but not a cardiac ultrasound, then the first modality set includes the ECG modality and the structured data modality, and the second modality set includes the ultrasound imaging modality and the ultrasound reporting modality. It should be noted that if the examination data for a certain modality cannot be read due to equipment failure, format incompatibility, or data corruption, it is also considered as a missing modality and included in the second modality set.

[0047] In this embodiment, each available modality of the target object has a corresponding risk identification model. After the available modality data corresponding to the available modality is input into the risk identification model, the risk identification model can output the risk probability of the target object experiencing a medical event under that available modality.

[0048] In this embodiment of the application, after obtaining available modal data and a first risk probability, the processor needs to retrieve historical samples similar to the target object from a preset multimodal sample library so as to use the complete modal information of these similar samples to compensate for the missing modalities of the target object.

[0049] It should be noted that available modal data reflects the objective performance of the target object at the physiological signal level, such as the electrocardiogram waveform morphology and image texture features of the target object, while the first risk probability reflects the semantic risk level of the target object in the occurrence of the target medical event, such as the risk level of electrophysiological abnormality. In the embodiments of this application, similar samples are retrieved from the multimodal sample library based on modal data and risk probability, which can achieve dual alignment of modal feature similarity and risk semantic similarity, so that the retrieved similar samples are not only similar to the target object at the original data level, but also comparable to the target object at the risk prognosis level.

[0050] In this embodiment, after identifying similar samples, the processor uses the complete information of the retrieved similar samples in the missing modality to generate modality compensation information for the target object in that missing modality. The modality compensation information includes at least the compensation modality features of the target object in the missing modality, the reference risk probability of the target object experiencing the target medical event, and the compensation confidence level for modality compensation of the target object. Specifically, the compensation modality features are the data of the missing modality of the target object obtained by model inference on the available modalities of similar samples; the reference risk probability is the risk probability of the target object experiencing the target medical event determined through inference; and the compensation confidence level characterizes the reliability of the compensation modality features and / or the reference risk probability.

[0051] After obtaining the modal compensation information, the processor can fuse the available modal information and compensation information of the target object to generate the final auxiliary reference information.

[0052] It should be noted that since the first risk probability is a risk assessment result of the target object under available modalities, its reliability is relatively high. When performing decision fusion, using the first risk probability as a constraint and incorporating available modal data and modal compensation information into the decision-making process can improve the reliability and accuracy of the decision.

[0053] Furthermore, since there is a discrepancy between the compensated modal features and the true modal features, and the reliability of the reference risk probability depends on the compensation confidence level, considering the compensation confidence level when performing decision fusion can differentiate the use of modal compensation information and further improve the reliability and accuracy of the decision.

[0054] In this embodiment, auxiliary reference information is used to assist medical personnel in monitoring the status of target subjects. Auxiliary reference information includes, but is not limited to, the final risk probability and / or risk level of the target subject experiencing the target medical event, the risk probability under each available modality, the compensated confidence level of missing modalities, the modality and / or key abnormal features that contribute most to the final outcome, and case summaries or statistical information of similar samples. Medical personnel can use the above auxiliary reference information to develop follow-up plans, risk stratification, or conduct clinical research and management decisions.

[0055] For example, in a scenario of screening for sudden cardiac death risk in a community or large-scale population cohort, the processor can perform batch calculations on the existing electrocardiogram, echocardiogram, physical examination and follow-up data of the patients participating in the screening based on the above-mentioned auxiliary reference information, thereby screening out individuals with a high risk of sudden cardiac death and realizing risk stratification management of a large-scale population.

[0056] For example, in the scenario of auxiliary risk stratification in a hospital cardiovascular department or physical examination center, after the patient completes the electrocardiogram, echocardiogram and laboratory tests, the processor can automatically generate a risk stratification report based on the above auxiliary reference information to help doctors decide whether a re-examination, referral or enhanced follow-up is needed.

[0057] For example, wearable devices with ECG acquisition capabilities can be used in scenarios such as community screening, home monitoring, and chronic disease follow-up to achieve early out-of-hospital identification of sudden cardiac death risk. In scenarios where wearable ECG devices enable dynamic out-of-hospital early warning, the cloud platform combines the aforementioned auxiliary reference information, ECG signals, clinical data, and previous examination information to provide dynamic risk warnings and tiered management support for seemingly healthy individuals and high-risk cardiovascular individuals.

[0058] For example, in chronic disease management and follow-up scenarios, the chronic disease management platform can use the above-mentioned auxiliary reference information as one of the bases for managing high-risk groups, and formulate follow-up frequency, re-examination items and health management plans.

[0059] For example, in remote imaging / ECG computing scenarios, the processor can perform cloud-based analysis of ECG XML and echocardiogram DICOM data from primary healthcare institutions or regional healthcare platforms, and generate standardized risk reports by combining the aforementioned auxiliary reference information.

[0060] For example, in a research cohort analysis scenario, researchers can use the aforementioned auxiliary reference information to study the association between electrocardiogram, echocardiogram, clinical metabolic indicators, and text report features and sudden cardiac death, and can also achieve iterative multimodal modeling.

[0061] As described above, in this embodiment, when the medical data of the target object has modal missing features, modal compensation for the missing modality is performed using the modal features of similar samples. These similar samples are not determined by simple similarity calculations, but are searched in a multimodal sample library based on the modal data of the target object's available modalities and the corresponding risk probabilities. This approach ensures that the retrieved similar samples are not only similar to the target object in modal features but also aligned with the target object at the risk semantic level, improving the accuracy of risk probability prediction and thus enhancing the accuracy of auxiliary reference information generation. Furthermore, modal compensation information, including compensating modal features, reference risk probabilities, and compensation confidence, is generated using the modal features of similar samples under missing modalities. This ensures that the modal compensation information includes not only modal feature compensation but also risk reference levels and reliability metrics. Finally, the modal compensation information and the first risk probability are fused together for decision-making. The contribution of the compensation information is constrained by the already reliable first risk probability, avoiding unreliable compensation information from misleading prediction results, thereby generating accurate and reliable auxiliary reference information under modal missing conditions.

[0062] Therefore, the solution provided in this application can generate accurate and reliable auxiliary reference information even when modalities are missing, thereby improving the accuracy of auxiliary management of patients.

[0063] The following describes the specific process of implementing a processor-based information generation method based on multimodal data.

[0064] In some embodiments, after determining the first set of modalities, the processor can predict the probability of a target medical event occurring in the target object based on the existing modal data.

[0065] Specifically, the processor first acquires the risk identification model corresponding to the available modes, then inputs the available mode data into the risk identification model to obtain the modal characteristics of the target object under the available modes and the initial risk probability output by the risk identification model; then, the initial risk probability is calibrated to obtain the first risk probability.

[0066] In this embodiment, different data modalities have different data structures and information characteristics, requiring different types of neural network architectures or machine learning models for processing. Therefore, for each modality, the processor uses a pre-configured risk identification model to perform risk identification.

[0067] When the available modality is the electrocardiogram (ECG) modality, the risk identification model can employ a dual-path ECG-FM (ECG Foundation Model) feature extraction framework for ECG representation learning. One path is a pre-trained ECG encoder with frozen parameters, used to extract a general ECG representation; the other path is a supervised fine-tuning ECG encoder for tasks such as predicting the risk of sudden cardiac death or a pre-defined target cardiovascular outcome, used to learn task-specific features related to clinical outcomes. The two encoders output ECG feature embeddings, which are then concatenated and fused to form a joint ECG representation for subsequent risk prediction, multimodal fusion, and downstream clinical task analysis.

[0068] For example, in a scenario where the available modality is the ECG modality, the data input to the risk identification model corresponding to the ECG modality is preprocessed multi-lead ECG waveform data. This risk identification model parses the multi-lead ECG waveform data and performs sampling rate unification, baseline drift removal, noise filtering, abnormal lead identification, R-peak detection, RR interval calculation, and fixed-length segment truncation on the parsed data. The processed waveform is then input into the ECG deep learning model or the ECG basic model to obtain the ECG embedded feature F_ECG and the initial risk probability P_ECG of the ECG single modality.

[0069] In this embodiment, the risk identification model corresponding to the ECG modality, in addition to employing the aforementioned dual-channel ECG-FM feature extraction framework, may also utilize one-dimensional convolutional neural networks, residual networks, Transformers, recurrent neural networks, basic ECG models, and traditional ECG feature-based machine learning models. Furthermore, the input to the risk identification model corresponding to the ECG modality can be a complete twelve-lead waveform, a fixed-duration segment, an average heartbeat waveform, an RR interval sequence, frequency domain features, or automatically extracted ECG parameters.

[0070] When the available modality is cardiac ultrasound, the data input to the risk identification model corresponding to the cardiac ultrasound modality can be the image data of the cardiac ultrasound modality, which can be uniformly represented in a spatiotemporal data format of (N,3,16,224,224), containing 3 channels, 16 frames, and a spatial resolution of 224×224. For static images or videos with insufficient frame count, standardization is achieved through frame duplication, resampling, or padding. The processor can use a pre-trained EchoPrime coding framework for ultrasound feature extraction. The content encoder learns the spatiotemporal image features of the ultrasound video and outputs an image embedding representation. Simultaneously, the view classifier identifies ultrasound sections and outputs the corresponding section category features. Subsequently, the image features and view classification features are concatenated to form an instance-level ultrasound representation. For multiple ultrasound video instances of the same patient, the processor can use a multi-instance learning pooling mechanism to aggregate patient-level features and generate a patient-level ultrasound representation for subsequent patient risk assessment, multimodal feature fusion, and clinical decision-related tasks.

[0071] For example, in a scenario where the available modality is cardiac ultrasound, image data of the cardiac ultrasound modality is read, and an image classification model is trained based on manually labeled image category tags to filter out non-cardiac ultrasound images and low-quality images; and an image segmentation model is trained based on manually labeled effective region masks to locate the fan-shaped effective region in the cardiac ultrasound image. Then, cropping, scaling, grayscale channel expansion, frame sampling, and standardization are performed to form image data of a preset shape; the view classification model is used to identify the section type, and the video encoder is used to extract image features. Multiple videos or multiple sections of the same subject are aggregated at the patient level to obtain the ultrasound image feature F_ECHO_IMG and the initial risk probability P_ECHO_IMG of the single image modality.

[0072] In the embodiments of this application, in addition to the above-mentioned models, the risk identification model corresponding to the cardiac ultrasound modality may also employ two-dimensional CNN (Convolutional Neural Network), three-dimensional CNN, video Transformer, MViT, ViT, Swin Transformer, temporal attention network, radiomics model, or pre-trained medical image basic model; view classification can be completed before image encoding or can be jointly trained with the image encoder.

[0073] For example, when the available modality is cardiac ultrasound, the data input to the risk identification model corresponding to the cardiac ultrasound modality can be ultrasound reports from sources such as Word, Excel, PDF, and electronic medical record text. This risk identification model can perform report information extraction, text cleaning, indicator extraction, unit standardization, terminology mapping, outlier checking, and impression text encoding. Numerical variables in the report can be used as tabular feature inputs to preprocess the ultrasound impression text. The preprocessed ultrasound impression text is then input into a large language model for English translation to obtain English impression text. The English impression text is then input into a pre-trained medical language model, Clinical BERT, to extract the text features corresponding to the ultrasound impression text. The output of this risk identification model can also form a joint image-report representation with the output of the risk identification model corresponding to the cardiac ultrasound imaging modality.

[0074] For example, the risk identification model corresponding to the cardiac ultrasound modality performs batch reading and text extraction of ultrasound reports from sources such as Word, Excel, PDF, and electronic medical records, identifying examination date, measurement items, values, units, normal / abnormal descriptions, and impression conclusions. It normalizes synonyms, abbreviations, units, and writing habits to form structured variables including atrioventricular size, ventricular wall thickness, ejection fraction, valvular regurgitation, pressure gradient, TAPSE, and Simpson's EF. Simultaneously, it performs text encoding on the physician's impression text to obtain the ultrasound report feature F_ECHO_REP and the initial risk probability P_ECHO_REP for the single report modality. F_ECHO_REP can be obtained by fusing the ultrasound report structured indicator feature F_ECHO_STR and the ultrasound report text feature F_ECHO_TEXT.

[0075] In the embodiments of this application, the risk identification model corresponding to the cardiac ultrasound modality can also be implemented by rule extraction, named entity recognition, medical dictionary matching, traditional machine learning, Chinese medical language model, fine-tuning of general large language model, prompt word extraction, or structured mapping after manual correction; the level description in the report can be processed by binary classification, multi-classification, ordered grading, continuous mapping, or retaining the original text labels.

[0076] When the available modality is a structured clinical data modality, the data input to the risk identification model corresponding to the structured clinical data modality includes demographic data, comorbidities, medications, physical signs, and biochemical indicators. This risk identification model can employ logistic regression, Cox model, random forest, XGBoost, LightGBM, CatBoost, multilayer perceptron, TabNet, TabTransformer, DeepSurv, or other tabular models. For missing values, in addition to imputed values, the model can also use mean / median imputation, grouped imputation, multiple imputation, model imputation, missing value coding, or a tree model without imputation.

[0077] For example, the risk identification model corresponding to the structured clinical data modality performs cleaning, outlier handling, missing value labeling, intergroup imputation, and standardization on demographic, lifestyle, comorbidities, medication, blood pressure, heart rate, BMI, blood lipids, blood glucose, medical history, and other indicators; the processed variables are input into the tabular model to obtain the clinical structured data feature F_EHR and the initial risk probability P_EHR of the structured data unimodality.

[0078] It should be noted that in the above examples, the risk identification models corresponding to different modalities can be obtained by training neural network models, deep learning models, etc. When the target medical event is a low-incidence medical event, such as sudden cardiac death, if the model is trained directly using the entire non-event population as a control, the resulting risk identification model is easily affected by sample class imbalance, becoming overly biased towards the non-event category, leading to insufficient ability to identify individuals at high risk of sudden cardiac death. To solve the above problem, in this embodiment, a case-control sample set based on age and gender matching is constructed before model training.

[0079] Taking sudden cardiac death as the target medical event as an example, the processor first identifies sudden cardiac death cases based on follow-up outcomes, cause of death, diagnostic codes, or event determination rules. Then, in the candidate control population where no sudden cardiac death has occurred, a control subject is matched for each sudden cardiac death case according to the same or similar gender and age criteria as the case, forming a 1:1 matched sample pair. Matching criteria may include, but are not limited to: the control subject and case subject being of the same gender; the control subject and case subject being of the same age or with an age difference within a preset range; the control subject being still in follow-up status before the case event time or the corresponding index time; the control subject not experiencing a sudden cardiac death event within the corresponding follow-up window; and the control subject possessing at least one or more target modalities that can be used for modeling.

[0080] Through the matching method described above, the processor constructs a sample set during the training phase with the same number of participants in both the event group and the control group, and similar age and gender distributions. This reduces model training bias caused by the low incidence of sudden cardiac death and baseline demographic differences. Subsequently, the ECG branch, echocardiography branch, ultrasound reporting branch, structured clinical data branch, and multimodal fusion model are trained on this matched sample set. After model training is complete, probability calibration is further performed on a validation set or the target application population that maintains the distribution of real-world event incidence rates, so that the model output more closely approximates the absolute risk level in the real population.

[0081] Because the initial risk probabilities of different modalities differ significantly in their distribution characteristics—for example, the initial risk probability output by the risk identification model corresponding to ECG data is relatively high, while the initial risk probability output by the risk identification model corresponding to structured data is relatively low—if these uncalibrated probabilities are directly used to retrieve similar samples or as fusion input, the probability scale of one modality will dominate other modalities, causing retrieval or fusion bias. Therefore, a calibration function is needed to map the initial risk probabilities of each modality to the same comparable scale.

[0082] In some embodiments, the processor obtains a risk calibration function corresponding to the available modes and uses the risk calibration function to calibrate the initial risk probability to obtain a first risk probability.

[0083] For example, for the i-th target object, let the modality set M = {ECG, ECHO, EHR}, where ECG represents the electrocardiogram modality; ECHO is the cardiac ultrasound modality, obtained by fusing three ultrasound sub-branches: ultrasound image IMG, report value NUM, and diagnostic text TEXT; and EHR represents the structured clinical data modality. Further, ECHO_REP can be obtained by fusing two sub-modalities: the ultrasound report structure index ECHO_STR and the ultrasound report text ECHO_TEXT. The processor first independently encodes each modality and forms comprehensive ultrasound features within the ultrasound modality, as shown in formulas (1) to (5): (1) (2) (3) (4) (5) In the above formulas (1) to (5), This is a feature extraction function for extracting features from ECG modal data; Features extracted from cardiac ultrasound imaging modal data; Features extracted from the modal data of cardiac ultrasound reports; Ultrasound composite features generated by combining relevant features of cardiac ultrasound; Features extracted from structured clinical modal data; X_{i,ECG} represents the preprocessed ECG waveform or segment, V_{i,l} represents the l-th cardiac ultrasound video, section, or image instance, S_{i,ECHO} represents the structured measurement indicators in the ultrasound report, T_{i,ECHO} represents textual information such as ultrasound impressions and positive findings, and X_{i,EHR} represents structured clinical variables such as demographic data, medical history, medications, vital signs, and laboratory tests. Pool represents the patient-level aggregation function, and g_{ECHO} represents the ultrasound modal fusion function.

[0084] Therefore, the modal feature set corresponding to the target object can be represented by formula (6): (6) The initial risk probability corresponding to each single mode can be calculated by formula (7): (7) In formula (7), For the first The initial risk probability corresponding to each mode; h_m Let be the risk prediction head corresponding to the m-th mode, and sigma be the Sigmoid function or other probability mapping function.

[0085] For each single-modality risk identification model, the initial risk probability is output. The initial risk probability is then converted into the first risk probability through a calibration function, as shown in formula (8): (8) In formula (8), For the first The first risk probability corresponding to each modality; This can be Plattscaling, isotonic regression, temperature scaling, or a trained calibration function.

[0086] In some embodiments, after obtaining the modal characteristics of the target object in each available modality and the first risk probability after calibration, the processor needs to retrieve samples similar to the target object from a preset multimodal sample library so as to use the complete modal information of these similar samples to compensate for the missing modalities of the target object.

[0087] Specifically, the processor first acquires the physiological characteristics data of the target object and the time window information for monitoring the state of the target object; then, it extracts features from the available modal data to obtain available modal features; next, it constructs a query vector based on the available modal features, the first risk probability corresponding to the available modality, the physiological characteristics data, and the time window information; finally, it acquires similar samples from the multimodal sample library whose vector similarity with the query vector meets the preset conditions.

[0088] In the above embodiments, the physiological characteristics data of the target subject are vital sign parameters and anthropometric indicators reflecting the basic physiological state of the target subject, such as age, gender, height, weight, body mass index (BMI), systolic blood pressure, diastolic blood pressure, resting heart rate, body temperature, and respiratory rate. This physiological characteristic data can be obtained from hospital information systems, physical examination reports, or electronic health records, and is used to characterize the overall health status and baseline risk level of the target subject. For example, target subjects who are older, have abnormal BMI, or high blood pressure have a systematic difference in the risk of sudden cardiac death compared to healthy individuals of the same age; therefore, when searching for similar samples, historical cases with similar physiological characteristics should be prioritized.

[0089] In this embodiment, time window information is used to characterize the temporal relationship between the acquisition time of each modality data of the target object and the prediction window of interest to the prediction task. Different examinations may be completed at different times; for example, an electrocardiogram (ECG) was completed on January 15, 2024, and a cardiac ultrasound was completed on January 20, 2024, while the prediction task focuses on the risk of sudden cardiac death within one year. The processor acquires the aforementioned time window information, including the acquisition date, indexing date, and prediction window length of each modality data. This time window information is used to ensure that the modality data involved in the retrieval are clinically comparable in time; that is, modality data of historically similar cases should come from disease stages and follow-up time points similar to those of the target object, avoiding inappropriate comparisons of cases under different time windows.

[0090] When constructing query vectors, in addition to using available modal features, first-risk probabilities, physiological data, and time window information, the underlying disease status and data quality information of the target object can also be included. The underlying disease status reflects the existing disease burden of the target object to ensure the comparability of risk levels; data quality information characterizes the reliability of each modal data. When retrieving similar samples, modal features corresponding to high-quality data should be given higher weights, while modal features corresponding to low-quality data should be appropriately deweighted to avoid noise introduced by low-quality data interfering with the search results.

[0091] In the above embodiments, for each available modality of the target object, the processor converts the original modality data into a fixed-dimensional embedding vector through the feature extractor corresponding to that modality, thereby obtaining the available modality data.

[0092] For example, the feature extractor is the feature encoding part of the aforementioned risk identification model. As mentioned above, each modality's risk identification model first generates a fixed-dimensional embedded feature before outputting the risk probability. Therefore, in this embodiment, the processor can utilize the modal features already calculated in the risk identification model without repeated extraction.

[0093] In the above embodiments, after obtaining the modal features of each available modality, the calibrated first risk probability corresponding to each available modality, and the auxiliary information of the target object, the processor fuses this information into a unified query vector for subsequent similar sample retrieval. This query vector is a holistic encoding of the current multimodal information state of the target object. In subsequent retrieval steps, this query vector is compared with the index vectors of each case record in the multimodal sample library to find the historical sample most similar to the target object.

[0094] For example, for a target object, let the existing modality set (i.e., the first modality set) be O_i, and the missing modality set (i.e., the second modality set) be A_i=M\O_i. The processor constructs a query vector based on the available modality features of the existing modalities, the first risk probability, data quality information, time window information, and physiological sign data, as shown in formula (9): (9) In formula (9), The query vector is r_{i,m}, which represents the data quality, time window matching degree, or existence marker of the m-th modality, and C_i represents auxiliary information such as age, gender, underlying disease, and examination time window. The processor retrieves the K most similar samples to q_i in the multimodal sample library D (i.e., the preset conditions), as shown in formulas (10) and (11): (10) (11) In formulas (10) and (11), For similar samples, K is a positive integer; z_j is the index vector of the reference sample j; sim is the cosine similarity, Euclidean distance transformed similarity, Mahalanobis distance transformed similarity, or a learned metric function; tau is the temperature coefficient; w_{i,j} is the normalized retrieval weight.

[0095] In this embodiment, the multimodal sample library is a reference sample library constructed based on the training set, historical cohort data, or authorized past case samples. Each sample record in the multimodal sample library includes at least the subject's basic information, embedding features of each modality, risk probability of each single modality, final fusion risk probability, follow-up outcome, risk label, key abnormal indicators, and optional textual sample summary. For samples with complete modalities, their full-modal data features are preserved; for samples with incomplete modalities but clear outcomes, their existing modal features and labels are also preserved to expand the retrieval space.

[0096] It should be noted that the samples in the multimodal sample library are constructed only from the training set or independent historical data, and the patient data in the test set are not in the multimodal sample library.

[0097] In addition, in this application, a similar sample is a sample whose similarity to the target object is greater than a preset similarity threshold, and the available modalities of the similar sample must include the missing modalities of the target object.

[0098] For example, in constructing a multimodal sample library, a large amount of multimodal medical data of historical cases is first collected, including electrocardiogram modality, echocardiogram modality and structured clinical data modality; the data of each modality are standardized and feature extracted to obtain the embedding features and risk probabilities of each sample in each modality; the embedding features and risk probabilities are stored in the form of index vectors and associated with the sample's outcome label, key abnormal indicators and textual summary.

[0099] In this embodiment, the index vector is constructed in the same way as the query vector, that is, the modal features, risk probabilities, physiological signs data and related auxiliary information are fused into a fixed-dimensional vector according to the same rules, thereby ensuring that the query vector and the sample library index vector are located in the same semantic space and are comparable.

[0100] Through the above embodiments, the calibrated first risk probability is incorporated into the query vector, adding the constraint of risk semantic similarity to the existing modal feature similarity. This overcomes the deficiency in existing technologies that rely solely on modal features for retrieval, where similar features lead to mismatched risks, thus improving the relevance of similar samples at the risk prognosis level. Furthermore, by incorporating auxiliary information such as physiological signs, underlying disease states, and time window information into the query vector, the retrieval space is further constrained, ensuring that retrieved similar samples have higher consistency with the target object in terms of clinical features and time dimensions, thereby enhancing the reference value of similar samples. The weights of each modal feature, risk probability, and auxiliary information in the query vector can be flexibly configured according to the actual application scenario and modal quality, giving the retrieval strategy good adaptability and adjustability.

[0101] In some embodiments, after retrieving similar samples from a multimodal sample library that are similar to the target object, the processor further utilizes the complete information of the similar samples in the missing modality to generate modality compensation information for the target object in that missing modality. As mentioned above, the modality compensation information includes at least three components: compensation modality features, reference risk probability, and compensation confidence. This embodiment details the specific generation methods of these three components.

[0102] Specifically, when there are multiple similar samples, the processor acquires the initial compensation modal features of the multiple similar samples under the missing modality; determines the retrieval weights corresponding to the multiple similar samples based on the vector similarity between the multiple similar samples and the query vector; performs a weighted summation of the multiple initial compensation modal features based on the retrieval weights corresponding to the multiple similar samples to obtain the compensation modal features of the target object under the missing modality; performs a weighted summation of the risk probabilities corresponding to the multiple initial compensation modal features based on the retrieval weights to obtain the reference risk probability of the target object experiencing the target medical event; and determines the compensation confidence based on the feature similarity between the feature vectors corresponding to the similar samples and the query vector, as well as the completeness of the modal data corresponding to the missing modality among the multiple similar samples.

[0103] In the embodiments of this application, when the target object lacks one or more modalities in electrocardiogram, echocardiogram or structured clinical data, the modality compensation module in the processor generates a query vector using the existing modalities of the target object, retrieves similar samples from the multimodal sample library, and generates compensation modal features, reference risk probabilities and compensation confidence of the missing modality based on the corresponding modal features and risk probabilities of the similar samples.

[0104] In addition, the processor can also use methods such as weighted averaging of similar cases, cluster center representation, cross-modal distillation, variational autoencoder generation, generative adversarial network generation, cross-modal Transformer generation, implicit completion within the model, or cue word generation to generate summaries to generate compensated modal features, reference risk probabilities, and compensated confidence scores for missing modalities.

[0105] For each missing modality 'a' of the target object, the processor generates modality compensation information based on the features of that modality, the hazard probability of the single modality, the follow-up outcome, and the textual description in the top K similar cases retrieved. In addition to the compensated modality features F_hat_a, the reference hazard probability P_hat_a, and the compensated confidence Conf_a mentioned above, the modality compensation information may also include textual auxiliary evidence E_hat_a related to the missing modality.

[0106] For missing modes, compensated mode features can be used for characterization. For missing modes... The processor utilizes the embedding vectors of corresponding modes in similar samples. The retrieval weights w_{i,k} are combined to perform weighted aggregation, generating compensating modality features for the missing modalities. Among them, the compensation modal features Used for subsequent multimodal fusion and risk prediction to enhance the model's robustness and generalization ability in modality-deficient scenarios. These compensated modal features are only used as compensation information during the model inference stage and are not considered as real inspection data.

[0107] For the generation of the reference risk probability P_hat_a, for the missing mode a, the processor uses the first risk probability P'_a,k of the corresponding single mode in similar samples to perform weighted aggregation to generate the reference risk probability P_hat_a=Σw_kP'_a,k, which represents the reference risk level of the target object under the missing mode.

[0108] For the generation of the compensation confidence score Conf_a, the processor calculates Conf_a based on the similarity distribution of similar samples, the consistency of risk among Top-K cases, the completeness of the corresponding missing modalities, the number of samples, and retrieval stability. This compensation confidence score Conf_a is used to control the weight of the missing modality compensation information in the final fusion.

[0109] For example, for any missing mode a∈A_i, the processor generates compensating mode features and reference risk probabilities based on the features of that mode in similar samples and the first risk probability, as shown in formulas (12) and (13): (12) (13) In formulas (12) and (13), To compensate for modal characteristics; For reference risk probability.

[0110] The processor further calculates the compensation confidence of the missing modes to constrain the contribution of the compensation representation in the final fusion, as shown in Equation (14): (14) In formula (14), Comp_{i,a} is used to characterize the integrity of mode a in the retrieved similar samples, Qual_{i,a} is used to characterize the data quality or follow-up outcome reliability of the corresponding mode in the similar samples, and phi is a preset rule function or a confidence estimation function obtained through training. When the similarity is high, the risk of similar samples is consistent, and the mode integrity is high, Conf_{i,a} increases; conversely, Conf_{i,a} decreases.

[0111] In this embodiment, the compensating modality features and reference risk probabilities of the missing modality can be weighted and aggregated according to similarity weights; the compensation confidence can be determined based on the number of similar cases, similarity distribution, completeness of similar case modalities, and consistency of search results.

[0112] In this way, the modal compensation information generated by the processor not only includes the feature compensation data of the missing modal (i.e., the compensation modal features) and the risk reference value (i.e., the reference risk probability), but also includes a quantitative assessment of the reliability of the modal compensation information, so that the fusion stage can perform differentiated processing accordingly and avoid low-quality compensation information from misleading the final prediction results.

[0113] In some embodiments, after obtaining the available modal data and modal compensation information for missing modalities, the processor needs to fuse the aforementioned multi-source information to comprehensively assess the risk of the target object experiencing the target medical event. In the embodiments of this application, the processor can achieve controlled fusion of multimodal information through a bottleneck attention mechanism. This mechanism uses a preset, learnable bottleneck token as a medium for cross-modal information interaction, enabling efficient information exchange and fusion of each modality while maintaining its own characteristics.

[0114] Specifically, the processor performs bottleneck attention fusion on the available modal data, the risk tokens corresponding to the available modal data, and the modal compensation information to obtain fused features; then, based on the fused features, it determines the target risk probability of the target object experiencing the target medical event, and generates auxiliary reference information based on the target risk probability.

[0115] In the above embodiments, bottleneck attention fusion employs a bottleneck attention mechanism. This mechanism uses pre-defined, learnable bottleneck tokens as intermediaries for cross-modal information interaction, performing cross-modal attention computation on multiple modal tokens. The bottleneck attention mechanism is a core component of the Multimodal Bottleneck Transformer (MBT). It reduces the computational complexity of cross-modal information interaction by introducing a set of bottleneck tokens, far fewer in number than the input tokens. Specifically, each modal token does not directly perform cross-modal attention computation with the others; instead, each interacts with the bottleneck token, which acts as the intermediary for information exchange.

[0116] It should be noted that, in addition to using the bottleneck attention mechanism to achieve modality fusion, early feature splicing, mid-term feature interaction, late-term probability fusion, logistic regression stacking, gated weighting, attention fusion, Bayesian model averaging, weighted averaging, ensemble learning, or hybrid fusion methods can also be used for modality fusion.

[0117] For example, after receiving the modal data (or modal features) of the available modalities of the target object, the first risk probability, the compensation modal features of the missing modalities, the reference risk probability, and the compensation confidence, the multimodal fusion module in the processor can use formula (15) to generate the target risk probability: (15) In formula (15), The target risk probability.

[0118] The processor can also pre-set multiple risk thresholds theta_1, theta_2, and theta_3. These multiple risk thresholds serve as dividing points for risk levels. For example, the three risk thresholds mentioned above can map P_{i,FINAL} to low risk, medium risk, high risk, or extremely high risk levels, as shown in formula (16): (16) In formula (16), It is classified as a risk level.

[0119] For example, for a given subject, the processor acquires their 12-lead ECG data, ultrasound report, and structured clinical data, but not the original DICOM echocardiogram images. In this scenario, the processor first extracts the ECG features, report features, and structured clinical features, respectively, and obtains the corresponding single-modal risk probabilities. Subsequently, the processor constructs a query vector based on the existing modal features and retrieves K historical samples from a multimodal sample library that are similar to the subject in terms of age, gender, medical history, ECG abnormalities, ultrasound report descriptions, and clinical metabolic indicators. If most of the retrieved similar samples have complete DICOM echocardiogram images, the processor extracts the ultrasound image embedding vector and the image single-modal risk probability of the similar samples, and aggregates them according to similarity to form a compensatory representation and image risk probability for the subject's missing ultrasound image modality.

[0120] For example, for subjects with DICOM echocardiogram images but lacking ultrasound report text, the processor retrieves similar samples based on their image features, electrocardiogram features, and structured clinical features. The processor extracts structured indicator distributions and impression text features from the ultrasound reports of similar samples, generates a reference embedding and report risk probability for the report modality, and assists in the final fusion prediction. This process does not directly generate a formal clinical report, but rather generates report modality compensation features for use by the risk prediction model.

[0121] For example, for subjects lacking ECG data but possessing echocardiogram images, ultrasound reports, and structured clinical data, the processor retrieves similar samples using existing modalities and generates ECG modal compensation information based on the ECG embeddings and ECG monomodal risk probabilities of the similar samples. This modal compensation information, together with the existing modalities, is used for the final risk prediction.

[0122] Furthermore, after obtaining the target risk probability, the processor generates auxiliary reference information based on this probability to assist medical personnel in monitoring the target subject's status. For example, the processor outputs the auxiliary reference information in the form of a visual report, which may include a risk probability bar chart, risk level indicators, radar charts of contribution of each modality, a list of key abnormal features, and statistical information on similar samples. Medical personnel can understand the target subject's risk status and sources of risk by viewing the visual report. As another example, the processor can generate follow-up recommendations based on the target risk probability and risk level. For high-risk or very high-risk target subjects, it is recommended to shorten the follow-up interval, increase cardiac examinations, or refer them to a cardiovascular specialist; for low-risk or medium-risk target subjects, it is recommended to maintain the routine follow-up plan.

[0123] In this way, the auxiliary reference information generated by the processor includes not only the final risk assessment results, but also interpretable risk source analysis and verifiable similar case references, providing medical personnel with comprehensive and interpretable decision support information, which helps to develop individualized follow-up plans and management programs.

[0124] In some embodiments, after obtaining the available modal data and modal compensation information for missing modalities, the processor needs to unify the aforementioned multi-source heterogeneous information into a format that the model can process, and generate comprehensive fusion features through cross-modal fusion. In this embodiment, each modal information can be converted into a unified token format through tokenization encoding, and then a preset bottleneck token can be used as a medium for cross-modal information interaction to achieve controlled fusion of multimodal information.

[0125] Specifically, the processor encodes the modal data, the first risk probability, and the modal compensation information in the first modal set to obtain the first modal token, the risk token, and the compensation modal token. Then, the processor performs bottleneck attention fusion on the first modal token, the risk token, and the compensation modal token to obtain fusion features. The bottleneck attention fusion is used to perform cross-modal attention calculation on multiple modal tokens by using a preset number of learnable bottleneck tokens as mediators for cross-modal information interaction.

[0126] Because different modalities have different data sources, data structures, and information levels—for example, electrocardiogram data is a time-series signal, ultrasound images are image or video data, structured data is tabular data, and risk probabilities are scalar values—this information cannot be directly input into the same model for processing in its original form. Therefore, the processor encodes all input information into a unified token format, representing all information as a fixed-dimensional vector, thus enabling subsequent attention calculations and fusion within the same feature space.

[0127] In this embodiment, the processor can encode the available modal data, modal compensation information, and first risk probability to obtain the corresponding modal token.

[0128] For example, for available modality data, for each available modality possessed by the target object, the processor encodes the embedding features of that modality into a corresponding modality token. It should be noted that the embedding features here are those already obtained in the single-modality feature extraction step described above. Before outputting the risk probability, the risk identification model for each modality first generates a fixed-dimensional embedding vector as an intermediate representation of that modality. In this embodiment, the processor maps the obtained embedding features to the token space through a linear projection layer, rather than re-extracting the features.

[0129] Specifically, the processor encodes the embedding features of the ECG modality into ECG tokens. These tokens carry electrophysiological information from the ECG waveform related to the target medical event. When the ECG modality data contains multiple leads or time segments, the embedding features of each lead are encoded into multiple independent tokens, or the embedding features of multiple leads are aggregated into a single token using attention pooling or average pooling, and then mapped to the token space through a linear projection layer.

[0130] The processor encodes the embedding features of cardiac ultrasound images into image tokens. These image tokens carry information about the cardiac structure and function within the ultrasound images. When multiple ultrasound videos or standard sections exist for the same target object, the embedding features of each video or section are encoded into multiple independent tokens, and their section type or temporal order is marked by positional encoding; alternatively, multiple embedding features are weighted and pooled using view classification weights before being encoded into a single token.

[0131] The processor encodes the embedded features of the cardiac ultrasound report into a report token. The report token carries semantic information of the quantitative measurements and physician impressions from the ultrasound report.

[0132] The processor encodes the embedding features of structured clinical data modalities into structured data tokens. These structured data tokens carry information such as demographic data, past medical history, medication information, vital signs, and laboratory test results.

[0133] For example, for modality compensation information, the processor encodes each component of the modality compensation information into a corresponding token. For instance, the processor encodes the compensated modality features into a first modality token, which carries the predicted feature information of the missing modality generated based on similar sample aggregation. This token plays the same role as the real modality token in subsequent fusion, but its participation in the fusion is controlled by the compensated confidence level. As another example, the processor encodes the first risk probability into a risk token, which maps the potential risk level of the target object under the missing modality into a fixed-dimensional vector representation. Yet another example is that the processor encodes the compensated confidence level in the modality compensation information into a compensated modality token, which maps this scalar value into a fixed-dimensional vector representation.

[0134] For example, for the first risk probability, the processor encodes the first risk probability corresponding to each available modality into a corresponding risk token. The first risk probability is a calibrated risk probability value, representing the model-calibrated risk assessment result of the target object in the corresponding modality.

[0135] During tokenization encoding, each scalar value, such as risk probability and compensated confidence, is mapped to a vector of the same dimension as the modal token through an embedding layer or a linear projection layer. Specifically, the processor pre-sets a learnable embedding matrix and projects the scalar values ​​onto the target dimension space through this matrix. After repeatedly expanding the scalar values ​​to the target dimension, mapping is performed through a linear layer; or a variant of sine / cosine positional encoding is used to encode the scalar values. This projection method allows scalar information to participate in multi-head attention computation in the form of tokens.

[0136] Through the tokenization encoding described above, the processor transforms all input information into a unified token sequence, in which each token has the same feature dimension, enabling computation and interaction within the same attention mechanism.

[0137] After obtaining the tokens from each modality, the processor performs cross-modal fusion of tokens from different modalities that carry different information to generate a comprehensive fused feature. In this embodiment, the processor achieves the above fusion using a preset bottleneck token.

[0138] A bottleneck token is a special type of token that is a learnable embedding vector whose parameters are optimized via gradient descent during model training. In one example, the bottleneck token is initially sampled from a standard normal or uniform distribution during model initialization and updated as the model parameters are optimized during training. After training, the bottleneck token learns how to extract key information from various modal tokens and integrate it into a fused representation.

[0139] Bottleneck tokens act as intermediaries for information interaction in cross-modal fusion. In the bottleneck attention mechanism, all modal tokens do not directly perform cross-modal attention computation with each other, but instead interact with the bottleneck token separately. The bottleneck token acts as the query, and the modal tokens act as the key and value. The bottleneck token queries and extracts key information from each modal token, converging the cross-modal information into its representation.

[0140] By using bottleneck tokens as an intermediary, the modal tokens do not undergo direct pairwise attention calculations, thus reducing computational complexity and overhead. Bottleneck tokens, as learnable parameters specifically designed for cross-modal fusion, are trained to optimize the quality of cross-modal information integration. Compared to directly inducing mutual attention among modal tokens, bottleneck tokens can more accurately extract and aggregate key cross-modal information, improving the precision of information interaction.

[0141] In this embodiment, when the compensation confidence corresponding to the compensation modal feature is less than the first confidence threshold, in the bottleneck attention mechanism, the processor adjusts the attention weight corresponding to the compensation modal feature from the first weight to the second weight, and / or adjusts the gating coefficient from the first coefficient to the second coefficient, and / or performs masking processing on the token corresponding to the compensation modal feature. The gating coefficient is used to characterize the information retention ratio of the token corresponding to the compensation modal feature before it is input into the bottleneck attention mechanism. The first weight is greater than the second weight, and the first coefficient is greater than the second coefficient.

[0142] After enhancing the generated modality compensation information for the missing modality using RAG retrieval, the processor calculates the compensation confidence score to quantify the reliability of this information. Ideally, when the compensation confidence score is high, the compensation information can effectively compensate for the information loss of the missing modality and improve prediction accuracy. However, when the compensation confidence score is low, such as when the number of similar samples retrieved is insufficient, the risk probabilities among similar samples are inconsistent, or the data quality of similar samples in the missing modality is poor, if the modality compensation information is still fused with the same weight as the true information, noise may be introduced and mislead the final prediction result, leading to a decrease in the accuracy of the auxiliary reference information.

[0143] In this embodiment, the processor employs an attention control mechanism based on compensation confidence to dynamically adjust the participation level of modality compensation information in the fusion process. Specifically, when the compensation confidence corresponding to the compensated modal feature is less than a first confidence threshold, the impact of the compensation information on the fusion result is weakened by reducing attention weights, decreasing gating coefficients, and / or applying mask constraints. These three control methods can be used individually or in combination according to actual needs.

[0144] In this embodiment, the processor further obtains multiple similarities between the feature vectors corresponding to multiple similar samples and the query vector; when the variance between the multiple similarities is less than the variance threshold and the compensation confidence is greater than or equal to the second confidence threshold, the bottleneck interaction weight corresponding to the compensation modality feature is adjusted from the third weight to the fourth weight, wherein the third weight is less than the fourth weight, and the bottleneck interaction weight is used to characterize the attention weight between the token corresponding to the compensation modality feature and the bottleneck token.

[0145] In the above embodiments, the processor dynamically adjusts the participation level of compensation information based on the compensation confidence level. When the compensation confidence level is lower than the first confidence level threshold, the contribution of the compensation information is weakened by reducing the attention weight, decreasing the gating coefficient, or applying mask constraints. Conversely, when the compensation confidence level is high and the consistency of the retrieval results is good, it indicates that the compensation information is not only reliable, but also that the multiple similar samples retrieved have a high degree of consistency in terms of risk. In this case, the participation level of the compensation information in cross-modal fusion should be enhanced so that high-quality compensation information can more fully play its role in compensating for missing modal information.

[0146] Therefore, in this embodiment of the application, the processor adopts a bottleneck interaction weight enhancement mechanism driven by retrieval consistency. By evaluating the similarity consistency among multiple similar samples retrieved and the level of compensation confidence, the processor dynamically adjusts the attention weight (i.e., bottleneck interaction weight) between the token corresponding to the compensation modality feature and the bottleneck token.

[0147] For example, Figure 2 This diagram illustrates the architecture for generating auxiliary reference information when the target object lacks a modality. For example... Figure 2 As shown, the architecture includes a modal compensation section, a single-modal risk prediction section, a fusion decision section, and an information output section.

[0148] Regarding the modality compensation component, when a target object lacks a certain modality, similar samples are retrieved from a multimodal sample library based on the target object's existing modality data. The modality data of these similar samples is then used to compensate for the missing modality of the target object, resulting in modality compensation information. The lack of a certain modality can include the absence of all data for that modality, or the absence of only a portion of the data. For example, in the cardiac ultrasound modality, this could include missing ultrasound image data, reported numerical variables, or a portion of the diagnostic text.

[0149] For the single-modal risk prediction part, for data of different modalities, corresponding risk identification models are used to identify the first risk probability corresponding to the modality and the modal characteristics corresponding to the modality.

[0150] For the fusion decision-making part, a bottleneck attention mechanism is used to fuse the modal compensation information output by the modal compensation part and the risk probability and modal characteristics corresponding to each existing modality output by the single risk prediction part to make a decision, and output the target risk probability of the target object experiencing the target medical event.

[0151] Regarding the information output section, the target risk probability is processed according to task requirements or application scenarios to output auxiliary reference information to assist medical personnel in monitoring the status of the target object.

[0152] In some embodiments, when the modal data of the target object is complete, the processor may use a risk identification model corresponding to the first modality to identify the risk of the first modality data and obtain a first risk probability; then, bottleneck attention fusion is performed on the first modality data and the first risk probability to obtain fusion features; then, the target risk probability of the target object occurring in the target medical event is determined based on the fusion features, and auxiliary reference information is generated based on the target risk probability.

[0153] For example, the processor checks the existence and quality of each modality data of the target object one by one according to the preset multimodal set. If the data of all preset modalities of the target object exists and passes the quality check, the modal data is determined to be complete; if the data of any modality is missing or of poor quality, such as the file being unreadable, the data being corrupted, or the signal-to-noise ratio being too low, the modal data is determined to be incomplete and needs to be processed using the missing modality compensation method in the aforementioned embodiment.

[0154] It should be noted that, in this embodiment, modal data integrity is used to characterize that the target object has completed all preset modal checks, and each modal data has been determined to be usable after quality checks. After determining that the data is complete, the processor can use methods such as... Figure 3 The architecture shown performs multimodal fusion prediction on the target object. Among them, Figure 3 and Figure 2 The difference is that, Figure 3 The modal compensation component is missing, that is, in Figure 3 Modal compensation for the target object is not required. Furthermore, in... Figure 3 In the architecture shown, the modal data in the single-modal risk prediction section is the complete modal data of the target object.

[0155] Through the above embodiments, for target objects that have completed all examinations, a comprehensive evaluation is performed based on real data from all modalities. This approach fully utilizes the complementary information between modalities, improving the accuracy of risk probability prediction. By sharing the same fusion framework with the aforementioned implementation methods that have missing modalities—that is, using the bottleneck attention mechanism—unified processing and flexible switching under different data integrity conditions are achieved, eliminating the need to design different fusion architectures for different scenarios. Moreover, the auxiliary reference information includes multi-dimensional information such as the risk probability of each modality, modality contribution, and key abnormal features, providing medical personnel with comprehensive and interpretable decision support.

[0156] This concludes the introduction to the steps performed by the processor.

[0157] The following describes the information generation method based on multimodal data provided in the embodiments of this application.

[0158] This application also provides an information generation method based on multimodal data, such as... Figure 4 The flowchart shown is a method for generating information based on multimodal data. The method includes the following steps S401 to S405: Step S401: Obtain a first modality set and a second modality set of the target object, wherein the first modality set includes the available modalities of the target object that have been obtained, and the second modality set includes the missing modalities of the target object that need to be compensated. Step S402: Determine the first risk probability of the target object experiencing the target medical event based on the available modal data of available modalities; Step S403: Based on the available modal data and the first risk probability, determine similar samples corresponding to the target object from the preset multimodal sample library; Step S404: Modal compensation is performed on the target object based on the modal characteristics of similar samples under the missing modality to obtain modal compensation information. The modal compensation information includes at least the compensation modal characteristics of the target object under the missing modality, the reference risk probability of the target object occurring the target medical event, and the compensation confidence of the modal compensation of the target object. Step S405: Decision fusion is performed on modal compensation information, available modal data, and the first risk probability to generate auxiliary reference information, which is used to assist medical personnel in monitoring the status of the target object.

[0159] It should be noted that steps S401 to S405 described above can be executed by the processor in the above embodiments. The execution process of the processor has been described above and will not be repeated here.

[0160] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0161] Figure 5 A schematic diagram of the hardware structure of the electronic device provided in an embodiment of this application is shown.

[0162] The electronic device may include a processor 10 and a memory 20 storing computer program instructions. This electronic device may refer to the multimodal data-based information generation system described in the above embodiments.

[0163] Specifically, the processor 10 may include a central processing unit (CPU), a GPU (Graphics Processing Unit), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0164] Memory 20 may include mass storage for data or instructions. For example, and not limitingly, memory 20 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 20 may include removable or non-removable (or fixed) media. Where appropriate, memory 20 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 20 is non-volatile solid-state memory.

[0165] Memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Therefore, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the methods according to one aspect of this disclosure.

[0166] The processor 10 reads and executes computer program instructions stored in the memory 20 to implement any of the information generation methods based on multimodal data in the above embodiments.

[0167] In one example, the electronic device may also include a communication interface 501 and a bus 502. Wherein, as... Figure 5 As shown, the processor 10, memory 20, and communication interface 501 are connected via bus 502 and communicate with each other.

[0168] The communication interface 501 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0169] Bus 502 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCIe) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these buses. Where appropriate, bus 502 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, this application contemplates any suitable bus or interconnect.

[0170] Furthermore, in conjunction with the information generation method based on multimodal data in the above embodiments, this application embodiment can provide a computer-readable storage medium for implementation. This computer-readable storage medium stores computer program instructions; when executed by a processor, these computer program instructions implement any of the information generation methods based on multimodal data in the above embodiments.

[0171] Furthermore, in conjunction with the information generation method based on multimodal data in the above embodiments, this application embodiment can provide a computer program product for implementation. When the instructions in this computer program product are executed by the processor of an electronic device, the electronic device implements any of the information generation methods based on multimodal data as described in the above embodiments.

[0172] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0173] The functional modules shown in the above-described block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0174] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0175] The flowcharts and / or block diagrams above, referring to embodiments of the information generation method, apparatus, storage medium, and product based on multimodal data according to the present disclosure, have described various aspects of the present disclosure. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to create a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowcharts and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0176] The above description is merely a specific implementation of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. An information generation system based on multimodal data, characterized in that, It includes a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it performs the following steps: Obtain a first modality set and a second modality set of the target object, wherein the first modality set includes the available modalities of the target object that have been acquired, and the second modality set includes the missing modalities of the target object that need to be compensated; Based on the available modal data of the available modalities, determine the first risk probability of the target object experiencing the target medical event; Based on the available modal data and the first risk probability, similar samples corresponding to the target object are determined from a preset multimodal sample library; Modal compensation is performed on the target object based on the modal characteristics of the similar samples under the missing modality to obtain modal compensation information, wherein the modal compensation information includes at least the compensated modal characteristics of the target object under the missing modality, the reference risk probability of the target object occurring the target medical event, and the compensation confidence of the modal compensation of the target object; The modal compensation information, the available modal data, and the first risk probability are fused to generate auxiliary reference information, which is used to assist medical personnel in monitoring the status of the target object.

2. The system according to claim 1, characterized in that, The processor is also used for: Acquire the physiological characteristics data of the target object and the time window information for monitoring the state of the target object; Feature extraction is performed on the available modal data to obtain available modal features; A query vector is constructed based on the available modality features, the first risk probability corresponding to the available modality, the physiological sign data, and the time window information; Obtain similar samples from the multimodal sample library whose vector similarity to the query vector meets preset conditions.

3. The system according to claim 2, characterized in that, The processor is also used for: When there are multiple similar samples, the initial compensation mode features of the multiple similar samples under the missing mode are obtained respectively; Based on the vector similarity between the multiple similar samples and the query vector, the retrieval weights corresponding to the multiple similar samples are determined; Based on the retrieval weights corresponding to the multiple similar samples, the multiple initial compensation modal features are weighted and summed to obtain the compensation modal features of the target object under the missing modality; Based on the retrieval weights, the risk probabilities corresponding to multiple initial compensation modal features are weighted and summed to obtain the reference risk probability of the target object experiencing the target medical event. The compensation confidence level is determined based on the feature similarity between the feature vectors corresponding to the similar samples and the query vector, as well as the completeness of the modality data corresponding to the missing modalities among the multiple similar samples.

4. The system according to claim 3, characterized in that, The processor is also used for: Bottleneck attention fusion is performed on the available modal data, the risk tokens corresponding to the available modal data, and the modal compensation information to obtain fusion features. The bottleneck attention fusion is used to perform cross-modal attention calculation on multiple modal tokens by using a preset learnable bottleneck token as a medium for cross-modal information interaction. Based on the fusion features, determine the target risk probability of the target object experiencing the target medical event; The auxiliary reference information is generated based on the target risk probability.

5. The system according to claim 4, characterized in that, The processor is also used for: The modal data, the first risk probability, and the modal compensation information in the first modal set are encoded respectively to obtain the first modal token, the risk token, and the compensation modal token; The first modal token, the risk token, and the compensation modal token are subjected to bottleneck attention fusion to obtain the fusion feature. The bottleneck attention fusion is used to perform cross-modal attention calculation on multiple modal tokens by using a preset number of learnable bottleneck tokens as mediators for cross-modal information interaction.

6. The system according to claim 5, characterized in that, The processor is also used for: When the compensation confidence corresponding to the compensation modality feature is less than the first confidence threshold, in the bottleneck attention mechanism, the attention weight corresponding to the compensation modality feature is adjusted from the first weight to the second weight, and / or the gating coefficient is adjusted from the first coefficient to the second coefficient, and / or the token corresponding to the compensation modality feature is masked, wherein the gating coefficient is used to characterize the information retention ratio of the token corresponding to the compensation modality feature before it is input into the bottleneck attention mechanism, the first weight is greater than the second weight, and the first coefficient is greater than the second coefficient.

7. The system according to claim 6, characterized in that, The processor is also used for: Obtain multiple similarities between the feature vectors corresponding to the multiple similar samples and the query vector; If the variance among the multiple similarities is less than a variance threshold and the compensation confidence is greater than or equal to a second confidence threshold, the bottleneck interaction weight corresponding to the compensation modality feature is adjusted from a third weight to a fourth weight, wherein the third weight is less than the fourth weight, the bottleneck interaction weight is used to characterize the attention weight between the token corresponding to the compensation modality feature and the bottleneck token, and the first confidence threshold is less than the second confidence threshold.

8. The system according to any one of claims 1 to 7, characterized in that, The processor is also used for: Obtain the risk identification model corresponding to the available modalities; The available modal data is input into the risk identification model to obtain the modal characteristics and initial risk probability of the target object under the available modalities output by the risk identification model; The initial risk probability is calibrated to obtain the first risk probability.

9. The system according to claim 8, characterized in that, The processor is also used for: Obtain the risk calibration function corresponding to the available modes; The initial risk probability is calibrated using the risk calibration function to obtain the first risk probability.

10. The system according to any one of claims 1 to 7, characterized in that, The processor is also used for: If the modal data of the target object is complete, a risk identification model corresponding to the available modal data is used to identify the risks in the available modal data to obtain the first risk probability; Bottleneck attention fusion is performed on the available modal data and the first risk probability to obtain fused features; Based on the fusion features, determine the target risk probability of the target object experiencing the target medical event; The auxiliary reference information is generated based on the target risk probability.

11. An information generation method based on multimodal data, characterized in that, The method, applied in a processor of the information generation system based on multimodal data according to any one of claims 1 to 10, comprises: Obtain a first modality set and a second modality set of the target object, wherein the first modality set includes the available modalities of the target object that have been acquired, and the second modality set includes the missing modalities of the target object that need to be compensated; Based on the available modal data of the available modalities, determine the first risk probability of the target object experiencing the target medical event; Based on the available modal data and the first risk probability, similar samples corresponding to the target object are determined from a preset multimodal sample library; Modal compensation is performed on the target object based on the modal characteristics of the similar samples under the missing modality to obtain modal compensation information, wherein the modal compensation information includes at least the compensated modal characteristics of the target object under the missing modality, the reference risk probability of the target object occurring the target medical event, and the compensation confidence of the modal compensation of the target object; The modal compensation information, the available modal data, and the first risk probability are fused to generate auxiliary reference information, which is used to assist medical personnel in monitoring the status of the target object.

12. An electronic device, characterized in that, The electronic device includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, it implements the information generation method based on multimodal data as described in claim 11.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the information generation method based on multimodal data as described in claim 11.

14. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device is able to perform the information generation method based on multimodal data as described in claim 11.

Images