Heart ultrasonic monitoring system and method based on machine learning

Abstract

The invention discloses a heart ultrasonic monitoring system and method based on machine learning, and relates to the technical field of heart ultrasonic monitoring. The method comprises the steps of obtaining multi-modal data, performing standardization processing on the multi-modal data, and then performing space-time alignment processing to obtain a space-time aligned multi-modal feature cube; performing primary judgment on the time-space aligned multi-modal feature cube to obtain a primary judgment result; if the initial judgment result is normal, outputting multi-modal characteristic data; if the initial judgment result is abnormal, performing feature fusion processing on the time-space aligned multi-modal feature cube based on a hierarchical feature extraction network to obtain a joint feature coding vector; and performing abnormal ultrasonic monitoring on the combined feature coding vector, and outputting a cardiac ultrasonic monitoring result, so that the time-space correlation features of the ultrasonic image, the electrocardiosignal, the blood pressure and the blood biochemical index can be effectively fused, an abnormal mode is automatically identified through machine learning, and the early warning capability and the diagnosis precision of heart diseases are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of cardiac ultrasonic monitoring, and in particular to a cardiac ultrasonic monitoring system and method based on machine learning. Background Art

[0002] With the development of science and technology, the application of image enhancement technology is becoming more and more extensive. For example, it can be applied to the enhancement of echocardiography in cardiac space-occupying detection. At present, when enhancing an image, the commonly used method is to enhance the image by using grayscale histogram equalization.

[0003] The Chinese invention patent with the announcement number CN118918112B discloses a method and system for detecting cardiac occupancy in ultrasound images based on artificial intelligence, including: obtaining an ultrasound cardiac image to be enhanced, mapping each preset direction based on the pixel value corresponding to the central window and each neighborhood window corresponding to the pixel point; determining the artifact change interference factor between the pixel point and each preset direction of the central window and each neighborhood window corresponding to the pixel point; determining the artifact interference influence index corresponding to the pixel point, and clustering the pixels in the initial image; performing regional segmentation on the initial image, determining the overall artifact interference factor corresponding to each sub-block; adjusting the preset scale parameter, and enhancing it through the Ret i nex algorithm, and performing cardiac occupancy detection on the target image. The present invention realizes the enhancement of the ultrasound cardiac image to be enhanced, improves the image enhancement effect, and thus improves the accuracy of cardiac occupancy detection.

[0004] However, existing monitoring methods have problems such as insufficient accuracy of single-modality cardiac ultrasound monitoring and asynchrony of multi-modality data. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a cardiac ultrasound monitoring system and method based on machine learning, which solves the problems of insufficient accuracy of single-modality cardiac ultrasound monitoring and asynchrony of multi-modal data in the existing monitoring methods.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a cardiac ultrasound monitoring method based on machine learning, comprising the following steps: acquiring multimodal data, standardizing the multimodal data, and then performing spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, wherein the multimodal data includes a 4D sequence cardiac ultrasound video, a 12-lead ECG signal, blood pressure data, and blood test data, and the spatiotemporal aligned multimodal feature cube includes an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block, and an aligned biochemical indicator multimodal data block; performing an initial judgment on the spatiotemporal aligned multimodal feature cube to obtain an initial judgment result; if the initial judgment result is normal, outputting the multimodal feature data; if the initial judgment result is abnormal, performing feature fusion processing on the spatiotemporal aligned multimodal feature cube based on a hierarchical feature extraction network to obtain a joint feature coding vector; performing abnormal ultrasound monitoring on the joint feature coding vector, and outputting a cardiac ultrasound monitoring result.

[0007] Furthermore, the multimodal data is standardized, including the following steps: for 4D cardiac ultrasound video sequence: using the improved Farneback optical flow algorithm to estimate the motion vector field between each frame of the 4D cardiac ultrasound video sequence, using the spatiotemporal coordinate transformation formula to perform spatiotemporal transformation on each voxel coordinate in the motion vector field, and obtaining a motion-compensated 4D sequence; the motion-compensated 4D sequence is processed by a cascaded residual network to obtain dynamic 3D volume data; for ECG signals: synchronously sampling the 12-lead ECG signals based on the set sampling frequency, removing the power frequency interference of the set Hz, Output the filtered ECG signal; for blood pressure data: use the variational mode decomposition algorithm for processing, decompose the blood pressure waveform into K different modes, each mode corresponds to a central frequency, extract blood pressure related features from the decomposed modes, and obtain the blood pressure digital feature vector; for blood test data: retain direct indicators related to cardiac function, and obtain derived indicators based on the direct indicators to obtain a set of blood candidate features, perform Spearman correlation coefficient processing on the blood candidate feature set, and retain blood features whose absolute value of correlation with ultrasound indicators is greater than the set threshold, which are recorded as the blood-heart ultrasound related feature subset.

[0008] Furthermore, a spatiotemporal alignment process is performed to obtain a spatiotemporal aligned multimodal feature cube, including the following steps: aligning the dynamic 3D volume data, the filtered ECG signal, the blood pressure digitization feature vector and the blood-heart ultrasound related feature subset in time to obtain a joint feature matrix; performing principal component analysis on the joint feature matrix to identify the principal components related to the ultrasound indexes to obtain a spatiotemporal aligned multimodal feature cube, including an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block and an aligned biochemical index multimodal data block.

[0009] Furthermore, a preliminary judgment is made on the spatiotemporally aligned multimodal feature cube to obtain a preliminary judgment result, including the following steps: data extraction is performed on the spatiotemporally aligned multimodal feature cube to obtain multimodal feature data, including ultrasound multimodal feature data, physiological signal multimodal feature data and biochemical indicator multimodal feature data; ultrasound multimodal feature abnormal data sets, physiological signal multimodal feature abnormal data sets and biochemical indicator multimodal feature abnormal data sets stored in the database are obtained; ultrasound multimodal feature data are compared with ultrasound multimodal feature abnormal data sets to determine the ultrasound multimodal feature abnormality level; physiological signal multimodal feature data are compared with physiological signal multimodal feature abnormal data sets to determine the physiological signal Abnormal level of multimodal features; compare and process the multimodal feature data of biochemical indicators with the multimodal feature abnormal data set of biochemical indicators to determine the abnormal level of multimodal features of biochemical indicators; if one of the abnormal level of ultrasound multimodal features, the abnormal level of physiological signals multimodal features and the abnormal level of biochemical indicators multimodal features is greater than the limited level, the initial judgment result is abnormal; if the abnormal level of ultrasound multimodal features, the abnormal level of physiological signals multimodal features and the abnormal level of biochemical indicators multimodal features are not greater than the limited level, the abnormal degree of single modal features is determined, if the abnormal degree of single modal features is less than the set abnormal threshold, the initial judgment result is abnormal, if the abnormal degree of single modal features is not less than the set abnormal threshold, the initial judgment result is normal.

[0010] Further, the ultrasound multimodal feature data includes ultrasound parameters, elastic imaging parameters and three-dimensional ultrasound parameters, and the ultrasound multimodal feature abnormality data set includes ultrasound multimodal feature abnormality data corresponding to a plurality of different ultrasound multimodal feature abnormality levels, including ultrasound abnormality parameters, elastic imaging abnormality parameters and three-dimensional ultrasound abnormality parameters;

[0011] Performing cosine similarity calculation on the ultrasonic multimodal feature data and each ultrasonic multimodal feature abnormal data to obtain each ultrasonic feature abnormal matching value, determining the ultrasonic multimodal feature abnormal data corresponding to the maximum ultrasonic feature abnormal matching value, and determining the corresponding ultrasonic multimodal feature abnormal level;

[0012] Cs=σ(cS,cC)+σ(tS,tC)+σ(wS,wC);

[0013] Wherein, Cs is the ultrasound feature abnormal matching value, cS is the ultrasound parameter, cC is the ultrasound abnormal parameter, tS is the elastic imaging parameter, tC is the elastic imaging abnormal parameter, wS is the three-dimensional ultrasound parameter, wC is, the three-dimensional ultrasound abnormal parameter σ(cS,cC) is the cosine similarity calculation function of cS and cC, σ(tS,tC) is the cosine similarity calculation function of tS and tC, σ(wS,wC) is the cosine similarity calculation function of wS and wC;

[0014] The physiological signal multimodal feature data includes heart rate xS, RR interval xRR, ST segment deviation xST and QRS complex width xQRS, the physiological signal multimodal feature abnormality data set includes physiological signal multimodal feature abnormality data corresponding to multiple different physiological signal multimodal feature abnormality levels, and the physiological signal multimodal feature abnormality data includes heart rate abnormality value cS, RR interval abnormality value cRR, ST segment deviation abnormality value cST and QRS complex width abnormality value cQRS;

[0015] Calculate the similarity between the multimodal feature data of the physiological signal and the multimodal feature abnormal data of each physiological signal to obtain the abnormal matching value of each physiological signal feature, determine the multimodal feature abnormal data of the physiological signal corresponding to the maximum physiological signal feature abnormal matching value, and determine the corresponding multimodal feature abnormal level of the physiological signal;

[0016] Cs=|xS-cS|+|xRR-cRR|+|xST-cST|+|xQRS-cQRS|;

[0017] Where, Cs is the abnormal matching value of physiological signal characteristics;

[0018] The multimodal characteristic data of biochemical indicators include blood glucose xX, blood lipids xZ, C-reactive protein xF and creatinine xG. The multimodal characteristic abnormal data set of biochemical indicators includes the multimodal characteristic abnormal data of biochemical indicators corresponding to different multimodal characteristic abnormal levels of biochemical indicators, including abnormal blood glucose value cX, abnormal blood lipid value cZ, abnormal C-reactive protein value cF and abnormal creatinine value cG.

[0019] Calculate the similarity between the biochemical indicator multimodal feature data and each biochemical indicator multimodal feature abnormal data to obtain each biochemical indicator feature abnormal matching value, determine the biochemical indicator multimodal feature abnormal data corresponding to the maximum biochemical indicator feature abnormal matching value, and determine the corresponding biochemical indicator multimodal feature abnormal level;

[0020] Ss=|xX-cX|+|xZ-cZ|+|xF-cF|+|xG-cG|;

[0021] Ss is the abnormal matching value of biochemical index characteristics.

[0022] Furthermore, if the abnormal level of the ultrasound multimodal feature, the abnormal level of the physiological signal multimodal feature and the abnormal level of the biochemical indicator multimodal feature are not greater than the limited level, the maximum value of the three is determined. If the abnormal level of the ultrasound multimodal feature is the maximum value, the cosine similarity calculation is performed based on the ultrasound multimodal feature data and the ultrasound multimodal feature parameter data stored in the database to obtain the degree of single modal feature abnormality.

[0023] Furthermore, feature fusion processing is performed on the spatiotemporally aligned multimodal feature cubes based on a hierarchical feature extraction network to obtain a joint feature coding vector, including the following steps: 3D ResNeXt block processing is performed on the aligned ultrasound multimodal data blocks based on the ultrasound branch of the hierarchical feature extraction network, and the obtained feature map is subjected to a global average pooling operation to compress the spatiotemporal dimension to 1 to obtain a feature vector, and then its dimension is adjusted to 256 through a fully connected layer to obtain an ultrasound modality independent feature vector;

[0024] The physiological signal branch based on the hierarchical feature extraction network performs feature processing and sequence processing on the aligned multimodal data blocks of physiological signals, and then adjusts its dimension to 256 through a fully connected layer to obtain the modality-independent feature vector of the physiological signal;

[0025] The biochemical index branch based on the hierarchical feature extraction network performs feature processing on the aligned biochemical index multimodal data blocks, and then adjusts its dimension to 256 through a fully connected layer to obtain the biochemical index modality independent feature vector;

[0026] The ultrasound modality independent feature vector, the physiological signal modality independent feature vector and the biochemical index modality independent feature vector are weightedly fused through the fusion weight vector stored in the database to obtain a joint feature coding vector.

[0027] Furthermore, the joint feature coding vector is subjected to abnormal ultrasound monitoring, and a cardiac ultrasound prediction report is output, including the following steps: positive and negative sample pairs are obtained based on the joint feature coding vector, and the initial encoder is trained to obtain a trained encoder; the joint feature coding vector is labeled, and the labeled joint feature coding vector is processed based on the trained encoder to generate an encoded high-dimensional feature vector; the encoded high-dimensional feature vector is clustered based on K-means clustering, and abnormal pattern centers corresponding to the number of annotations are output, each abnormal pattern center corresponds to a cardiac ultrasound monitoring result and corresponds to a parameterized high-dimensional feature vector after clustering; the joint feature coding vector to be monitored is input into the trained encoder to obtain the high-dimensional feature vector to be monitored, and the high-dimensional feature vector to be monitored is compared with each parameterized high-dimensional feature vector after clustering by cosine similarity one by one to obtain comparison similarity, and the maximum comparison similarity is determined, and the cardiac ultrasound monitoring result corresponding to the abnormal pattern center corresponding to the maximum comparison similarity is defined as the cardiac ultrasound monitoring result of the joint feature coding vector to be monitored.

[0028] Furthermore, positive and negative sample pairs are obtained based on the joint feature coding vector, including the following steps: masking some feature dimensions of the joint feature coding vector based on a random masking strategy, perturbing the unmasked features based on Gaussian noise to obtain positive samples; randomly extracting joint feature coding vectors from a pre-stored joint feature coding vector database of different patients as negative samples; the positive samples and negative samples constitute positive and negative sample pairs.

[0029] A cardiac ultrasound monitoring system based on machine learning, used for the above-mentioned cardiac ultrasound monitoring method based on machine learning, including a multimodal data processing module, an initial data judgment module, a joint feature coding vector acquisition module and a cardiac ultrasound monitoring result acquisition module, wherein: the multimodal data processing module is used to acquire multimodal data, and perform standardization processing on the multimodal data, and then perform spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, the multimodal data includes 4D sequence cardiac ultrasound video, 12-lead ECG signal, blood pressure data and blood test data, the spatiotemporal aligned multimodal feature cube includes alignment the aligned ultrasound multimodal data blocks, the aligned physiological signal multimodal data blocks and the aligned biochemical indicator multimodal data blocks; an initial data judgment module, used to make an initial judgment on the spatiotemporal aligned multimodal feature cubes to obtain an initial judgment result; if the initial judgment result is normal, the multimodal feature data is output; a joint feature coding vector acquisition module, used to perform feature fusion processing on the spatiotemporal aligned multimodal feature cubes based on a hierarchical feature extraction network to obtain a joint feature coding vector when the initial judgment result is abnormal; a cardiac ultrasound monitoring result acquisition module, used to perform abnormal ultrasound monitoring on the joint feature coding vectors and output cardiac ultrasound monitoring results.

[0030] The present invention has the following beneficial effects:

[0031] This machine learning-based cardiac ultrasound monitoring method achieves a comprehensive assessment of cardiac function. It can effectively integrate the spatiotemporal correlation characteristics of ultrasound images, electrocardiogram signals, blood pressure, and blood biochemical indicators. It automatically identifies abnormal patterns through machine learning, significantly improving the early warning capability and diagnostic accuracy of heart disease, and solving the problems of insufficient accuracy and asynchronous multimodal data in traditional single-modality cardiac ultrasound monitoring.

[0032] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is a flow chart of the cardiac ultrasound monitoring method based on machine learning of the present invention.

[0034] Figure 2 This is a flowchart of the cardiac ultrasound monitoring system based on machine learning of the present invention. DETAILED DESCRIPTION

[0035] See also Figure 1 , an embodiment of the present invention provides a technical solution: a cardiac ultrasound monitoring method based on machine learning, comprising the following steps: acquiring multimodal data, and performing standardization processing on the multimodal data, and then performing spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, wherein the multimodal data includes a 4D sequence cardiac ultrasound video, a 12-lead ECG signal, blood pressure data, and blood test data, and the spatiotemporal aligned multimodal feature cube includes an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block (including features of the 12-lead ECG signal and blood pressure data), and an aligned biochemical indicator multimodal data block;

[0036] For 4D cardiac ultrasound video sequences: the improved Farneback optical flow algorithm is used to estimate the motion vector field between each frame of the 4D cardiac ultrasound video sequence, and the spatiotemporal coordinate transformation formula is used to perform spatiotemporal transformation on each voxel coordinate in the motion vector field to obtain a motion-compensated 4D sequence, where the spatiotemporal coordinate transformation formula is expressed as P(t+Δt)=P(t)+V(t)*Δt*M, where P(t) is the voxel coordinate at time t, Δt is the time interval, P(t+Δt) is the voxel coordinate at t+Δt, V(t) represents the cloud information of each position in the 4D cardiac ultrasound video image at time t, which is estimated by the improved Farneback optical flow algorithm, and M is the spatial scaling matrix used to convert the displacement into actual spatial coordinate changes. The improved optical flow algorithm and spatiotemporal transformation eliminate motion artifacts, and the cascaded residual network enhances the details of dynamic 3D volume data to ensure the accuracy of cardiac motion features.

[0037] The motion-compensated 4D sequence is processed by a cascaded residual network to obtain dynamic 3D volume data. For ECG signals: the 12-lead ECG signals are synchronously sampled based on the set sampling frequency, the power frequency interference of the set Hz is removed, and the filtered ECG signals are output. Synchronous sampling and power frequency interference removal ensure the stability of the ECG signals and provide a reliable basis for abnormal detection such as arrhythmias.

[0038] For blood pressure data: the variational mode decomposition algorithm is used for processing. The blood pressure waveform is decomposed into K different modes, each mode corresponds to a central frequency, and blood pressure-related features such as systolic pressure, diastolic pressure and pulse pressure are extracted from the decomposed modes to obtain the digital feature vector of blood pressure; variational mode decomposition extracts multi-scale features, accurately captures blood pressure fluctuation patterns, and improves the ability to identify hemodynamic abnormalities.

[0039] For blood test data: retain direct indicators related to cardiac function, and obtain derivative indicators based on direct indicators to obtain a set of candidate blood features. Perform Spearman correlation coefficient processing on the set of candidate blood features, and retain blood features whose absolute value of correlation with ultrasound indicators is greater than the set threshold, which are recorded as blood-heart ultrasound related feature subsets. Retain indicators that are strongly related to cardiac function, reduce redundant information, and improve subsequent analysis efficiency and model robustness.

[0040] Performing spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube includes the following steps: aligning the dynamic 3D volume data, the filtered ECG signal, the blood pressure digitized feature vector and the blood-heart ultrasound related feature subset in time to obtain a joint feature matrix; performing principal component analysis on the joint feature matrix to identify the principal components related to the ultrasound index to obtain a spatiotemporal aligned multimodal feature cube, including an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block and an aligned biochemical index multimodal data block.

[0041] The dynamic 3D ultrasound volume, ECG signal, blood pressure and blood characteristics are accurately matched on the time axis to ensure the synchronization of multimodal data in the same cardiac cycle (such as ultrasound images during cardiac contraction and ECG R waves), providing a time reference for analyzing the relationship between mechanical movement and electrical activity. The principal components highly correlated with ultrasound indicators are extracted through dimensionality reduction, while retaining core information, reducing feature redundancy (such as from 192 dimensions to 12 dimensions), improving the computational efficiency and noise resistance of subsequent models, and ensuring the accuracy of abnormal pattern recognition. The time misalignment between modalities is eliminated through spatiotemporal alignment, and PCA is combined to enhance feature representativeness, which significantly improves the effectiveness of multimodal data fusion and provides more accurate input for cardiac ultrasound monitoring.

[0042] Perform a preliminary judgment on the spatiotemporally aligned multimodal feature cube to obtain a preliminary judgment result;

[0043] The method comprises the following steps: performing data extraction on the time-space aligned multimodal feature cube to obtain multimodal feature data, including ultrasound multimodal feature data, physiological signal multimodal feature data and biochemical index multimodal feature data; acquiring ultrasound multimodal feature abnormality data sets, physiological signal multimodal feature abnormality data sets and biochemical index multimodal feature abnormality data sets stored in a database; performing comparison processing on the ultrasound multimodal feature data and the ultrasound multimodal feature abnormality data sets to determine the ultrasound multimodal feature abnormality level; performing comparison processing on the physiological signal multimodal feature data and the physiological signal multimodal feature abnormality data sets to determine the physiological signal multimodal feature abnormality level. The abnormal level of biochemical indicators is determined by comparing the multimodal feature data of biochemical indicators with the abnormal data set of multimodal features of biochemical indicators to determine the abnormal level of multimodal features of biochemical indicators; if one of the abnormal level of ultrasound multimodal features, the abnormal level of physiological signals multimodal features and the abnormal level of biochemical indicators multimodal features is greater than the limited level, the initial judgment result is abnormal; if the abnormal level of ultrasound multimodal features, the abnormal level of physiological signals multimodal features and the abnormal level of biochemical indicators multimodal features are all not greater than the limited level, the abnormal degree of single modal features is determined, if the abnormal degree of single modal features is less than the set abnormal threshold, the initial judgment result is abnormal, if the abnormal degree of single modal features is not less than the set abnormal threshold, the initial judgment result is normal.

[0044] By comparing the abnormal data sets of ultrasound, physiological signals, and biochemical indicators, the system can quickly identify significant abnormalities in any modality, ensuring timely triggering of warnings and avoiding missed detections of a single modality. The cosine similarity is used to calculate the matching value of each modality and the abnormal data set, which can effectively measure the similarity of high-dimensional features (such as the directional consistency of ultrasound elastic parameters and abnormal patterns) and improve the accuracy of abnormal level classification.

[0045] When the multi-modal abnormality levels are within the limit, further calculation of the abnormality degree of single-modal features (such as the cosine similarity between ultrasound and normal parameter data) can capture potential risks (such as mild ventricular wall motion abnormalities).

[0046] The ultrasound multimodal feature data includes ultrasound parameters, elastic imaging parameters and three-dimensional ultrasound parameters, and the ultrasound multimodal feature abnormality data set includes ultrasound multimodal feature abnormality data corresponding to multiple different ultrasound multimodal feature abnormality levels, including ultrasound abnormality parameters, elastic imaging abnormality parameters and three-dimensional ultrasound abnormality parameters;

[0047] Performing cosine similarity calculation on the ultrasonic multimodal feature data and each ultrasonic multimodal feature abnormal data to obtain each ultrasonic feature abnormal matching value, determining the ultrasonic multimodal feature abnormal data corresponding to the maximum ultrasonic feature abnormal matching value, and determining the corresponding ultrasonic multimodal feature abnormal level;

[0048] Cs=σ(cS,cC)+σ(tS,tC)+σ(wS,wC);

[0049] Wherein, Cs is the ultrasound feature abnormal matching value, cS is the ultrasound parameter, cC is the ultrasound abnormal parameter, tS is the elastic imaging parameter, tC is the elastic imaging abnormal parameter, wS is the three-dimensional ultrasound parameter, wC is, the three-dimensional ultrasound abnormal parameter σ(cS,cC) is the cosine similarity calculation function of cS and cC, σ(tS,tC) is the cosine similarity calculation function of tS and tC, σ(wS,wC) is the cosine similarity calculation function of wS and wC;

[0050] The physiological signal multimodal feature data includes heart rate xS, RR interval xRR, ST segment deviation xST and QRS complex width xQRS, the physiological signal multimodal feature abnormality data set includes physiological signal multimodal feature abnormality data corresponding to multiple different physiological signal multimodal feature abnormality levels, and the physiological signal multimodal feature abnormality data includes heart rate abnormality value cS, RR interval abnormality value cRR, ST segment deviation abnormality value cST and QRS complex width abnormality value cQRS;

[0051] Calculate the similarity between the multimodal feature data of the physiological signal and the multimodal feature abnormal data of each physiological signal to obtain the abnormal matching value of each physiological signal feature, determine the multimodal feature abnormal data of the physiological signal corresponding to the maximum physiological signal feature abnormal matching value, and determine the corresponding multimodal feature abnormal level of the physiological signal;

[0052] Cs=|xS-cS|+|xRR-cRR|+|xST-cST|+|xQRS-cQRS|;

[0053] Where, Cs is the abnormal matching value of physiological signal characteristics;

[0054] The multimodal characteristic data of biochemical indicators include blood glucose xX, blood lipids xZ, C-reactive protein xF and creatinine xG. The multimodal characteristic abnormal data set of biochemical indicators includes the multimodal characteristic abnormal data of biochemical indicators corresponding to different multimodal characteristic abnormal levels of biochemical indicators, including abnormal blood glucose value cX, abnormal blood lipid value cZ, abnormal C-reactive protein value cF and abnormal creatinine value cG.

[0055] Calculate the similarity between the biochemical indicator multimodal feature data and each biochemical indicator multimodal feature abnormal data to obtain each biochemical indicator feature abnormal matching value, determine the biochemical indicator multimodal feature abnormal data corresponding to the maximum biochemical indicator feature abnormal matching value, and determine the corresponding biochemical indicator multimodal feature abnormal level;

[0056] Ss=|xX-cX|+|xZ-cZ|+|xF-cF|+|xG-cG|;

[0057] Ss is the abnormal matching value of biochemical index characteristics.

[0058] Abnormal levels are classified for ultrasound, physiological signals, and biochemical indicators respectively to ensure that significant abnormalities in any modality can be captured, reducing the risk of missed detection in a single modality. The cosine similarity calculation has low complexity and can quickly complete multi-modal comparison in the initial judgment stage, avoiding triggering complex in-depth analysis processes for normal data and significantly improving the system response speed.

[0059] If the abnormal level of ultrasound multimodal features, the abnormal level of physiological signal multimodal features and the abnormal level of biochemical index multimodal features are not greater than the limit level, the maximum value among the three is determined. If the abnormal level of ultrasound multimodal features is the maximum value, the abnormal degree of single modal features is obtained by calculating the cosine similarity based on the ultrasound multimodal feature data and the ultrasound multimodal feature parameter data stored in the database. When it is initially judged that all modal abnormality levels are within the limit, by calculating the cosine similarity of the single modal feature and the parameter data, it is possible to accurately capture slight abnormalities close to the threshold (such as a slight decrease in ultrasound ventricular wall motion velocity), avoiding missed diagnosis due to too strict threshold setting. The mode with the highest abnormality level is preferentially selected for comparison (such as only analyzing ultrasound when the ultrasound abnormality level is the maximum value), focusing on potential risk sources, reducing redundant calculations and improving efficiency. This step, as a supplementary mechanism for initial judgment, effectively reduces the probability of misjudgment caused by ignoring slight abnormalities of single modal features, and is particularly suitable for screening of latent abnormalities such as early cardiac function decline.

[0060] Similarly, if the abnormal level of the multimodal feature of the physiological signal is the maximum value, the degree of abnormality of the single modal feature is obtained by performing cosine similarity calculation based on the multimodal feature data of the physiological signal and the multimodal feature parameter data of the physiological signal stored in the database.

[0061] If the initial judgment result is normal, the multimodal feature data is output; if the initial judgment result is abnormal, the multimodal feature cube aligned in time and space is subjected to feature fusion processing based on the hierarchical feature extraction network to obtain a joint feature encoding vector;

[0062] Ultrasound branch based on hierarchical feature extraction network (3D ResNeXt structure, the improved 3D ResNeXt network is composed of multiple 3D ResNeXt blocks stacked together, each block contains multiple groups of convolutions, for the input ultrasound data block, first through an initial 3D convolution layer, the number of input channels is adjusted to an appropriate value, in each 3D ResNeXt block, the input feature map will be divided into multiple groups, each group for independent 3D convolution operation, and then each group of convolution results will be spliced ​​together, and then through a 1x1x1 3D convolution layer for channel fusion, finally through the residual connection, the input feature map and the fused feature map are added, after multiple 3D After being processed by the ResNeXt block, the obtained feature map is subjected to a global average pooling operation to compress the spatiotemporal dimension to 1, and a feature vector is obtained. The aligned ultrasound multimodal data block is processed by a 3DResNeXt block, and the obtained feature map is subjected to a global average pooling operation to compress the spatiotemporal dimension to 1, and a feature vector is obtained. The dimension is then adjusted to 256 through a fully connected layer to obtain an ultrasound modality-independent feature vector. The 3D ResNeXt block efficiently captures the three-dimensional structure and motion characteristics of the heart (such as ventricular wall motion velocity and strain rate) through multiple groups of convolutions and residual connections, and global average pooling compresses the spatiotemporal dimension to retain core information.

[0063] Based on the physiological signal branch of the hierarchical feature extraction network (CNN-LSTM hybrid architecture, the input physiological signal data block first passes through a CNN network. The CNN network consists of multiple convolutional layers and pooling layers. After convolution and pooling operations, the feature map obtained is input into the LSTM network. After being processed by the LSTM network, the hidden state at the last moment is taken) the aligned physiological signal multimodal data block is feature processed and sequenced, and then its dimension is adjusted to 256 through a fully connected layer to obtain the physiological signal modality independent feature vector; the CNN-LSTM hybrid architecture combines the local feature extraction of CNN with the time series modeling capability of LSTM to accurately capture the dynamic change patterns of ECG and blood pressure (such as ST segment deviation and blood pressure fluctuation trend).

[0064] The biochemical indicator branch based on the hierarchical feature extraction network (Transformer feature encoder, for the input biochemical indicator data block, first perform an embedding operation to map each feature to a high-dimensional vector space. The Transformer encoder consists of multiple encoding layers, each of which contains a multi-head self-attention mechanism and a feedforward neural network. After processing by multiple encoding layers, the encoded features are obtained, and global average pooling is performed to obtain a feature vector) performs feature processing on the aligned biochemical indicator multimodal data blocks, and then adjusts its dimension to 256 through a fully connected layer to obtain a modality-independent feature vector of the biochemical indicator. The Transformer encoder uses the self-attention mechanism to mine the long-range dependency between blood indicators (such as the synergistic changes of NT-proBNP and creatinine) to enhance the feature representation capability.

[0065] The ultrasound modality-independent feature vector, the physiological signal modality-independent feature vector, and the biochemical index modality-independent feature vector are weighted and fused through the fusion weight vector stored in the database to obtain a joint feature encoding vector. The features of each modality are uniformly compressed to 256 dimensions, significantly reducing feature redundancy (such as from ultrasound 128 dimensions + physiological 24 dimensions + biochemical 8 dimensions → 256 dimensions after fusion), and improving the efficiency of subsequent clustering and comparison. Based on the fusion weight vector stored in the database, the contribution of each modality is dynamically adjusted according to the clinical scenario (such as increasing the weight of ultrasound and biochemical indicators in acute heart failure) to enhance the sensitivity of the model to specific pathologies.

[0066] Perform abnormal ultrasound monitoring on the joint feature coding vector and output the cardiac ultrasound monitoring results.

[0067] Based on the joint feature coding vector, positive and negative sample pairs are obtained, and the initial encoder is trained to obtain a trained encoder; the joint feature coding vector is labeled (a large number of joint feature coding vectors with different pathological conditions are collected and labeled by medical experts to determine a variety of different pathological combinations), and the labeled joint feature coding vector is processed based on the trained encoder to generate an encoded high-dimensional feature vector; the encoded high-dimensional feature vector is clustered based on K-means clustering, and the abnormal pattern center corresponding to the number of annotations is output, and each abnormal pattern center corresponds to a cardiac ultrasound monitoring result and corresponds to a parameterized high-dimensional feature vector after clustering; the joint feature coding vector to be monitored is input into the trained encoder to obtain the high-dimensional feature vector to be monitored, and the high-dimensional feature vector to be monitored is compared with each parameterized high-dimensional feature vector after clustering by cosine similarity one by one to obtain the comparison similarity, and the maximum comparison similarity is determined, and the cardiac ultrasound monitoring result corresponding to the abnormal pattern center corresponding to the maximum comparison similarity is defined as the cardiac ultrasound monitoring result of the joint feature coding vector to be monitored.

[0068] The encoder encodes the joint features into a high-dimensional vector, effectively integrating the spatiotemporal correlation information of ultrasound, ECG, blood pressure and blood indicators, reducing redundancy (such as compression from 256 dimensions to 42 centers after clustering), while retaining the ability to distinguish pathology. K-means clustering generates 42 abnormal pattern centers, each of which represents a typical combination of pathological features (such as abnormal wall motion + ST segment elevation + elevated troponin), providing a standardized reference template for clinical diagnosis.

[0069] The method of obtaining positive and negative sample pairs based on the joint feature coding vector includes the following steps: masking some feature dimensions of the joint feature coding vector based on a random masking strategy, perturbing the unmasked features based on Gaussian noise to obtain positive samples; randomly extracting joint feature coding vectors from a pre-stored database of joint feature coding vectors of different patients as negative samples; and constructing positive and negative sample pairs with positive samples and negative samples. The encoder is trained to learn the difference between normal and abnormal features by random masking and Gaussian noise, thereby improving the sensitivity of the model to pathological features.

[0070] A cardiac ultrasound monitoring system based on machine learning, used for the above-mentioned cardiac ultrasound monitoring method based on machine learning, such as Figure 2 As shown, it includes a multimodal data processing module, an initial data judgment module, a joint feature coding vector acquisition module and a cardiac ultrasound monitoring result acquisition module, wherein: the multimodal data processing module is used to acquire multimodal data, and perform standardization processing on the multimodal data, and then perform spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, the multimodal data includes a 4D sequence cardiac ultrasound video, a 12-lead ECG signal, blood pressure data and blood test data, and the spatiotemporal aligned multimodal feature cube includes an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block (ECG signal, blood pressure signal) and aligned biochemical index multimodal data blocks; an initial data judgment module, used to make an initial judgment on the spatiotemporal aligned multimodal feature cube to obtain an initial judgment result; if the initial judgment result is normal, the multimodal feature data is output; a joint feature coding vector acquisition module, used to perform feature fusion processing on the spatiotemporal aligned multimodal feature cube based on a hierarchical feature extraction network when the initial judgment result is abnormal, to obtain a joint feature coding vector; a cardiac ultrasound monitoring result acquisition module, used to perform abnormal ultrasound monitoring on the joint feature coding vector and output the cardiac ultrasound monitoring result.

[0071] An electronic device comprises: a processor; and a memory, wherein computer program instructions are stored in the memory, and when the computer program instructions are executed by the processor, the processor executes the cardiac ultrasound monitoring method based on machine learning as described above.

[0072] A computer-readable storage medium is used to store a program, which, when executed by a processor, implements the cardiac ultrasound monitoring method based on machine learning as described above.

[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0074] The present invention is described with reference to flowcharts and / or block diagrams of systems, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0077] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0078] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A cardiac ultrasound monitoring method based on machine learning, characterized in that: The following steps are involved: Acquire multimodal data, perform standardization processing on the multimodal data, and then perform spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, wherein the multimodal data includes a 4D sequence cardiac ultrasound video, a 12-lead ECG signal, blood pressure data, and blood test data, and the spatiotemporal aligned multimodal feature cube includes an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block, and an aligned biochemical indicator multimodal data block; Perform a preliminary judgment on the spatiotemporally aligned multimodal feature cube to obtain a preliminary judgment result; If the initial judgment result is normal, the multimodal feature data is output; If the initial judgment result is abnormal, the multimodal feature cube aligned in time and space is subjected to feature fusion processing based on the hierarchical feature extraction network to obtain a joint feature encoding vector; Perform abnormal ultrasound monitoring on the joint feature coding vector and output the cardiac ultrasound monitoring result.

2. The method for cardiac ultrasound monitoring based on machine learning according to claim 1, characterized in that: The multimodal data is standardized, including the following steps: For 4D cardiac ultrasound video sequences: the improved Farneback optical flow algorithm is used to estimate the motion vector field between each frame of the 4D cardiac ultrasound video sequence, and the temporal and spatial coordinate transformation formula is used to perform temporal and spatial transformation on each voxel coordinate in the motion vector field to obtain a motion-compensated 4D sequence; the motion-compensated 4D sequence is processed by a cascaded residual network to obtain dynamic 3D volume data; For ECG signals: synchronously sample the 12-lead ECG signals based on the set sampling frequency, remove the power frequency interference of the set Hz, and output the filtered ECG signals; For blood pressure data: the variational mode decomposition algorithm is used for processing. The blood pressure waveform is decomposed into K different modes. Each mode corresponds to a central frequency. Blood pressure related features are extracted from the decomposed modes to obtain the blood pressure digital feature vector. For blood test data: retain direct indicators related to cardiac function, and obtain derivative indicators based on the direct indicators to obtain a set of blood candidate features. Perform Spearman correlation coefficient processing on the blood candidate feature set, and retain blood features whose absolute value of correlation with ultrasound indicators is greater than the set threshold, which are recorded as the blood-heart ultrasound related feature subset.

3. The method for cardiac ultrasound monitoring based on machine learning according to claim 2, characterized in that: Performing spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube includes the following steps: The dynamic 3D volume data, the filtered ECG signal, the blood pressure digitized feature vector and the blood-heart ultrasound related feature subset are aligned in time to obtain a joint feature matrix; The principal component analysis is performed on the joint feature matrix to identify the principal components related to the ultrasound indexes, and a spatiotemporally aligned multimodal feature cube is obtained, including the aligned ultrasound multimodal data blocks, the aligned physiological signal multimodal data blocks and the aligned biochemical index multimodal data blocks.

4. The method for cardiac ultrasound monitoring based on machine learning according to claim 1, characterized in that: Performing an initial judgment on the spatiotemporally aligned multimodal feature cube to obtain an initial judgment result includes the following steps: Extract data from the spatiotemporally aligned multimodal feature cube to obtain multimodal feature data, including ultrasound multimodal feature data, physiological signal multimodal feature data, and biochemical index multimodal feature data; Acquire an ultrasound multimodal feature abnormality data set, a physiological signal multimodal feature abnormality data set, and a biochemical index multimodal feature abnormality data set stored in a database; Compare and process the ultrasound multimodal feature data with the ultrasound multimodal feature abnormality data set to determine the ultrasound multimodal feature abnormality level; Comparing the physiological signal multimodal feature data with the physiological signal multimodal feature abnormality data set to determine the physiological signal multimodal feature abnormality level; Compare and process the biochemical index multimodal feature data with the biochemical index multimodal feature abnormal data set to determine the biochemical index multimodal feature abnormality level; If one of the abnormal level of ultrasound multimodal features, the abnormal level of physiological signal multimodal features and the abnormal level of biochemical index multimodal features is greater than the limit level, the initial judgment result is abnormal; If the abnormality level of the ultrasound multimodal characteristics, the abnormality level of the physiological signal multimodal characteristics and the abnormality level of the biochemical index multimodal characteristics are not greater than the limited level, the abnormality degree of the single modal characteristics is determined. If the abnormality degree of the single modal characteristics is less than the set abnormality threshold, the initial judgment result is abnormal. If the abnormality degree of the single modal characteristics is not less than the set abnormality threshold, the initial judgment result is normal.

5. The method for cardiac ultrasound monitoring based on machine learning according to claim 4, characterized in that: The ultrasound multimodal feature data includes ultrasound parameters, elastic imaging parameters and three-dimensional ultrasound parameters, and the ultrasound multimodal feature abnormality data set includes ultrasound multimodal feature abnormality data corresponding to multiple different ultrasound multimodal feature abnormality levels, including ultrasound abnormality parameters, elastic imaging abnormality parameters and three-dimensional ultrasound abnormality parameters; Performing cosine similarity calculation on the ultrasonic multimodal feature data and each ultrasonic multimodal feature abnormal data to obtain each ultrasonic feature abnormal matching value, determining the ultrasonic multimodal feature abnormal data corresponding to the maximum ultrasonic feature abnormal matching value, and determining the corresponding ultrasonic multimodal feature abnormal level; Cs=σ(cS,cC)+σ(tS,tC)+σ(wS,wC); Wherein, Cs is the ultrasound feature abnormality matching value, cS is the ultrasound parameter, cC is the ultrasound abnormality parameter, tS is the elastic imaging parameter, tC is the elastic imaging abnormality parameter, wS is the three-dimensional ultrasound parameter, wC is the three-dimensional ultrasound abnormality parameter, σ(cS,cC) is the cosine similarity calculation function of cS and cC, σ(tS,tC) is the cosine similarity calculation function of tS and tC, σ(wS,wC) is the cosine similarity calculation function of wS and wC; The physiological signal multimodal feature data includes heart rate xS, RR interval xRR, ST segment deviation xST and QRS complex width xQRS, the physiological signal multimodal feature abnormality data set includes physiological signal multimodal feature abnormality data corresponding to multiple different physiological signal multimodal feature abnormality levels, and the physiological signal multimodal feature abnormality data includes heart rate abnormality value cS, RR interval abnormality value cRR, ST segment deviation abnormality value cST and QRS complex width abnormality value cQRS; Calculate the similarity between the multimodal feature data of the physiological signal and the multimodal feature abnormal data of each physiological signal to obtain the abnormal matching value of each physiological signal feature, determine the multimodal feature abnormal data of the physiological signal corresponding to the maximum physiological signal feature abnormal matching value, and determine the corresponding multimodal feature abnormal level of the physiological signal; Cs=|xS-cS|+|xRR-cRR|+|xST-cST|+|xQRS-cQRS|; Where, Cs is the abnormal matching value of physiological signal characteristics; The multimodal characteristic data of biochemical indicators include blood glucose xX, blood lipids xZ, C-reactive protein xF and creatinine xG. The multimodal characteristic abnormal data set of biochemical indicators includes the multimodal characteristic abnormal data of biochemical indicators corresponding to different multimodal characteristic abnormal levels of biochemical indicators, including abnormal blood glucose value cX, abnormal blood lipid value cZ, abnormal C-reactive protein value cF and abnormal creatinine value cG. Calculate the similarity between the biochemical indicator multimodal feature data and each biochemical indicator multimodal feature abnormal data to obtain each biochemical indicator feature abnormal matching value, determine the biochemical indicator multimodal feature abnormal data corresponding to the maximum biochemical indicator feature abnormal matching value, and determine the corresponding biochemical indicator multimodal feature abnormal level; Ss=|xX-cX|+|xZ-cZ|+|xF-cF|+|xG-cG|; Ss is the abnormal matching value of biochemical index characteristics.

6. The method for cardiac ultrasound monitoring based on machine learning according to claim 5, characterized in that: If the abnormal level of the ultrasound multimodal feature, the abnormal level of the physiological signal multimodal feature and the abnormal level of the biochemical indicator multimodal feature are not greater than the limited level, the maximum value of the three is determined. If the abnormal level of the ultrasound multimodal feature is the maximum value, the cosine similarity calculation is performed based on the ultrasound multimodal feature data and the ultrasound multimodal feature parameter data stored in the database to obtain the degree of single modal feature abnormality.

7. The method for cardiac ultrasound monitoring based on machine learning according to claim 1, characterized in that: Based on the hierarchical feature extraction network, feature fusion processing is performed on the spatiotemporally aligned multimodal feature cube to obtain a joint feature encoding vector, including the following steps: The ultrasound branch based on the hierarchical feature extraction network performs 3DResNeXt block processing on the aligned ultrasound multimodal data blocks. The obtained feature map is subjected to a global average pooling operation to compress the spatiotemporal dimension to 1 to obtain a feature vector, which is then resized to 256 through a fully connected layer to obtain an ultrasound modality-independent feature vector. The physiological signal branch based on the hierarchical feature extraction network performs feature processing and sequence processing on the aligned multimodal data blocks of physiological signals, and then adjusts its dimension to 256 through a fully connected layer to obtain the modality-independent feature vector of the physiological signal; The biochemical index branch based on the hierarchical feature extraction network performs feature processing on the aligned biochemical index multimodal data blocks, and then adjusts its dimension to 256 through a fully connected layer to obtain the biochemical index modality independent feature vector; The ultrasound modality independent feature vector, the physiological signal modality independent feature vector and the biochemical index modality independent feature vector are weightedly fused through the fusion weight vector stored in the database to obtain a joint feature coding vector.

8. The method for cardiac ultrasound monitoring based on machine learning according to claim 1, characterized in that: Performing abnormal ultrasound monitoring on the joint feature coding vector and outputting a cardiac ultrasound prediction report includes the following steps: Based on the joint feature coding vector, positive and negative sample pairs are obtained, and the initial encoder is trained to obtain a trained encoder; The joint feature coding vector is labeled, and the labeled joint feature coding vector is processed based on the trained encoder to generate a high-dimensional feature vector after encoding; Perform cluster analysis on the encoded high-dimensional feature vector based on K-means clustering, and output the abnormal pattern center corresponding to the number of annotations. Each abnormal pattern center corresponds to a cardiac ultrasound monitoring result and a high-dimensional feature vector after clustering. The joint feature coding vector to be monitored is input into the trained encoder to obtain the high-dimensional feature vector to be monitored, and the cosine similarity is compared between the high-dimensional feature vector to be monitored and the high-dimensional feature vectors after each parameter clustering one by one to obtain the comparison similarity, and the maximum comparison similarity is determined. The cardiac ultrasound monitoring result corresponding to the abnormal pattern center corresponding to the maximum comparison similarity is defined as the cardiac ultrasound monitoring result of the joint feature coding vector to be monitored.

9. The method for cardiac ultrasound monitoring based on machine learning according to claim 8, characterized in that: Obtaining positive and negative sample pairs based on the joint feature encoding vector includes the following steps: Based on the random masking strategy, some feature dimensions of the joint feature encoding vector are masked, and the unmasked features are perturbed based on Gaussian noise to obtain positive samples; Randomly extract joint feature encoding vectors from a pre-stored database of joint feature encoding vectors of different patients as negative samples; Positive samples and negative samples constitute positive and negative sample pairs.

10. A cardiac ultrasound monitoring system based on machine learning, used in a cardiac ultrasound monitoring method based on machine learning according to any one of claims 1 to 9, characterized in that: It includes a multimodal data processing module, an initial data judgment module, a joint feature coding vector acquisition module and a cardiac ultrasound monitoring result acquisition module, wherein: A multimodal data processing module is used to acquire multimodal data, perform standardization processing on the multimodal data, and then perform spatiotemporal alignment processing to obtain a spatiotemporal aligned multimodal feature cube, wherein the multimodal data includes a 4D sequence cardiac ultrasound video, a 12-lead ECG signal, blood pressure data, and blood test data, and the spatiotemporal aligned multimodal feature cube includes an aligned ultrasound multimodal data block, an aligned physiological signal multimodal data block, and an aligned biochemical indicator multimodal data block; The initial data judgment module is used to make an initial judgment on the spatiotemporally aligned multimodal feature cube to obtain an initial judgment result; if the initial judgment result is normal, the multimodal feature data is output; The joint feature coding vector acquisition module is used to perform feature fusion processing on the spatiotemporally aligned multimodal feature cube based on the hierarchical feature extraction network to obtain the joint feature coding vector when the initial judgment result is abnormal; The cardiac ultrasound monitoring result acquisition module is used to perform abnormal ultrasound monitoring on the joint feature coding vector and output the cardiac ultrasound monitoring result.

Images