Left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion

Abstract

The invention discloses a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion, and belongs to the field of medical image analysis. The problem that an existing deep learning method is poor in left ventricular ejection fraction prediction accuracy is solved. According to the method, the A4C view angle echocardiogram data and the PSAX view angle echocardiogram data are subjected to independent model pre-training, and then the pre-trained model parameters are subjected to transfer learning, so that a better initialization effect can be obtained, and model convergence and performance optimization are accelerated. The double-view-angle fusion regression module can well combine spatial and temporal features output by the two single-view-angle models, feature relevance of echocardiogram data of the two view angles is greatly utilized, and the accuracy of final model prediction can be improved. In addition, through fusion of the multi-view echocardiogram data features, the influence of anatomical structure differences on the detection result is effectively reduced. The method can be applied to left ventricular ejection fraction regression prediction.

Description

Technical Field

[0001] This invention belongs to the field of medical image analysis, specifically relating to a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion. Background Technology

[0002] Left ventricular ejection fraction (LVEF) is a crucial indicator for assessing cardiac function, reflecting the proportion of blood pumped out by the heart with each stroke relative to the volume of the left ventricle. Accurate LVEF assessment is essential for the diagnosis, treatment, and follow-up of heart disease. Traditional LVEF estimation methods rely on manual measurement and physician experience, which is time-consuming and prone to subjective errors. Furthermore, due to significant differences in cardiac anatomy between children and adults, ultrasound video quality fluctuates considerably, limiting the performance of existing model-based methods. Moreover, most current deep learning-based LVEF regression methods predict LVEF from a single perspective. Even when using data from two perspectives, the predictions are merely weighted averages, failing to fully integrate information from different perspectives or adequately learn the correlation between the two perspectives. Therefore, the accuracy of current deep learning methods for LVEF prediction remains poor, especially in pediatric cases. Summary of the Invention

[0003] The purpose of this invention is to address the problem of poor accuracy in predicting left ventricular ejection fraction using existing deep learning methods, and to propose a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion.

[0004] The technical solution adopted by this invention to solve the above-mentioned technical problems is: a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion, the system comprising an echocardiogram data acquisition module, a first image feature learning module, a first time feature learning module, a first viewpoint-specific regression module, a second image feature learning module, a second time feature learning module, a second viewpoint-specific regression module, and a dual-view fusion regression module, wherein:

[0005] The echocardiogram data acquisition module is used to acquire an echocardiogram dataset consisting of echocardiogram data from the apical four-chamber view and echocardiogram data from the short-axis view of the papillary muscles.

[0006] The apical four-chamber view data in the echocardiogram dataset were divided into three parts: training set, validation set, and test set. The papillary muscle short-axis view data in the echocardiogram dataset were then divided into three parts: training set, validation set, and test set. The data were combined according to the division results to obtain all the echocardiogram data in the training set, all the echocardiogram data in the validation set, and all the echocardiogram data in the test set.

[0007] During the training process:

[0008] The first image feature learning module is used to perform frame-by-frame processing on the apical four-chamber view data in the training set, and then extract the features of each image obtained by the frame-by-frame processing.

[0009] Using the first The feature vectors of all images corresponding to the apical four-chamber view data in a pair of echocardiogram data form a spatial feature vector sequence. ;

[0010] The second image feature learning module is used to perform frame segmentation on the short-axis view data of the nipple muscle in the training set, and then extract the features of each image obtained by the frame segmentation.

[0011] Using the first The feature vectors of all images corresponding to the short-axis view data of the papillary muscle in a pair of echocardiogram data form a spatial-level feature vector sequence. ;

[0012] The first temporal feature learning module is used to learn based on the first The spatial feature vector sequence of the apical four-chamber view data in the first pair of echocardiographic data was used to extract the first... A sequence of spatiotemporal feature vectors of apical four-chamber view data in an echocardiographic data pair;

[0013] The second time feature learning module is used to learn based on the first The spatial feature vector sequence of the nipple muscle short-axis view data in the first echocardiographic data pair was used to extract the first... A sequence of spatiotemporal feature vectors of short-axis view data of the papillary muscle in a pair of echocardiographic data;

[0014] The first perspective-specific regression module is used to output the ejection fraction value regression prediction result of the apical four-chamber perspective data based on the spatiotemporal feature vector sequence output by the first time feature learning module.

[0015] The working process of the first-view specific regression module is as follows:

[0016]

[0017] in, For the first A sequence of spatiotemporal feature vectors of apical four-chamber view data from a pair of echocardiographic data. The prediction results for a first-person perspective regression module. For activation function, For batch normalization;

[0018] The second perspective-specific regression module is used to output the ejection fraction value regression prediction result of the nipple muscle short-axis perspective data based on the spatiotemporal feature vector sequence output by the second time feature learning module.

[0019] The working process of the second-perspective specific regression module is as follows:

[0020]

[0021] in, For the first A sequence of spatiotemporal feature vectors of short-axis view data of the papillary muscle from a pair of echocardiographic data. The prediction results for a second-view specific regression module;

[0022] The dual-view fusion regression module is used to fuse the spatiotemporal feature vector sequences output by the first time feature learning module and the second time feature learning module, and obtain the ejection fraction numerical regression prediction result based on the feature fusion result.

[0023] The working process of the dual-view fusion regression module is as follows:

[0024]

[0025]

[0026]

[0027]

[0028] in, The prediction results are from the dual-view fusion regression module. For multi-headed cross-attention;

[0029] The first image feature learning module, the second image feature learning module, the first time feature learning module, the second time feature learning module, the first viewpoint-specific regression module, the second viewpoint-specific regression module, and the dual-viewpoint fusion regression module are used to process each pair of echocardiogram data in the training set. Then, the total loss function value is calculated based on the prediction results of the first viewpoint-specific regression module, the second viewpoint-specific regression module, and the dual-viewpoint fusion regression module. The network parameter values ​​of each module are adjusted based on the total loss function value.

[0030] The echocardiographic data from the validation and test sets were used to validate and test each module until the parameters of each module were finally trained.

[0031] During the testing process:

[0032] For a single patient, echocardiographic data from the apical four-chamber view and the papillary muscle short-axis view were collected separately. The apical four-chamber view data was used as input to a trained first image feature learning module, and the papillary muscle short-axis view data was used as input to a trained second image feature learning module. The output of the trained first image feature learning module was then used as input to a trained first temporal feature learning module, and the output of the trained second image feature learning module was used as input to a trained second temporal feature learning module. The outputs of the trained first and second temporal feature learning modules were then used as input to a trained dual-view fusion regression module, which outputs the predicted left ventricular ejection fraction.

[0033] The beneficial effects of this invention are:

[0034] This invention employs separate model pre-training on A4C and PSAX echocardiographic data, followed by transfer learning of the pre-trained model parameters. This allows for better initialization of the model, accelerating model convergence and performance optimization. The dual-view fusion regression module effectively combines the spatiotemporal features output from the two single-view models, maximizing the feature correlation between the two echocardiographic data and improving the accuracy of the final model prediction. Based on transfer learning and multi-view echocardiographic data fusion, this invention achieves accurate prediction of left ventricular ejection fraction. Furthermore, by fusing features from multi-view echocardiographic data, it effectively reduces the impact of anatomical differences on the detection results. This method is also highly applicable to left ventricular ejection fraction prediction in children. Attached Figure Description

[0035] Figure 1 This is a flowchart of a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to the present invention;

[0036] Image encoder, Time encoder, Cross-attention, and regressor. Detailed Implementation

[0037] Specific implementation method one: CombiningFigure 1 This embodiment describes a left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion. The system includes an echocardiogram data acquisition module, a first image feature learning module, a first temporal feature learning module, a first viewpoint-specific regression module, a second image feature learning module, a second temporal feature learning module, a second viewpoint-specific regression module, and a dual-view fusion regression module, wherein:

[0038] The echocardiogram data acquisition module is used to acquire an echocardiogram dataset consisting of echocardiogram data from the apical four-chamber (A4C) view and echocardiogram data from the papillary muscle short axis (PSAX) view.

[0039] It should be noted that, in order to improve the generalization performance of the model, the acquired echocardiogram dataset should include both pediatric and adult echocardiogram data.

[0040] The apical four-chamber view data in the echocardiogram dataset were divided into three parts: training set, validation set, and test set. The papillary muscle short-axis view data in the echocardiogram dataset were then divided into three parts: training set, validation set, and test set. The data were combined according to the division results to obtain all the echocardiogram data in the training set, all the echocardiogram data in the validation set, and all the echocardiogram data in the test set.

[0041] It should be noted that during the acquisition of echocardiographic data, it is necessary to acquire both apical four-chamber view data and papillary muscle short-axis view data for the same person. Furthermore, the apical four-chamber view data and papillary muscle short-axis view data for the same person should be in the same set, which can be the training set, validation set, or test set.

[0042] During the training process:

[0043] The first image feature learning module is used to perform frame-by-frame processing on the apical four-chamber view data in the training set, and then extract the features of each image obtained by the frame-by-frame processing.

[0044] Using the first The feature vectors of all images corresponding to the apical four-chamber view data in a pair of echocardiogram data form a spatial feature vector sequence. , recorded as ;

[0045] The second image feature learning module is used to perform frame segmentation on the short-axis view data of the nipple muscle in the training set, and then extract the features of each image obtained by the frame segmentation.

[0046] Using the first The feature vectors of all images corresponding to the short-axis view data of the papillary muscle in a pair of echocardiogram data form a spatial-level feature vector sequence. , recorded as ;

[0047] The first temporal feature learning module is used to learn based on the first The spatial feature vector sequence of the apical four-chamber view data in the first pair of echocardiographic data was used to extract the first... A sequence of spatiotemporal feature vectors of apical four-chamber view data in an echocardiographic data pair;

[0048] The second time feature learning module is used to learn based on the first The spatial feature vector sequence of the nipple muscle short-axis view data in the first echocardiographic data pair was used to extract the first... A sequence of spatiotemporal feature vectors of short-axis view data of the papillary muscle in a pair of echocardiographic data;

[0049] The first perspective-specific regression module is used to output the ejection fraction value regression prediction result of the apical four-chamber perspective data based on the spatiotemporal feature vector sequence output by the first time feature learning module.

[0050] The working process of the first-view specific regression module is as follows:

[0051]

[0052] in, For the first A sequence of spatiotemporal feature vectors of apical four-chamber view data from a pair of echocardiographic data. The prediction results for a first-person perspective regression module. For activation function, For batch normalization, the input and output feature dimensions of the first multilayer perceptron are both 512, and the output feature dimension of the second multilayer perceptron is 1.

[0053] The second perspective-specific regression module is used to output the ejection fraction value regression prediction result of the nipple muscle short-axis perspective data based on the spatiotemporal feature vector sequence output by the second time feature learning module.

[0054] The working process of the second-perspective specific regression module is as follows:

[0055]

[0056] in, For the first A sequence of spatiotemporal feature vectors of short-axis view data of the papillary muscle from a pair of echocardiographic data. The prediction results for a second-view specific regression module;

[0057] The dual-view fusion regression module is used to fuse the spatiotemporal feature vector sequences output by the first time feature learning module and the second time feature learning module, and obtain the ejection fraction numerical regression prediction result based on the feature fusion result.

[0058] The working process of the dual-view fusion regression module is as follows:

[0059]

[0060]

[0061]

[0062]

[0063] in, The prediction results are from the dual-view fusion regression module. For multi-head cross-attention, the fourth multilayer perceptron outputs a feature with a dimension of 1.

[0064] Multiple Cross-Attention (MCA) consists of several single-head Cross-Attention (CA) functions in parallel. In the cross-attention mechanism, temporal features from the A4C perspective are used as the Query, and temporal features from the PSAX perspective are used as the Key and Value. The calculation of single-head cross-attention is as follows:

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] When calculating the multi-head cross attention mechanism, the 512 in the first dimension needs to be divided into four parts, and the single-head cross attention calculation is performed separately for each part. Finally, the parts are merged back into 512 in the first dimension.

[0071] The first image feature learning module, the second image feature learning module, the first time feature learning module, the second time feature learning module, the first viewpoint-specific regression module, the second viewpoint-specific regression module, and the dual-viewpoint fusion regression module are used to process each pair of echocardiogram data in the training set. Then, the total loss function value is calculated based on the prediction results of the first viewpoint-specific regression module, the second viewpoint-specific regression module, and the dual-viewpoint fusion regression module. The network parameter values ​​of each module are adjusted based on the total loss function value.

[0072] The echocardiographic data from the validation and test sets were used to validate and test each module until the parameters of each module were finally trained.

[0073] During the testing process:

[0074] For a single patient, echocardiographic data from the apical four-chamber view and the papillary muscle short-axis view were collected separately. The apical four-chamber view data was used as input to a trained first image feature learning module, and the papillary muscle short-axis view data was used as input to a trained second image feature learning module. The output of the trained first image feature learning module was then used as input to a trained first temporal feature learning module, and the output of the trained second image feature learning module was used as input to a trained second temporal feature learning module. The outputs of the trained first and second temporal feature learning modules were then used as input to a trained dual-view fusion regression module, which outputs the predicted left ventricular ejection fraction.

[0075] In this invention, the structures of the first image feature learning module, the first temporal feature learning module, and the first viewpoint-specific regression module are identical to those of the second image feature learning module, the second temporal feature learning module, and the second viewpoint-specific regression module, respectively; however, their parameters are not shared. During training:

[0076] a) The first image feature learning module receives the gradient returned by the first time-series feature learning module;

[0077] b) The first-time feature learning module receives the gradients returned by the first-view specific regression module and the dual-view fusion module;

[0078] c) The first-person perspective regression module receives the gradient returned by the loss function;

[0079] d) The second image feature learning module receives the gradient returned by the second temporal feature learning module;

[0080] e) The second temporal feature learning module receives the gradients returned by the second-view specific regression module and the dual-view fusion module;

[0081] f) The second-view specific regression module receives the gradient returned by the loss function;

[0082] g) The dual-view fusion regression module receives the gradient returned by the loss function.

[0083] At the end of each epoch of training, on the validation set Calculate the loss on the data. If the loss on the validation set no longer decreases or reaches the maximum epoch within a specified epoch, then temporarily stop training; and then on the test set... The model is tested on the data. If the test results meet the standards, the entire training process ends. If the test results do not meet the standards, the training process is restarted.

[0084] This invention also employs a special preprocessing design tailored to the characteristics of the dataset. Specifically, compared to the standard sample, this invention performs an additional data amplification on a small number of samples with ejection fraction values ​​greater than 70 and less than 30, thereby mitigating the impact of data imbalance on model prediction to some extent. The ejection fraction regression prediction method provided by this invention is a highly practical new framework that addresses the limitations and lag of existing ejection fraction regression prediction methods, demonstrating broader application potential in clinical settings.

[0085] Specific Implementation Method Two: This implementation method differs from Specific Implementation Method One in that the training set is denoted as... , Represents the first in the training set A pair of echocardiogram data (data within the pair comes from the same person). Indicates the first Apical four-chamber view data from a pair of echocardiographic data. Indicates the first Short-axis view data of the nipple muscle in a pair of echocardiographic data. This represents the total number of echocardiogram data pairs in the training set. Indicates the first True ejection fraction labels for each pair of echocardiographic data (labels are represented as real numbers);

[0086] Verification set Recorded as , Represents the first in the verification set A pair of echocardiogram data (data within the pair comes from the same person). Indicates the first Apical four-chamber view data from a pair of echocardiographic data. Indicates the first Short-axis view data of the nipple muscle in a pair of echocardiographic data. This represents the total number of echocardiogram data pairs in the validation set. Indicates the first vertex in the verification set True ejection fraction labels for each pair of echocardiographic data;

[0087] Test set Recorded as , Indicates the first in the test set A pair of echocardiogram data (data within the pair comes from the same person). Indicates the first Apical four-chamber view data from a pair of echocardiographic data. Indicates the first Short-axis view data of the nipple muscle in a pair of echocardiographic data. This represents the total number of echocardiogram data pairs in the test set. Indicates the first in the test set True ejection fraction labels for each pair of echocardiographic data.

[0088] The other steps and parameters are the same as in Specific Implementation Method 1.

[0089] Specific Implementation Method Three: This implementation method differs from Specific Implementation Method One or Two in that the first image feature learning module includes an image encoder composed of 12 layers of self-attention mechanisms. The specific working process of the first image feature learning module is as follows:

[0090] The first in the training set Taking the processing of apical four-chamber view data in a pair of echocardiographic data as an example:

[0091] For the first training set The apical four-chamber view data of each echocardiogram pair is processed into frames to obtain each image. Then, each obtained image is divided into blocks to obtain several image patches for each image.

[0092] For any image, each image patch is passed through an embedding layer to obtain the feature vector of each image patch. The feature vectors of each image patch are then used to construct a patch-level feature vector sequence for the image. , ;

[0093] The patch-level feature vector sequence of each image is processed by an image encoder to obtain the spatial-level feature vector of each image.

[0094] Other steps and parameters are the same as in specific implementation method one or two.

[0095] Specific Implementation Method Four: This implementation method differs from Specific Implementation Methods One to Three in that the working process of the image encoder within the first image feature learning module is as follows:

[0096] Step S11: Initialize the number of self-attention layers ;

[0097] Step S12, As the first image encoder Layer input:

[0098]

[0099]

[0100] in, For bullish self-attention operations, For layer normalization, It is a multilayer perceptron;

[0101] Step S13: Determine if the condition is met. ;

[0102] If satisfied Then the output of the last layer will be... Global max pooling is performed in the first dimension, and the result is then passed through a multilayer perceptron to obtain the spatial feature vector of the image.

[0103] If not satisfied Then let Return to step S12.

[0104] The other steps and parameters are the same as those in one of the specific implementation methods one to three.

[0105] Specific Implementation Method Five: This implementation method differs from Specific Implementation Methods One to Four in that the second image feature learning module includes an image encoder composed of 12 layers of self-attention mechanisms. The specific working process of the second image feature learning module is as follows:

[0106] The first in the training set Taking the processing of short-axis view data of the papillary muscle in a pair of echocardiographic data as an example:

[0107] For the first training set The short-axis view data of the papillary muscle in each echocardiogram data pair are processed into frames to obtain each image. Then, each obtained image is divided into blocks to obtain several image patches for each image.

[0108] For any image, each image patch is passed through an embedding layer to obtain the feature vector of each image patch. The feature vectors of each image patch are then used to construct a patch-level feature vector sequence for the image. , ;

[0109] The patch-level feature vector sequence of each image is processed by an image encoder to obtain the spatial-level feature vector of each image.

[0110] The other steps and parameters are the same as those in one of the specific implementation methods one to four.

[0111] Specific Implementation Method Six: This implementation method differs from Specific Implementation Methods One to Five in that the working process of the image encoder within the second image feature learning module is as follows:

[0112] Step S21: Initialize the number of self-attention layers ;

[0113] Step S22, will As the first image encoder Layer input:

[0114]

[0115]

[0116] in, For bullish self-attention operations, For layer normalization;

[0117] Step S23: Determine if the condition is met. ;

[0118] If satisfied Then the output of the last layer will be... Global max pooling is performed in the first dimension, and the result is then passed through a multilayer perceptron to obtain the spatial feature vector of the image.

[0119] If not satisfied Then let Return to step S22.

[0120] The other steps and parameters are the same as those in one of the specific implementation methods one to five.

[0121] It should be noted that in the image encoder, the self-attention layer is a multiple self-attention (MSA) layer, and each multi-head self-attention layer is composed of several single-head self-attention (SA) layers in parallel.

[0122] The calculation of single-head self-attention is performed according to the following formula:

[0123]

[0124]

[0125]

[0126]

[0127]

[0128] When calculating the multi-head self-attention mechanism, it is necessary to... or The first dimension, 768, is divided into 12 parts, and then single-head self-attention is calculated for each part separately. Equivalent to or One part, and finally merge the calculation results back into 768 on the first dimension, and use the merged result as the output of each layer of self-attention mechanism.

[0129] Specific Implementation Method Seven: This implementation method differs from Specific Implementation Methods One through Six in that the first time feature learning module is used for time-dimensional feature learning. The first time feature learning module includes a time encoder composed of six layers of self-attention mechanisms. The working process of the first time feature learning module is as follows:

[0130] Step S31: Initialize the number of self-attention layers in the time encoder. ;

[0131] Step S32, to As the first The input of the layer is used to obtain the first layer. Layer output :

[0132]

[0133]

[0134] Step S33: Determine if the condition is met. ;

[0135] If satisfied Then, the output of the last self-attention layer in the temporal encoder is average-pooled along the frame dimension, and the result is used as the first... Spatiotemporal feature vector sequence of apical four-chamber view data in an echocardiographic data pair , ;

[0136] If not satisfied Then let Return to step S32.

[0137] The other steps and parameters are the same as those in one of the specific implementation methods one to six.

[0138] Specific Implementation Method Eight: This implementation method differs from Specific Implementation Methods One to Seven in that the second time feature learning module includes a time encoder composed of six layers of self-attention mechanisms. The working process of the second time feature learning module is as follows:

[0139] Step S41: Initialize the number of self-attention layers in the time encoder. ;

[0140] Step S42, will As the first The input of the layer is used to obtain the first layer. Layer output :

[0141]

[0142]

[0143] Step S43: Determine if the condition is met. ;

[0144] If satisfied Then, the output of the last self-attention layer in the temporal encoder is average-pooled along the frame dimension, and the result is used as the first... Spatiotemporal feature vector sequence of short-axis view data of the papillary muscle in a pair of echocardiographic data , ;

[0145] If not satisfied Then let Return to step S42.

[0146] The other steps and parameters are the same as those in any of the specific implementation methods one to seven.

[0147] It should be noted that the self-attention layers within the temporal encoder are multiple self-attention (MSA) layers, each of which consists of several single-attention (SA) layers in parallel. The calculation of a single-attention layer follows the formula below:

[0148]

[0149]

[0150]

[0151]

[0152]

[0153] Specifically, when calculating the multi-head self-attention mechanism, the first dimension of 512 needs to be divided into 8 parts, and the single-head self-attention calculation is performed separately for each part. It can be or A portion of the process is processed, and the outputs of each single-head self-attention layer are eventually merged back into 512 on the first dimension to obtain the output of each self-attention layer.

[0154] Specific Implementation Method Nine: This implementation method differs from Specific Implementation Methods One through Eight in that the total loss function value during the training process is:

[0155]

[0156] in, This represents the loss between the predicted value and the true label of a specific regression module in the first-person perspective. This represents the loss between the predicted value and the true label of a specific regression module in the second-view perspective. This represents the loss between the predicted values ​​of the dual-view fusion regression module and the true labels;

[0157]

[0158] in, This is the number of training samples in each batch (in this invention, n is 32). It is the first True ejection fraction regression values ​​for each sample pair It is a first-person perspective specific regression module for the first The predicted ejection fraction regression value for each sample pair It is a second-perspective specific regression module for the first The predicted ejection fraction regression value for each sample pair.

[0159] The other steps and parameters are the same as those in one of the specific implementation methods one to eight.

[0160] When calculating the loss function value on the validation and test sets, only the loss between the predicted ejection fraction of the dual-view fusion channel and the true label is calculated.

[0161] Specific Implementation Method 10: This implementation method differs from Specific Implementation Methods 1 to 9 in that the initialization parameters of the first image feature learning module, the second image feature learning module, the first time feature learning module, and the second time feature learning module are obtained through transfer learning. The transferred parameters are obtained by comparative learning using the Kinetics-400 dataset.

[0162] The other steps and parameters are the same as those in any of the specific implementation methods one to nine.

[0163] Furthermore, it should be noted that this invention strikes a balance between the number of model parameters and the size of the dataset, employing a unique training strategy. Specifically, the parameters of the first six self-attention layers of the image encoders in both the first and second image feature learning modules are frozen. Only the parameters of the last six self-attention layers of each image encoder need to be adjusted. By freezing only a portion of the layers in both modules, the model training performance is significantly improved. Specifically, the transfer learning parameters of the image encoders in the first image feature learning module are obtained through pre-training using A4C viewpoint data from the Kinetics-400 dataset, and the transfer learning parameters of the image encoders in the second image feature learning module are obtained through pre-training using PSAX viewpoint data from the Kinetics-400 dataset. This transfer learning approach accelerates model convergence.

[0164] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion, characterized in that, The system comprises an echocardiogram data acquisition module, a first image feature learning module, a first time feature learning module, a first view-specific regression module, a second image feature learning module, a second time feature learning module, a second view-specific regression module, and a dual-view fusion regression module, wherein: The echocardiogram data acquisition module is configured to acquire an echocardiogram dataset composed of echocardiogram data of an apical four-chamber view and echocardiogram data of a papillary muscle short-axis view; The apical four-chamber view data in the echocardiogram dataset is divided into a training set, a validation set, and a test set, and the papillary muscle short-axis view data in the echocardiogram dataset is divided into a training set, a validation set, and a test set, and the divided results are combined to obtain all the echocardiogram data in the training set, all the echocardiogram data in the validation set, and all the echocardiogram data in the test set; During the training process: The first image feature learning module is configured to perform frame division processing on the apical four-chamber view data in the training set, and then extract the features of each image obtained by the frame division processing; Utilizing the first ultrasound echocardiogram data, a space-level feature vector sequence is formed by using feature vectors of all images corresponding to the apical four-chamber view data in the first ultrasound echocardiogram data ; The second image feature learning module is configured to perform frame division processing on the papillary muscle short-axis view data in the training set, and then extract the features of each image obtained by the frame division processing; Utilizing the first ultrasound echocardiogram data, a space-level feature vector sequence is formed by using feature vectors of all images corresponding to short-axis view data of papillary muscle ​ The first time feature learning module is configured to extract a spatial-level feature vector sequence of the apical four-chamber view data in the first pair of echocardiogram data according to the spatial-level feature vector sequence of the apical four-chamber view data in the first pair of echocardiogram data. The first time feature learning module is configured to extract a spatial-level feature vector sequence of the apical four-chamber view data in the first pair of echocardiogram data according to the spatial-level feature vector sequence of the apical four-chamber view data in the first pair of echocardiogram data. The first time feature learning module is configured to extract a spatial-level feature vector sequence of the apical four-chamber view data in the first pair of echoc The second time feature learning module is configured to extract a spatial-level feature vector sequence of the papillary muscle short-axis view data in the first echocardiogram data pair according to the spatial-level feature vector sequence of the papillary muscle short-axis view data in the second echocardiogram data pair. The second time feature learning module is configured to extract a spatial-level feature vector sequence of the papillary muscle short-axis view data in the first echocardiogram data pair according to the spatial-level feature vector sequence of the papillary muscle short-axis view data in the second echocardiogram data pair. The second time feature learning module is configured to extract a spatial-level feature vector sequence of the papillary muscle short-axis view data in the first echocardiogram data pair according to the spatial-level feature The first view-specific regression module is configured to output the ejection fraction value regression prediction result of the apical four-chamber view data according to the spatio-temporal level feature vector sequence output by the first time feature learning module; The working process of the first view-specific regression module is as follows: wherein, is a spatial-temporal level feature vector sequence of the apical four-chamber view data in the i-th echocardiogram data pair, is a spatial-temporal level feature vector sequence of the apical four-chamber view data in the i-th echocardiogram data pair, is a prediction result of the first view-specific regression module, is an activation function, is a batch normalization; The second view-specific regression module is configured to output the ejection fraction value regression prediction result of the papillary muscle short-axis view data according to the spatio-temporal level feature vector sequence output by the second time feature learning module; The working process of the second view-specific regression module is as follows: in, For the first A sequence of spatiotemporal feature vectors of short-axis view data of the papillary muscle from a pair of echocardiographic data. The prediction results for a second-view specific regression module; The dual-view fusion regression module is configured to fuse the spatio-temporal level feature vector sequences output by the first time feature learning module and the second time feature learning module, and obtain the ejection fraction value regression prediction result according to the feature fusion result; The working process of the dual-view fusion regression module is as follows: wherein, is the prediction result of the dual-view fusion regression module, is the multi-head cross attention; The first image feature learning module, the second image feature learning module, the first time feature learning module, the second time feature learning module, the first view-specific regression module, the second view-specific regression module, and the dual-view fusion regression module are used to process each pair of echocardiogram data in the training set, and then the total loss function value is calculated according to the prediction results of the first view-specific regression module, the second view-specific regression module, and the dual-view fusion regression module, and the network parameter values of each module are adjusted according to the total loss function value; The echocardiogram data of the validation set and the test set are used to verify and test each module until the final trained parameters of each module are obtained; During the detection process: For a patient, echocardiogram data of an apical four-chamber view and a papillary muscle short-axis view of the patient are collected respectively, the apical four-chamber view data of the patient is taken as an input of a trained first image feature learning module, the papillary muscle short-axis view data is taken as an input of a trained second image feature learning module, the output of the trained first image feature learning module is taken as an input of a trained first time feature learning module, the output of the trained second image feature learning module is taken as an input of a trained second time feature learning module, and the output of the trained first time feature learning module and the output of the trained second time feature learning module are taken as inputs of a trained dual-view fusion regression module, and a left ventricular ejection fraction prediction result of the patient is output by the trained dual-view fusion regression module.

2. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 1, characterized in that, The training set is denoted as , The i-th echocardiogram data pair in the training set is denoted as , The j-th echocardiogram data pair in the training set is denoted as , The apical four-chamber view data in the i-th echocardiogram data pair is denoted as , The papillary muscle short-axis view data in the i-th echocardiogram data pair is denoted as , The total number of echocardiogram data pairs in the training set is denoted as , The true ejection fraction label of the i-th echocardiogram data pair is denoted as , The true ejection fraction label of the i-th echocardiogram data pair is denoted as , The true ejection fraction label of the i-th echocardiogram data pair is denoted as , The true ejection fraction label of the i-th echocardiogram data pair is denoted as , validation set denoted as , denotes the i-th echocardiogram data pair in the validation set, denotes the apical four-chamber view data in the i-th echocardiogram data pair, denotes the papillary muscle short-axis view data in the i-th echocardiogram data pair, denotes the total number of echocardiogram data pairs in the validation set, denotes the true ejection fraction label of the i-th echocardiogram data pair in the validation set;​​​​ Test set Recorded as , Indicates the first in the test set A pair of echocardiogram data. Indicates the first Apical four-chamber view data from a pair of echocardiographic data. Indicates the first Short-axis view data of the papillary muscle in a pair of echocardiographic data. This represents the total number of echocardiogram data pairs in the test set. Indicates the first in the test set True ejection fraction labels for each pair of echocardiographic data.

3. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 2, characterized in that, The working process of the first image feature learning module is specifically as follows: Taking the processing of the apical four-chamber view data in the first pair of echocardiogram data in the training set as an example: ​ For the first in the training set The apical four-chamber view data of each echocardiogram pair is processed into frames to obtain each image. Then, each obtained image is divided into blocks to obtain several image patches for each image. For any one image, respectively, each image patch of the image is subjected to an embedding layer to obtain a feature vector of each image patch, and a patch-level feature vector sequence of the image is composed of the feature vectors of each image patch ; The patch-level feature vector sequence of each image is input into an image encoder to obtain a spatial-level feature vector of each image.

4. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 3, characterized in that, The working process of the image encoder in the first image feature learning module is specifically as follows: Step S11, initializing the number of self-attention layers ; Step S12, the As a first Input of the layer: wherein, is a multi-head self-attention operation, is layer normalization, is a multi-layer perceptron; Step S13, judging whether or not ; If the following conditions are satisfied , the output of the last layer is Global maximum pooling is performed in the first dimension, and the processed result is passed through a multi-layer perceptron to obtain a spatial-level feature vector of the image. If not satisfied , then let , and return to step S12.

5. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 4, characterized in that, The working process of the second image feature learning module is specifically as follows: The papillary muscle short-axis view data in the first pair of echocardiogram data in the training set is taken as an example for processing: ​ The papillary muscle short-axis view data in the first ultrasound cardiogram data pair in the training set is frame processed to obtain images, and each image is then block processed to obtain a plurality of image patches of each image. For any one image, respectively, each image patch of the image is subjected to an embedding layer to obtain a feature vector of each image patch, and a patch-level feature vector sequence of the image is composed of the feature vectors of each image patch ; The patch-level feature vector sequence of each image is input into an image encoder to obtain a spatial-level feature vector of each image.

6. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 5, characterized in that, The working process of the image encoder in the second image feature learning module is specifically as follows: Step S21, initializing the number of self-attention layers ; Step S22, the As a first Input of the layer: wherein, is a multi-head self-attention operation, is layer normalization; Step S23, judging whether or not ; If the following conditions are satisfied , the output of the last layer is Global maximum pooling is performed in the first dimension, and the processed result is passed through a multi-layer perceptron to obtain a spatial-level feature vector of the image. If not satisfied , then let , and return to step S22.

7. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 6, characterized in that, The working process of the first time feature learning module is as follows: Step S31, initialize the number of self-attention layers of the time encoder ; Step S32, obtaining the output of the first layer As a first layer, obtaining the output of the first layer layer : Step S33, judging whether or not ; If the following conditions are met , the output of the last self-attention layer in the time encoder is averaged-pooled in the frame dimension, and the processing result is taken as the spatial-temporal level feature vector sequence of the apical four-chamber view data in the th echocardiogram data pair ; If not satisfied , then let , return to step S32.

8. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion of claim 6, wherein, The working process of the second time feature learning module is as follows: Step S41, initialize the number of self-attention layers of the time encoder ; Step S42, obtaining the output of the first layer As the first layer input, the output of the first layer is obtained layer output : Step S43, judging whether or not ; If the following conditions are met , the output of the last self-attention layer in the time encoder is averaged-pooled in the frame dimension, and the processing result is taken as the time-space level feature vector sequence of the papillary muscle short-axis view data in the i-th echocardiogram data pair ;​ If not satisfied , then let , and return to step S42.

9. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 7 or 8, characterized in that, The total loss function value in the training process is as follows: wherein, denotes a loss between the predicted value of the first view-specific regression module and the real label, denotes a loss between the predicted value of the second view-specific regression module and the real label, denotes a loss between the predicted value of the dual-view fusion regression module and the real label; wherein, is the number of training samples per Batch, is the true ejection fraction regression value of the th sample pair, is the predicted ejection fraction regression value of the th sample pair by the first view-specific regression module, is the predicted ejection fraction regression value of the th sample pair by the second view-specific regression module.

10. The left ventricular ejection fraction regression prediction system based on transfer learning and multi-view fusion according to claim 9, characterized in that, The initialization parameters of the first image feature learning module, the second image feature learning module, the first time feature learning module and the second time feature learning module are obtained through transfer learning, and the transferred parameters are obtained through contrast learning using a Kinetics-400 dataset.

Images