Medical time series data dynamic modeling and analysis method based on Shenchang differential equation

Abstract

The invention relates to a medical image intelligent analysis technology, and aims to provide a medical time series data dynamic modeling and analysis method based on a Shenchang differential equation. Comprising the following steps: acquiring medical time sequence data with known time resolution as input based on an echocardiogram video; performing time sequence initialization processing on the video data with different time resolutions, so that initial time steps of output time sequence data are aligned, and the time resolution is higher than that of input time sequence data; the method comprises the following steps: constructing an echocardiogram video data analysis network model based on an ordinary differential equation, initializing network parameters, and setting an encoder-decoder structure; and training, verifying and testing to obtain the echocardiogram video data analysis network model of the Echo-ODE structure. According to the method, high-precision frame-by-frame full-structure segmentation of the echocardiogram is realized, the potential ordinary differential equation is formed in combination with the characteristics of the Sheng ordinary differential equation and the convolutional neural network, the time dynamic of irregular time steps is successfully simulated, and space-time information analysis of the echocardiogram is facilitated.

Description

Technical Field

[0001] The present invention relates to a medical image intelligent analysis technology, and in particular to a dynamic modeling method of medical time series data based on Neural Ordinary Differential Equations, as well as a system and a computer-readable storage medium for implementing the method. Background Art

[0002] In the field of medical diagnosis, accurate assessment of the structure and function of the heart is of irreplaceable importance for the formulation of effective clinical treatment plans. As a non-invasive, real-time imaging technology, echocardiography has become one of the main means of evaluating cardiac abnormalities, and is particularly good at quantitative analysis of ventricular function and size, especially the evaluation of the left ventricle (LV). Among them, the left ventricular ejection fraction (LVEF) is a key indicator for measuring left ventricular systolic function, and its accurate quantification is crucial for disease diagnosis, treatment monitoring and prognosis evaluation. However, in traditional processing schemes, ultrasound physicians need to select the end-diastolic (ED) and end-systolic (ES) frames in the cardiac cycle through visual judgment, manually outline the ventricular boundaries and calculate the LVEF. This process is not only time-consuming and labor-intensive, but also easily affected by subjective factors, resulting in limited repeatability and accuracy of the results.

[0003] In order to overcome the above challenges, although some researchers have tried to improve the evaluation efficiency and accuracy through automatic segmentation technology of video echocardiography, they are faced with problems such as phase detection errors caused by time inconsistency and unreliable LVEF calculation. Early deep learning models focused on the processing of static images (such as ED frames and ES frames), ignoring the temporal dependency between frames in video sequences. Subsequently, although some researchers have tried to use models to integrate spatiotemporal information, they still mainly focus on single-frame prediction and fail to achieve continuous and consistent analysis of the entire video sequence.

[0004] Therefore, it is necessary to propose new solutions to solve the problem of difficulty in handling irregular time input and to achieve accurate analysis of the temporal and spatial information of echocardiography. Summary of the invention

[0005] The technical problem to be solved by the present invention is to overcome the deficiencies in the prior art and provide a dynamic modeling method for medical time series data based on neural ordinary differential equations.

[0006] To solve the technical problem, the solution of the present invention is:

[0007] A method for dynamic modeling of medical time series data based on neural ordinary differential equations is provided, comprising the following steps:

[0008] (1) Based on a sufficient amount of echocardiographic videos, medical time series data with known time resolution are obtained as input; video data with different time resolutions are subjected to time series initialization processing so that the initial time steps of the output time series data are aligned and the time resolution is higher than that of the input time series data;

[0009] (2) Constructing an echocardiography video data analysis network model based on the neural ordinary differential equation and initializing the network parameters; setting an encoder-decoder structure in the network model, wherein:

[0010] (2.1) Construct an encoder using convolutional encoder and Latent ODE blocks; use the convolutional encoder to learn the spatial features of the input temporal data, embed the input video into a consistent hidden state, and use Latent ODE in the latent space to capture the temporal dependency of the video;

[0011] (2.2) A multi-target decoder architecture is constructed using NODE, a reconstruction decoder, and a target decoder. The reconstruction decoder and the target decoder are connected by a jump connection. After acquiring the overall dynamic information of the echocardiogram, it is fused into the neural derivative function so that the network structure can generate the required prediction at any time node.

[0012] (3) Using the processed time series data in step (1), the network model described in step (2) is trained, verified and tested, and finally an echocardiography video data analysis network model with an Echo-ODE structure is obtained; the network model can analyze the input echocardiography video and automatically complete the segmentation, labeling and annotation of the cardiac structure in the video image.

[0013] As a preferred solution of the present invention, the echocardiographic videos in step (1) all include ED frames or ES frames marked with cardiac structure segmentation marks and information annotations.

[0014] As a preferred solution of the present invention, in step (1), the time series initialization processing refers to: converting the acquired medical image data into a matrix set, preprocessing it to obtain standardized medical image matrix data, and storing the feature matrix in time steps according to the time resolution.

[0015] As a preferred solution of the present invention, in step (1), the acquired medical time series data is divided into a training set, a validation set and a test set, and the ratio of the three is 8:1:1; the training set, the validation set and the test set are respectively used for the training, validation and testing operations of the network model in step (3).

[0016] As a preferred solution of the present invention, the step (2.1) specifically includes:

[0017] A parameter-sharing convolutional encoder is used to learn the spatial features of video images, and the input video sequence is embedded into a low-dimensional latent space. Then, Latent ODE is used in the latent space to learn the temporal dependencies between different video frames. The outputs of Latent ODE are the overall dynamic information of the video and the initial state of the NODE in the decoder.

[0018] As a preferred solution of the present invention, the step (2.2) specifically includes:

[0019] Firstly, the decoder NODE takes the initial state in the encoder output as the Cauchy initial condition, and fits the derivative function with a parameterized neural network. The derivative function takes the intermediate state at any time and the overall dynamic information of the encoder output as input, and the output is the value of the derivative function at the current time. After defining the initial condition and the derivative function, any known differential equation solver is used to solve the latent space state at any time after the current state. Finally, the reconstruction decoder and target decoder with parameter sharing are used to decode the state variables of any time node in the latent space to obtain the results of video reconstruction and target prediction.

[0020] As a preferred solution of the present invention, a loss function is set for the network model; wherein the reconstruction loss Loss R econstruction uses mean square error loss, segmentation task Loss S Egation uses cross entropy loss and uses the hyperparameter λ to make it proportional to the total loss Loss Multitask :

[0021] LosSMuttitask=LosSReconstruction+λLosSsegmentation

[0022]

[0023] Among them, M and They are the actual segmentation labels and the predicted segmentation labels respectively.

[0024] The present invention further provides a method for analyzing echocardiographic video data using the network model obtained by the aforementioned method, comprising the following steps:

[0025] (1) Standardizing the video image data to be analyzed, including: further unifying the size of the video image through resizing and cropping operations; calculating the mean and variance of the grayscale of the images in the training set and performing normalization processing.

[0026] (2) inputting the processed data into the echocardiography video data analysis network model, and outputting the result after calculation as video image data containing cardiac structure segmentation, labeling and annotation information;

[0027] (3) The original video images are sampled and the cardiac structures are manually segmented, labeled, and annotated. The manual analysis results are then compared with the results output by the network model in terms of segmentation effect, regional similarity, and consistency of prediction time.

[0028] The present invention also provides a system for analyzing echocardiogram video data, the system comprising an echocardiogram acquisition module, a segmentation inference module, an efficiency evaluation module, and a drawing module; wherein:

[0029] An echocardiogram acquisition module is used to acquire a sequence of two-dimensional matrix sets to be processed of echocardiogram video data, and preprocess the matrix sets to be processed to obtain regions of interest and category labels corresponding to the regions of interest;

[0030] A segmentation inference module is used to input the acquired two-dimensional matrix set sequence into a trained neural ordinary differential equation network to obtain an inferred frame-by-frame segmentation result;

[0031] The performance evaluation module is used to calculate the consistency measure between the inferred segmentation results and the true segmentation;

[0032] The drawing module is used to draw the image based on the inferred segmentation results and the original image.

[0033] The present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, it is used to implement the method for analyzing echocardiographic video data as described above.

[0034] Description of the invention principle:

[0035] Traditional neural networks are essentially complex composite functions, and the number of composite functions is the depth of the network. For example, a two-layer fully connected network can be expressed as Y = f(f(X,θ1),θ2). Therefore, each neural network layer can be regarded as a universal function approximator. Since the overall network is composed of composite functions, it is intuitive to use the chain rule for derivation. The gradient is gradually passed from the outermost layer to the inner layer, and the parameters θ of each layer are updated at each layer. However, after forward propagation, the activation values ​​of all layers need to be saved, and then these activation values ​​are used to calculate the gradient during back propagation, which takes up a lot of memory and limits the training of deep models.

[0036] Compared with traditional neural networks, Neural Ordinary Differential Equation (NODE) ​​chooses a different approach. The latter is a continuous form of ResNet and RNN, which simulates system dynamics by solving ordinary differential equations and parameterizes the derivatives of hidden states through neural networks instead of directly parameterizing the hidden states themselves. The derivative parameterization here constructs a continuous hierarchical structure instead of the traditional discrete hierarchy. Therefore, the parameter space also becomes continuous, and there is no need to pass gradients and update parameters in layers. In general, Neural Ordinary Differential Equation does not store any intermediate results in forward propagation, and only requires constant memory costs. NODE reduces the number of parameters, reduces the demand for computing resources, and can process non-uniformly spaced time series data. Backpropagation uses the adjoint sensitivity method to avoid the large amount of storage of forward propagation states. NODE is widely used in sequence interpolation, prediction, etc., and is controllable in terms of accuracy and efficiency.

[0037] As a complex periodic dynamic system, the dynamic behavior of the heart can be precisely controlled by the cardiac conduction system. Based on this feature, a new idea is provided for the modeling work of analyzing echocardiography videos. From a mathematical perspective, the dynamic system can be described by differential equations. Based on this theoretical framework, the applicant's research team innovatively proposed dynamic modeling of echocardiography based on neural ordinary differential equations to realize the analysis of echocardiography video data and the modeling of physical dynamic systems. By using neural ordinary differential equations (NODEs) to simulate the dynamic changes of the heart in echocardiography videos, not only does it give the model stronger physical interpretability, but it also breaks the limitation of traditional recurrent neural networks (RNNs) on the fixed time length of the input sequence, and realizes the dense and continuous output in the time dimension. This feature enables the model to flexibly deal with non-uniform sampling problems that may be encountered in practical applications, and improves computational efficiency and accuracy through internal prediction and dense output mechanisms.

[0038] This invention innovatively proposes a dynamic modeling method for echocardiography based on neural ordinary differential equations (which can be called the Echo-ODE framework), introduces NODE into echocardiography video analysis for the first time, and uses the continuous output characteristics of NODE for the first time in the industry to open up a new way for echocardiography video processing, making it possible to achieve advanced tasks such as video extrapolation and video interpolation, greatly expanding the application potential of echocardiography in clinical diagnosis and treatment planning. Through the modeling of physical dynamic systems, continuous and accurate evaluation of cardiac structure and function can be achieved, providing more reliable data support for clinical decision-making.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1. The present invention proposes a dynamic modeling method for medical time series data based on neural ordinary differential equations, which realizes high-precision frame-by-frame full-structure segmentation of echocardiograms.

[0041] 2. The present invention combines the characteristics of neural ordinary differential equations and convolutional neural networks to form potential ordinary differential equations; it can not only capture time information from the input sequence, but also capture spatial information; the present invention realizes the dynamic modeling of medical time series data using neural ordinary differential methods for the first time.

[0042] 3. The present invention solves the problem that the prior art is difficult to handle irregular time input, and can successfully simulate the temporal dynamics of irregular time steps, which is helpful for the temporal and spatial information analysis of echocardiography. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention is a flowchart of a method for analyzing echocardiography video data and modeling a physical dynamic system using a neural ordinary differential equation in Embodiment 1 of the present invention.

[0044] Figure 2 This is the Echo-ODE structure in the first embodiment of the present invention.

[0045] Figure 3 This is an example of heart segmentation of Echo-ODE in Example 1 of the present invention. The first line is the input video frame, the second line is the reconstruction result of the Echo-ODE reconstruction branch, and the third line is the segmentation result of Echo-ODE. DETAILED DESCRIPTION

[0046] The advantages of the present invention are further described below in conjunction with the accompanying drawings and specific embodiments.

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] The terms used in this application are only for the purpose of describing specific embodiments, and are not intended to limit the application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. In the subsequent description, the use of suffixes such as "module", "component" or "unit" used to represent elements is only for the purpose of facilitating the description of the present invention, and itself has no specific meaning. Therefore, "module" and "component" can be used in a mixed manner.

[0049] like Figure 1 As shown, an embodiment of the present invention provides a method for dynamic modeling of medical time series data based on neural ordinary differential equations, the method comprising the following process:

[0050] S100, based on a sufficient amount of echocardiography video, obtain medical time series data with known time resolution as input; perform time series initialization processing on video data with different time resolutions, so that the initial time steps of the output time series data are aligned, and the time resolution is higher than the input time series data. Specifically, the following steps are included:

[0051] S110, obtaining a set of to-be-processed matrices of image data of a medical image, and preprocessing the set of to-be-processed matrices to obtain standardized medical image matrix data, and storing feature matrices in time steps according to a time resolution;

[0052] Specifically: define a is a medical time series data of length n, and a particularly preferred embodiment of the present invention is an echocardiogram, wherein is a 2D image with c channels and size h×w. Define the time step as τ∶={t1,t2,…,t n}, where 0 <t1<t2<…<t n Given an input video X τ and a task T, the goal is to generate information about another set of time steps S∶={s1,s2,…,s m}Corresponding Define the initial time step to be aligned to 0, that is: t0=s0=0.

[0053] Exemplarily, the dataset resource comes from Echonet-LVH, which includes 12,000 labeled echocardiography videos and expert annotations (measurements and calculations) to provide a baseline for studying ventricular size and wall thickness. The dataset lacks segmentation labels for cardiac structures. Since the quality of some samples in Echo-LVH is not very good, the selection of videos is based on video quality, mainly considering the size and clarity of the image, the video recording method, and whether it is a parasternal long axis video. Finally, 2067 videos were selected. Then, each video was re-labeled into one or two frames, which are ED frames or ES frames. The new annotation adds segmentation labels for cardiac structures such as the left ventricle, left atrium, right ventricle, left ventricular posterior wall, ventricular septum, and right ventricular anterior wall. Labels are not necessarily present on every annotated frame because some of these structures are not completely visible in the image. All work was properly labeled and annotated by multiple ultrasound experts from Zhejiang Provincial People's Hospital (Affiliated People's Hospital) according to the generally recognized operating specifications in the industry. Each ultrasound expert has more than 5 years of practical experience in ultrasound image annotation. In this example, a total of 2067 parasternal long-axis videos were acquired, with a total of 4134 annotated frames.

[0054] The acquired data is divided into training set and test set.

[0055] Preferably, the data set is divided into a training subset, a validation set, and a test set in a ratio of 8:1:1. Exemplarily, 3308 annotated frames of 1654 ultrasound videos are included in the training set, 206 videos are used as the validation set, and 207 videos are used as the test set.

[0056] S200. Construct an echocardiography video data analysis (Echo-ODE) network model based on neural ordinary differential equations and initialize network parameters.

[0057] The Echo-ODE network model provided by the present invention adopts an encoder-decoder structure as a whole.

[0058] Among them, the Latent ODE network is used to model the temporal dependency of the video in the embedded feature space of the network input video, and the overall dynamic information of the video and the feature encoding of the initial state of the video are learned. Then, these two pieces of information are used to define the initial value and derivative function of the ordinary differential equation followed by the dynamics of the echocardiogram video, and the NODE network is constructed with this differential equation to generate the embedded features of any time node in the output space. Finally, the embedded features are passed through the convolution decoder to generate the target prediction.

[0059] Latent ODE was proposed in "Latent ODEs for Irregularly-Sampled Time Series". It has the advantages of both RNN model and NODE model. It can not only receive information input at non-equidistant time nodes, but also realize dynamic modeling of continuous time. Therefore, it can well learn the spatiotemporal feature encoding of video input.

[0060] The specific construction process includes the following steps:

[0061] S210. Construct an encoder using a convolutional encoder and a Latent ODE block; use the convolutional encoder to learn the spatial features of the input time series data, embed the input video into a consistent hidden state, and use LatentODE in the latent space to capture the temporal dependency of the video.

[0062] Specifically, it includes: using a parameter-sharing convolutional encoder to learn the spatial features of video images, embedding the input video sequence into a low-dimensional latent space, and then using Latent ODE in the latent space to learn the temporal dependencies between different video frames; the output of Latent ODE is the overall dynamic information of the video and the initial state of the NODE in the decoder.

[0063] First, given an input video sequence where t i For different time nodes of different frames, the parameter-sharing convolutional encoder network E is used to learn the spatial features of all video frames. This process embeds the video input into the latent space represented as in

[0064] Next, Latent ODE is used in the latent space to capture the temporal dependencies of the video and parse the overall dynamic information D of the echocardiogram. Then, a convolutional layer is used to learn the initial state of the decoder from D. To get a better initial state, The input to the Latent ODE is reversed in time, i.e. from the last frame to the first frame of the input video segment.

[0065] Specifically, Latent ODE can be expressed as:

[0066]

[0067] That is, Latent ODE learns the data flow h of the video in the latent space t , at any t i time First, we solve a differential equation at t i-1 The value of the moment Afterwards, i-1 Embedded feature input at time It is updated through the ConvGRU module as Among them, ODESolver() refers to the differential equation solver, which uses the derivative function of the differential equation The derivative function is fitted by a convolutional network f; ConvGRUCell() refers to the convolutional GRU module. Initial hidden state is set to 0 and ∈ is set to 0.01.

[0068] The final output of the Latent ODE is used as the overall dynamic information D of the video, followed by a convolutional layer to generate the initial state of the output embedding space

[0069] For example, The video clip input size is 4×192×256, where 4 is the number of video frames, 192 and 256 are the height and width of each frame. The size is 4×12×16, and Latent ODE takes it as input and outputs D and The size is 12×16.

[0070] S220, using NODE to connect the reconstruction decoder and the target decoder to construct a decoder architecture, and the reconstruction decoder and the target decoder are connected in a jump connection manner; since the NODE network can generate the state of the output embedding space at any time, the decoder architecture can generate the required prediction at any time node.

[0071] Firstly, the decoder NODE takes the initial state in the encoder output as the Cauchy initial condition, and fits the derivative function with a parameterized neural network. The derivative function takes the intermediate state at any time and the overall dynamic information of the encoder output as input, and the output is the value of the derivative function at the current time. After defining the initial condition and the derivative function, any known differential equation solver is used to solve the latent space state at any time after the current state. Finally, the reconstruction decoder and target decoder with parameter sharing are used to decode the state variables of any time node in the latent space to obtain the results of video reconstruction and target prediction.

[0072] S221. Construct a target decoder.

[0073] The NODE in the decoder is based on the initial state And the neural derivative function f θ (D,θ) at the target time node {s1,s2,…,s m} to generate the hidden state representation of the output embedding space. The newly generated hidden state at any time node can be represented as:

[0074]

[0075] Theoretically, the derivative function f of the differential equation θ It should be unique for each video clip. In order to meet the personalized dynamic requirements, the learned cardiac dynamic information of each clip is integrated into the derivative function, such as Figure 2 As shown in the lower left corner. For any time t, the derivative function will hide the state h t The result of the calculation of the three convolutional layers is the value of the derivative function at the current moment. The solution of this process is still realized by the differential equation solver. Since most of the existing differential equation solvers have dense output mode and this process does not increase the amount of calculation, it is possible to generate hidden states of any dense time nodes within a period of time.

[0076] In this example, the target decoder is a sequence of convolutional layers, whose output is a 7-channel probabilities corresponding to the background and other 6 intracardiac structures, and the argmax function is applied to obtain the final segmentation. In practical applications, since the model in the present invention is not limited to video segmentation, the target decoder can be changed according to different tasks.

[0077] Exemplarily, the D and The size is 12×16. Assume that the final prediction time node required is S∶={s1,s2,…,s m}, S can be adjusted according to different tasks. The output size after the NODE network is m×12×16, and the target decoder performs upsampling and other operators to restore the hidden state to the output prediction of the original image size, with a dimension of m×192×256.

[0078] S222: Construct a reconstruction decoder.

[0079] Theoretically, the target decoder can already complete the task of the present invention. However, in practical applications, echocardiography videos usually have only one or two frames of annotations, so if the network model only relies on sparsely annotated task labels, it is difficult to fully understand the dynamics of the heart. In order to solve this problem, the present invention further adds a reconstruction module to obtain the essential dynamic feature representation of the video through self-supervised learning. Since the output time step does not have to be the same as the input time step, if the reconstructed output time node happens to be the same as a certain time node of the input time, the module block can be understood as a reconstruction of the input image, otherwise it can be interpreted as interpolating the video. Through the tasks of video reconstruction and video interpolation, the network model can have a better grasp of the dynamics of the heart and learn the necessary dynamic representations into the model parameters of NODE.

[0080] S223: Use a hopping connection method to connect the reconstructed decoder and the target decoder.

[0081] Since video interpolation is implemented in the process of learning video dynamic representation, the upsampling process in the original video reconstruction module can provide high-resolution image information. Therefore, by adding a jump connection between the reconstruction decoder and the target decoder, the prediction accuracy of the target task can be improved.

[0082] S230, setting the loss function and initializing the network parameters;

[0083] Specifically, the reconstruction loss uses the mean square error loss, the segmentation task uses the cross entropy loss, and the hyperparameter λ is used to make it proportional to the total loss.

[0084] LosSMultitask=LosSReconstruction+λLosSsegmentation

[0085] in:

[0086]

[0087] Where M and They are the actual segmentation labels and the predicted segmentation labels respectively.

[0088] Initialize network parameters randomly.

[0089] S300. Using the time series data processed in step S100, the network model described in step S200 is trained, verified and tested, and finally an echocardiogram video data analysis network model with an Echo-ODE structure is obtained; the network model can analyze the input echocardiogram video and automatically complete the segmentation, labeling and annotation of the cardiac structure in the video image.

[0090] The model design and training in this example were done in Python using the PyTorch deep learning library on two NVIDIA Tesla P40 GPUs. A stochastic gradient descent (SGD) optimizer with a learning rate of 0.05 was used. The number of iterations was set to 200 and the batch size was 16. Data augmentation used random resizing, random cropping, random horizontal flipping, color jittering, and random Gaussian blurring. For the hyperparameters, λ was set to 2.0. As an example, Dopri5 can be selected as the ordinary differential equation solver of the present invention.

[0091] To demonstrate the effectiveness of the proposed network model, the performance of Echo-ODE was compared with that of the classic image-level model UNet and the video-level model ConvLSTM based on the classic RNN with the same loss function. To compare the ability of LSTM cells and NODE to model the dynamics of echocardiography, LSTM blocks were used to replace NODE in Echo-ODE to capture the temporal relationship of hidden states, and the method was simplified to Echo-LSTM. UNet was trained at the image level and applied to each frame during video prediction. The input of ConvLSTM was the same as that of Echo-LSTM and Echo-ODE, with 4 images sampled every 2 frames around the annotated image. During the training phase, each marked frame was randomly sorted to one of the 4 frames. During the validation and testing phases, the marked box was fixed to the last frame to ensure the repeatability of the experiment. When it was necessary to output the prediction of a complete video, Echo-ODE obtained all 7 intermediate predictions of the segment through dense output, and performed 2 ConvLSTMs, each shifting the input by one frame to obtain all the predictions.

[0092] S400, performing echocardiography video data analysis based on the network model constructed by the aforementioned method.

[0093] The specific steps include:

[0094] S410, standardizing the video image data to be analyzed; specifically, the process includes: further unifying the size of the video image by resizing and cropping; calculating the mean and variance of the grayscale of the image in the training set, and performing normalization processing;

[0095] As an example, video clips before and after the manually annotated video frames can be selected as input to the network;

[0096] S420, inputting the processed data into the echocardiography video data analysis network model, and outputting the result after calculation as video image data containing cardiac structure segmentation, labeling and annotation information;

[0097] S430, sampling the original video image, manually segmenting, marking and annotating the cardiac structure; then comparing the segmentation effect, regional similarity and prediction time consistency of the manual analysis result with the result output by the network model.

[0098] Typically, the intersection over union (IoU) and dice coefficient (dice) are used to measure the segmentation effect and the regional similarity between the predicted mask and the label mask.

[0099] Because the evolution of object shape is an important clue to downstream tasks, the temporal consistency of prediction is also an important aspect of video segmentation. Therefore, the present invention uses two metrics to measure temporal consistency (TC). The specific function is as follows.

[0100]

[0101] Where N is the number of videos; T is the number of frames in each video; is the predicted mask of the tth frame in the kth video; τ(.) is the cost function of the mask matching based on the shape context description.

[0102] For scenarios where annotations for the entire video are available, we propose a more intuitive metric based on the annotations. Assuming that the labels are consistent, the predictions are also consistent and well aligned with the labels, then the Dice coefficients of any adjacent segmentations that are consistently predicted should be very close. We define the temporal consistency based on the Dice coefficient as:

[0103]

[0104] in, is the dice similarity between the predicted mask and the annotated mask in the tth frame of the kth video. In theory, if the predictions are consistent in time, TC D Should be close to zero.

[0105] Echo-ODE is on par with the other methods in terms of Dice coefficient (0.834 vs. UNet 0.832, ConvLSTM 0.831, Echo-LSTM 0.824), and has the best temporal consistency for all six cardiac structures, with TC values ​​much lower than the other three methods (left atrium: 0.330 vs. UNet 0.334, ConvLSTM 0.353, Echo-LSTM 0.372; ventricular septum: 0.311 vs. UNet 0.337, ConvLSTM 0.360, Echo-LSTM 0.332; left ventricular posterior wall: 0.226 vs. UNet 0.337, ConvLSTM 0.360, Echo-LSTM 0.332; left ventricle: 0.184 vs. UNet 0.198, ConvLSTM 0.215, Echo-LSTM 0.210; right ventricle: 0.261 vs. UNet 0.341, ConvLSTM 0.382, Echo-LSTM 0.288; right ventricular anterior wall: 0.387 vs. UNet 0.631, ConvLSTM 0.551, Echo-LSTM 0.604). Echo-ODE performs better in terms of temporal consistency while keeping the segmentation accuracy unchanged.

[0106] In a second aspect, this embodiment further provides a system for analyzing echocardiogram video data based on Neural Ordinary Differential Equations, the system comprising an echocardiogram acquisition module, a segmentation inference module, a performance evaluation module, and a drawing module; wherein,

[0107] An echocardiogram acquisition module is used to acquire a sequence of two-dimensional matrix sets to be processed of echocardiogram video data, and preprocess the matrix sets to be processed to obtain regions of interest and category labels corresponding to the regions of interest;

[0108] A segmentation inference module is used to input the acquired two-dimensional matrix set sequence into a trained neural ordinary differential equation network to obtain an inferred frame-by-frame segmentation result;

[0109] The performance evaluation module is used to calculate the consistency measure between the inferred segmentation results and the true segmentation;

[0110] The drawing module is used to draw the image based on the inferred segmentation results and the original image.

[0111] In a third aspect, the present invention further discloses a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, it is used to implement the method for analyzing echocardiographic video data as described above.

[0112] It should be noted that the embodiments of the present invention have better practicability and do not impose any form of limitation on the present invention. Any technician familiar with the field may use the technical content disclosed above to change or modify it into an equivalent effective embodiment. However, any modification or equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention are still within the scope of the technical solution of the present invention.

Claims

1. A dynamic modeling method for medical time series data based on Neural Ordinary Differential Equations, characterized in that: The following steps are involved: (1) Based on a sufficient amount of echocardiographic videos, obtain medical time series data with known time resolution as input; Performing time series initialization processing on video data with different time resolutions so that the initial time steps of the output time series data are aligned and the time resolution is higher than that of the input time series data; (2) Constructing an echocardiography video data analysis network model based on the neural ordinary differential equation and initializing the network parameters; setting an encoder-decoder structure in the network model, wherein: (2.1) Construct an encoder using convolutional encoder and Latent ODE blocks; use the convolutional encoder to learn the spatial features of the input temporal data, embed the input video into a consistent hidden state, and use Latent ODE in the latent space to capture the temporal dependency of the video; (2.2) A multi-target decoder architecture is constructed using NODE, a reconstruction decoder, and a target decoder. The reconstruction decoder and the target decoder are connected by a jump connection. After acquiring the overall dynamic information of the echocardiogram, it is fused into the neural derivative function so that the network structure can generate the required prediction at any time node. (3) Using the processed time series data in step (1), the network model described in step (2) is trained, verified and tested, and finally an echocardiography video data analysis network model with an Echo-ODE structure is obtained; the network model can analyze the input echocardiography video and automatically complete the segmentation, labeling and annotation of the cardiac structure in the video image.

2. The method according to claim 1, characterized in that The echocardiographic videos in step (1) all include ED frames or ES frames marked with cardiac structure segmentation marks and information annotations.

3. The method according to claim 1, characterized in that In the step (1), the time series initialization processing refers to: converting the acquired medical image data into a matrix set, preprocessing it to obtain standardized medical image matrix data, and storing the feature matrix in time steps according to the time resolution.

4. The method according to claim 1, characterized in that: In the step (1), the acquired medical time series data is divided into a training set, a validation set and a test set, and the ratio of the three is 8:1:1; the training set, the validation set and the test set are respectively used for the training, validation and testing operations of the network model in the step (3).

5. The method according to claim 1, characterized in that The step (2.1) specifically includes: A parameter-sharing convolutional encoder is used to learn the spatial features of video images, and the input video sequence is embedded into a low-dimensional latent space. Then, Latent ODE is used in the latent space to learn the temporal dependencies between different video frames. The outputs of LatentODE are the overall dynamic information of the video and the initial state of NODE in the decoder.

6. The method according to claim 1, characterized in that The step (2.2) specifically includes: Firstly, the decoder NODE takes the initial state in the encoder output as the Cauchy initial condition, and fits the derivative function with a parameterized neural network. The derivative function takes the intermediate state at any time and the overall dynamic information of the encoder output as input, and the output is the value of the derivative function at the current time. After defining the initial condition and the derivative function, any known differential equation solver is used to solve the latent space state at any time after the current state. Finally, the reconstruction decoder and target decoder with parameter sharing are used to decode the state variables of any time node in the latent space to obtain the results of video reconstruction and target prediction.

7. The method according to claim 1, characterized in that A loss function is set for the network model; wherein the reconstruction loss Loss Reconstruction Use mean square error loss, segmentation task Loss Segmentation Use cross entropy loss and use the hyperparameter λ to make it proportional to the total loss Loss Multitask : Loss Multitask =Loss Reconstruction +λLoss Segmentation Among them, M and They are the actual segmentation labels and the predicted segmentation labels respectively.

8. A method for analyzing echocardiographic video data using the network model obtained by the method of claim 1, characterized in that: The following steps are involved: (1) Standardizing the video image data to be analyzed, specifically including: further unifying the size of the video image through resizing and cropping operations; calculating the mean and variance of the grayscale of the images in the training set and performing normalization processing; (2) inputting the processed data into the echocardiography video data analysis network model, and outputting the result after calculation as video image data containing cardiac structure segmentation, labeling and annotation information; (3) The original video images are sampled and the cardiac structures are manually segmented, labeled, and annotated. The manual analysis results are then compared with the results output by the network model in terms of segmentation effect, regional similarity, and consistency of prediction time.

9. A system for analyzing echocardiographic video data, characterized in that: The system includes an echocardiogram acquisition module, a segmentation inference module, a performance evaluation module, and a drawing module; wherein, An echocardiogram acquisition module is used to acquire a sequence of two-dimensional matrix sets to be processed of echocardiogram video data, and preprocess the matrix sets to be processed to obtain regions of interest and category labels corresponding to the regions of interest; A segmentation inference module is used to input the acquired two-dimensional matrix set sequence into a trained neural ordinary differential equation network to obtain an inferred frame-by-frame segmentation result; The performance evaluation module is used to calculate the consistency measure between the inferred segmentation results and the true segmentation; The drawing module is used to draw the image based on the inferred segmentation results and the original image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is used to implement the method for analyzing echocardiographic video data as claimed in claim 8 when executed by a processor.

Images