Zebrafish cardiovascular function intelligent identification and evaluation method

Abstract

The application discloses a zebrafish cardiovascular function intelligent identification and evaluation method, which comprises the following steps: taking an image of a zebrafish under a microscope, inputting the preprocessed image into a trained zebrafish identification and classification model, filtering out zebrafish images that are not conducive to observation, obtaining a segmentation result of cells or tissues through a zebrafish cell and tissue segmentation model, obtaining information data of blood cell flow rate, heart rate and ejection fraction through a cell tracking algorithm and waveform analysis, and visualizing the cardiovascular function. The method can improve the efficiency of zebrafish experimental data analysis and processing, automatically screen appropriate image data from a large amount of images obtained through experiments, analyze and obtain information data of the cardiovascular function with research value, reduce manual operation, has a simple processing procedure, needs relatively few parameters to be adjusted, and provides strong technical support for zebrafish related research and development of complex cardiovascular disease drugs.

Description

Technical Field

[0001] This invention relates to the field of image processing, specifically to a method for intelligent recognition and evaluation of cardiovascular function in zebrafish. Background Technology

[0002] Zebrafish, as a small model organism, is widely used in biomedical research experiments due to its small size, genetic composition similar to humans, optically transparent juveniles for easy observation, and rapid growth and reproduction. Advances in microscopy have enabled researchers to obtain a large number of zebrafish microscopic images. By analyzing these images, information on heart rate, ejection fraction, and blood cell flow velocity can be obtained to assess cardiovascular function, thereby evaluating drug efficacy, conducting genetic analysis, or researching cardiovascular diseases.

[0003] However, during the recording of live zebrafish images, issues such as distorted zebrafish poses, unclear observation of specific organs and tissues, and obstruction by debris often arise, making manual sorting and selection of easily analyzed images extremely tedious and time-consuming. Calculating blood flow velocity by manually tracking multiple cells across consecutive frames is inefficient. Existing multi-target automatic tracking methods mostly consist of three steps: acquiring detection results, extracting appearance or motion features of the detected targets, and matching the detected targets between consecutive frames based on these features. The detection quality significantly impacts the tracking results. Existing detection schemes are mostly based on deep learning object detection models such as YOLO and Fast R-CNN, or traditional binarization methods. However, due to cell adhesion, low resolution, background noise, and variations in brightness in the acquired zebrafish images, detection often suffers from significant errors. Training YOLO and Fast R-CNN deep learning object detection models also requires a large amount of labeled image data, resulting in high experimental costs. Existing methods for calculating heart rate and ejection fraction typically involve binarizing zebrafish heart fluorescence images using dynamic or static thresholding to extract and segment the heart region, obtaining the heart area, and then finding peaks and troughs by comparing each frame with the initial frame. This method performs relatively poorly under conditions of low image resolution, varying brightness, and weak fluorescence interference from other tissues. Furthermore, the frame difference method is less accurate in finding peaks and troughs when calculating heart rate and ejection fraction due to spikes and noise in the heart area waveform.

[0004] Therefore, there is an urgent need to develop an intelligent and automated method for identifying and evaluating the cardiovascular function of zebrafish for related experimental research. Summary of the Invention

[0005] The purpose of this invention is to improve the automation and intelligence of the data processing workflow for analyzing and calculating zebrafish heart rate, ejection fraction, and blood flow velocity from zebrafish images, so as to better assess the cardiovascular function of zebrafish and provide strong technical support for zebrafish-related research and the development of drugs for complex cardiovascular diseases.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for intelligent identification and evaluation of cardiovascular function in zebrafish includes: preprocessing each frame of a zebrafish video containing erythrocyte fluorescence and a zebrafish video containing cardiac fluorescence; then using a constructed zebrafish image recognition and classification model to select cellular fluorescence time-series images and cardiac fluorescence time-series images with suitable poses; preprocessing the selected time-series images and using a constructed zebrafish image cell / heart segmentation model to segment the cells in the cellular fluorescence time-series images and the heart in the cardiac fluorescence time-series images; then: tracking the segmented cells, calculating the trajectory information of each cell, and calculating the cell flow velocity in the blood; calculating the area of ​​the heart region for the segmented heart, plotting the waveform of the heart area changing over time, and obtaining the ejection fraction and heart rate data; finally, outputting the information data of cell flow velocity, heart rate, and ejection fraction to evaluate cardiovascular function.

[0008] Preferably, the zebrafish image recognition and classification model is constructed by the following steps:

[0009] (A-1) Acquire images of fluorescent zebrafish cells and fluorescent zebrafish hearts, respectively, and perform image resizing and data augmentation to complete image preprocessing;

[0010] (A-2) The preprocessed images are manually classified. The images with classification information are used as the dataset to train the zebrafish image recognition classification model to obtain the zebrafish image recognition classification model to classify cell fluorescent zebrafish images and heart fluorescent zebrafish images.

[0011] Preferably, classification is performed according to lateral pose, ventral pose, dorsal pose, absence of zebrafish in the field of vision, and zebrafish with distorted or deformed poses. Classifications can also be added or deleted as needed to meet various calculation and evaluation requirements.

[0012] Preferably, the zebrafish image recognition and classification model adopts the ResNet50 neural network model; during the training process, stochastic gradient descent is used as the optimizer for model training, and cross-entropy loss function is used as the loss function; the model parameters are initialized by the model trained on the ImageNet dataset as the initialization model parameters of the ResNet50 neural network model.

[0013] Preferably, the zebrafish image cell / heart segmentation model is constructed by the following steps:

[0014] (B-1) The cell fluorescence time series images / cardiac fluorescence time series images screened by the zebrafish image recognition classification model are resized and data augmented to complete the image preprocessing;

[0015] (B-2) The regions of interest in the preprocessed image are labeled to obtain a mask. The image and the corresponding mask are used as a dataset to train the segmentation model, and zebrafish image cell segmentation model and zebrafish image heart segmentation model are obtained for cell segmentation and heart segmentation of cell fluorescent zebrafish image and heart fluorescent zebrafish image, respectively.

[0016] Preferably, the segmentation model is the UET segmentation model; during training, the Adam optimizer is used to optimize the model parameters, and the Dice loss function is used as the loss function.

[0017] In actual identification and evaluation, a camera can be used to record microscopic images of zebrafish, which are then transmitted to a computer and stored. These images are then converted into multiple frames and further preprocessed before the intelligent identification and evaluation of the cardiovascular function of the zebrafish is performed.

[0018] As a preferred method, the specific steps for tracking the segmented cells, statistically analyzing the trajectory information of each cell, and calculating the cell flow velocity are as follows:

[0019] (C-1) Perform morphological opening and morphological erosion operations on the image output by the zebrafish image cell / heart segmentation model to obtain mutually separated cell regions;

[0020] (C-2) Extract the contour of the cell region, calculate the bounding box of the contour, and use it as the bounding box of the cell in each frame image;

[0021] (C-3) Use the Kalman filter and Hungarian matching algorithm to complete the matching and tracking of each cell and obtain the motion trajectory of each cell;

[0022] (C-4) Based on the cell movement trajectory, calculate the average movement speed of each tracked cell during the tracking period, and thus obtain the cell flow velocity.

[0023] Preferably, in step (C-3):

[0024] (I) For a target cell, based on the state of the cell in the previous frame, use a Kalman filter to estimate the predicted bounding box of the target cell in the current frame.

[0025] (II) Calculate the intersection-union ratio (IUU) of the predicted bounding box and the actual detected bounding box in the current frame, construct the allocation cost matrix, input the allocation cost matrix into the Hungarian matching algorithm, obtain the matching result, and update the trajectory: (i) If there is an actual detected bounding box that matches the predicted bounding box, update the state of the target cell with the information of the actual detected bounding box; (ii) If no cell bounding box that matches the predicted bounding box is detected within a set time, stop tracking; (iii) If the IUU of a detected cell bounding box and all existing targets is less than a set value, establish a new tracking target.

[0026] Preferably, when calculating the area of ​​the segmented heart region, plotting the waveform of the heart area changing over time, and obtaining ejection fraction and heart rate data, the following steps are included:

[0027] (D-1) For the segmented heart region, extract the zebrafish heart outline and calculate the area of ​​the zebrafish heart region in each frame.

[0028] (D-2) Draw a line graph showing the change in heart area over time;

[0029] (D-3) Use the automatic multi-scale peak finding algorithm to find peaks and troughs on the line chart;

[0030] (D-4) Calculate the heart rate based on the number of peaks and troughs; calculate the ejection fraction based on the peak and trough values.

[0031] Preferably, the cell fluorescence image and cardiac fluorescence image of the suitable posture are images of zebrafish in a lateral pose.

[0032] As another implementation scheme, a method for intelligent identification and evaluation of cardiovascular function in zebrafish includes the following steps:

[0033] (1) The camera records microscopic images of zebrafish, which are then transmitted to a computer and stored.

[0034] (2) Perform image preprocessing on the collected images, construct a zebrafish image recognition and classification dataset, and train the ResNet50 zebrafish image recognition and classification convolutional neural network.

[0035] (3) Using the zebrafish image recognition and classification model trained in step (2), zebrafish time-series images (including cell fluorescence time-series images and cardiac fluorescence time-series images) that are easy to segment and process in the large number of zebrafish images obtained in the experiment are selected.

[0036] (4) Perform image preprocessing on the zebrafish time series images obtained in step (3) to construct zebrafish cell segmentation dataset and heart segmentation dataset, and train the corresponding UET semantic segmentation convolutional neural network (UTS segmentation model) respectively.

[0037] (5) For each frame of the cell fluorescence video data (cell fluorescence time-series image) in the blood flow of zebrafish, the cell region in the image is segmented using the UET model trained in step (4), and the detection bounding box of each cell is calculated. The cell bounding box information detected in each frame is input into the cell tracker for cell tracking, and the cell flow velocity is calculated after the trajectory information of each cell is statistically analyzed.

[0038] (6) For each frame of the zebrafish heart fluorescence video (heart fluorescence time-series image), the heart region in the image is segmented using the UET segmentation model trained in (4), and the area of ​​the heart region is calculated. A waveform diagram of the heart area changing over time is plotted. The peaks and troughs of the heart area are statistically analyzed using an automatic multi-scale peak finding algorithm to obtain the ejection fraction and heart rate data.

[0039] (7) Output information data on blood cell flow velocity, heart rate, and ejection fraction to assess cardiovascular function.

[0040] Further, in step (1), fluorescent images of zebrafish are captured using a microscope camera, and the image data is transmitted via USB. The raw zebrafish image data is stored in a database. The zebrafish strain used for the image is determined based on the type of data required for assessing cardiovascular function. If heart rate and ejection fraction are required, heart-specific fluorescent zebrafish are selected; if blood flow velocity is required, zebrafish with fluorescent blood cells are selected (generally, erythrocyte-specific fluorescent zebrafish are selected).

[0041] Furthermore, step 2 consists of the following specific steps:

[0042] (21) Adjust the size of the acquired images to 224×224×3, and add transformations such as random cropping, random rotation, random flipping, random brightness adjustment, and normalization to the images to augment the image dataset.

[0043] (22) The preprocessed zebrafish microscopic images are classified according to lateral pose, ventral pose, dorsal pose, no zebrafish in the field of view, and zebrafish with distorted pose. A 5-fold cross-validation is used to divide the test set and training set (the ratio can be adjusted according to the training results). In each iteration, one set is selected sequentially as the test set, and the rest are used as the training set. In addition, several zebrafish images are randomly selected as the validation set to test the accuracy of the model.

[0044] (23) Perform transfer learning on the model. Set the model optimizer to SGD (Stochastic Gradient Descent), the batch size to 4 (adjustable based on training results), the number of iterations to 25 (adjustable based on training results), and the loss function to cross-entropy loss, the calculation formula of which is:

[0045]

[0046] Where M represents the number of categories, y ic The sign function is p, which takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic This represents the predicted probability that sample i belongs to class c. During model training, transfer learning is performed, using model parameters trained on the large-scale ImageNet dataset as feature extractors. The model weights, except for those in the fully connected layers, are frozen, and the fully connected layer parameters are trained. Depending on the training results, it is also possible to choose not to freeze the model weights except for those in the fully connected layers and optimize all parameters during training.

[0047] Furthermore, the ResNet50 zebrafish image recognition and classification convolutional neural network trained in step (2) consists of 7 parts. The first part consists of convolutional layers (7×7 kernel size), batch normalization layers, ReLU activation function layers, and max pooling layers (3×3 kernel size). The second, third, fourth, and fifth parts consist of residual blocks, containing 3, 4, 6, and 3 residual blocks respectively. A residual block is a neural network structure that adds the input before several layers (here, several layers refer to 2 1×1 convolutional layers, 2 batch normalization layers, 2 ReLU activation function layers, and 1 3×3 convolutional layer) to the output after several layers and then outputs it through the ReLU activation function. This can prevent gradient vanishing and model degradation. The structure of each residual block varies slightly depending on the size of the input and output. If the third dimension of the input and output are equal, the input before several layers and the output after several layers are directly added. Otherwise, the original input before several layers is convolved (1×1 kernel size), batch normalized, and then added to the result after several layers. After the calculations in the first five parts, the output size is 7×7×2048. The sixth part is an average pooling layer that transforms it into a feature vector. The final seventh part consists of a fully connected layer and a softmax layer, which outputs the probability of each image classification.

[0048] Furthermore, in step (3), when using the zebrafish image recognition classification model to select zebrafish images that are easy to segment, the original acquired image is input into the trained classification model to obtain the probability distribution of image classification. The classification with the highest probability is selected as the image classification. Zebrafish image classification includes lateral pose, ventral pose, dorsal pose, no zebrafish in the field of view, and zebrafish with distorted or deformed poses. Based on the tissue part to be segmented laterally, the corresponding pose image is selected. When assessing cardiovascular function, a lateral pose zebrafish image is selected, as the heart and blood vessels of the zebrafish in this pose are relatively clear and easy to see, making it convenient for segmentation.

[0049] Furthermore, step (4) consists of the following specific steps:

[0050] (41) For the zebrafish image to be segmented, convert the image to grayscale and adjust the image size to 512×512. Image data needed for training zebrafish heart segmentation or blood cell segmentation models also requires random brightness adjustments, random noise reduction, and other transformations to enhance the data and improve the model's generalization ability. The division of the training set, test set, and validation set is handled in the same way as when training the zebrafish detection and classification model.

[0051] (42) Use annotation software to annotate the regions of interest (cells or heart) to obtain a mask (an image where a pixel is 1 if it belongs to the segmented region, and 0 otherwise). Divide the training set, test set, and validation set into a 4:1:1 ratio (this can be adjusted according to the training effect).

[0052] (43) Train the zebrafish cell segmentation model and the heart segmentation model. The Dice loss function is selected as the loss function, and its calculation formula is as follows:

[0053]

[0054] Where `pred` is the set of predicted values ​​and `true` is the set of true values. During training, the batch size is set to 2 (which can be adjusted based on the training results), the number of iterations is set to 30 (which can be adjusted based on the training results), the model optimizer is set to `adam` (which can be adjusted based on the training results), and the model parameters are initialized either randomly or using existing `unet` splitting model parameters.

[0055] Further, the UET semantic segmentation convolutional neural network trained in step (4) takes the filtered and preprocessed zebrafish image as input and outputs the segmentation results of zebrafish cells, tissues, or organs. The UET neural network is a symmetrical encoder-decoder structure. The left side of the network is the encoder, which consists of 4 downsampling operations. The downsampling operation consists of 2 convolutions (3×3 kernel size, after each convolution, it will pass through a batch normalization layer and a ReLU activation function layer) and max pooling (2×2 kernel size) operations. The right side of the network is the decoder, which consists of 4 upsampling operations. The upsampling operation consists of 2 convolutions (3×3 kernel size) and upconvolution (2×2 kernel size) operations. The image is processed by the encoder to obtain a 1024-dimensional feature vector. In the decoder stage, each upsampling will also stitch the feature map generated in the symmetrical downsampling stage across levels onto the generated feature map.

[0056] Furthermore, step (5) includes the following specific steps:

[0057] (51) Perform morphological opening operation (first erosion operation, then dilation operation) on each frame of the segmented zebrafish blood flow red blood cell fluorescence video to reduce the situation where background noise is incorrectly segmented into cells for tracking; then perform morphological erosion operation on the image to separate the adhered cells.

[0058] (52) Extract the contours of the segmented cells, calculate the bounding boxes of the contours, and input the target detection results into the prediction module. The prediction module is based on a Kalman filter, and its state vector is a 7-dimensional vector:

[0059]

[0060] Where u and v represent the horizontal and vertical pixel positions of the target center, respectively, while s and r represent the size and aspect ratio of the target cell's bounding box (the aspect ratio is a constant).

[0061] (53) Predict the new position of each target in the current frame using a Kalman filter, estimate its bounding box shape, and then perform data association. Calculate the allocation cost matrix (calculated as 1-IoU) based on the intersection-over-union (IoU) ratio between each currently detected bounding box and the predicted bounding boxes of existing targets. Match targets with currently detected bounding boxes using the Hungarian matching algorithm. If a target matches a detected cell bounding box, update the target's observation state in the Kalman filter and its corresponding trajectory information. If the IoU between a detected cell bounding box and all existing targets is small (i.e., less than IoU),... minIf a target is not detected to have a matching cell bounding box within a certain time period, it is considered to have disappeared from the field of view and is no longer tracked. The recorded cell movement trajectories are then plotted on the image.

[0062] (54) Based on the tracking trajectory, calculate the average movement speed of each tracked cell during the tracking period. The calculation formula is as follows:

[0063]

[0064] Where s is the distance the tracking cell moves from the start frame to the end frame, fps is the video recording frame rate, and N is the number of tracking frames.

[0065] Furthermore, step 6 includes the following specific steps:

[0066] (61) Binarization segmentation results: extract the zebrafish heart contour obtained from the segmentation, calculate the area of ​​the zebrafish heart region in each frame, and store it locally.

[0067] (62) Draw a line graph showing the change in heart area over time.

[0068] (63) Calculate the peaks using the Automatic Multi-Scale Peak Finder (AMPD) algorithm, multiply the waveform by -1, and calculate the troughs using the same algorithm.

[0069] (64) Calculate the number of troughs within one minute to obtain the heart rate. Calculate the ejection fraction (EF) using the following formula:

[0070]

[0071] Among them, EDV is the end-diastolic volume of the ventricle, and ESV is the end-systolic volume of the ventricle. Since two-dimensional video cannot measure ventricular volume, researchers usually use area as an estimate (that is, the peak value and trough value calculated in step 3).

[0072] The beneficial effects of this invention are as follows: For zebrafish images where a large number are unsuitable for segmentation, a zebrafish image classification convolutional neural network is trained for automatic image filtering, improving the efficiency of experimental data processing, reducing manual operations, and simplifying the processing flow. In calculating cell flow velocity in zebrafish blood, existing cell tracking solutions often use target detection convolutional neural networks like YOLO and Fast R-CNN or binarization algorithms to detect cells. However, in zebrafish images with cell adhesion, low resolution, background noise, and varying brightness, the detection accuracy is relatively poor. Furthermore, target detection neural networks require a large amount of data samples for training. This invention, after segmenting the image using the Unet semantic segmentation network, further separates adhered cells through morphological operations, eliminates particle interference, and extracts cell contours to obtain bounding boxes for cell detection. The Unet network can be trained on small datasets, reducing the workload of collecting zebrafish image data for training and maintaining relatively good detection accuracy even on low-quality images. When calculating heart rate and ejection fraction, the Unet semantic segmentation network segments the heart region to calculate its area, which is more accurate than the traditional method of setting a threshold and binarizing the image to calculate the heart area. This invention employs an automatic multi-scale peak finding algorithm to locate the peaks and troughs of the cardiac area waveform for calculating heart rate and ejection fraction, thus avoiding the impact of waveform noise on accuracy. The proposed method combines a deep learning classification and segmentation model with cell tracking and waveform analysis algorithms to obtain information on blood cell flow velocity, heart rate, and ejection fraction, enabling automated and intelligent assessment of cardiovascular function. This provides strong technical support for zebrafish-related research and the development of drugs for complex cardiovascular diseases. Attached Figure Description

[0073] Figure 1 This is a flowchart of the intelligent identification and evaluation method for cardiovascular function in zebrafish described in this invention;

[0074] Figure 2 This is a structural diagram of the zebrafish detection and classification convolutional neural network described in this invention;

[0075] Figure 3 This is a structural diagram of the zebrafish image segmentation convolutional neural network described in this invention. Detailed Implementation

[0076] To make the objectives, technical solutions, and effects of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described in this specification are merely illustrative of the invention and not intended to limit it.

[0077] This invention discloses an intelligent identification and evaluation method for cardiovascular function in zebrafish. The method involves capturing microscopic images of zebrafish, preprocessing them, and then inputting the pre-trained zebrafish recognition and classification deep learning model. Images unfavorable to observation (blurred, obstructed, deformed, or improperly positioned images, etc.) are filtered out. Cell or tissue segmentation results are obtained using a zebrafish cell and tissue segmentation model. Blood cell flow velocity, heart rate, and ejection fraction are then obtained through cell tracking algorithms and waveform analysis to assess and visualize cardiovascular function. This method improves the efficiency of zebrafish experimental data analysis and processing, automatically selecting suitable image data from a large number of images acquired in experiments and analyzing it to obtain valuable information for assessing cardiovascular function. It reduces manual operation, simplifies the processing flow, and requires relatively few parameter adjustments, providing strong technical support for zebrafish-related research and the development of drugs for complex cardiovascular diseases.

[0078] The flowchart of the intelligent identification and evaluation method for cardiovascular function in zebrafish described in this invention is attached. Figure 1 As shown, in this embodiment, the specific steps of the method are as follows:

[0079] (1) The image acquisition module of the system captures fluorescent images of zebrafish (the zebrafish strain is Tg(LCR:EGFP) or TgBAC(-36nkx2.5:ZsYellow), the former is erythrocyte fluorescent zebrafish used to calculate the flow rate of cells in the blood, and the latter is cardiac precursor cell fluorescent zebrafish used to calculate the heart area) through a microscope camera, and transmits the image data to the computer and stores it in the database in png format.

[0080] (2) Preprocess the image data obtained in step (1) and construct a zebrafish image classification dataset to train a zebrafish image recognition classification model. In this embodiment, the image obtained in step (1) is an RGB three-channel image with a resolution of 2048×1536. Adjust the image size to 224×224×3. Add random rotation (0-10 degrees), random flipping, random brightness adjustment (0-0.1), and normalization transformations to the image dataset to perform data augmentation. Select about 200 processed images and classify them according to lateral pose, ventral pose, back pose, no zebrafish in the field of view, and zebrafish pose distortion. Build a ResNet50 zebrafish image classification convolutional neural network (zebrafish image recognition classification model) in the PyTorch framework. During training, set the model optimizer to SGD (stochastic gradient descent), the batch size to 4, the number of iterations to 25, the learning rate to 0.001, and the loss function to the cross-entropy loss function, the calculation formula of which is:

[0081]

[0082] Where M represents the number of categories, N represents the number of samples, and y ic The sign function is p, which takes the value 1 if the true class of sample i is equal to c, and 0 otherwise. ic This represents the predicted probability that sample i belongs to category c.

[0083] (3) Use the trained zebrafish image recognition and classification model to determine whether the image is appropriate. When analyzing zebrafish blood flow cells or heart, select zebrafish images with lateral poses and store them in the database for segmentation processing.

[0084] (4) Train the zebrafish image segmentation model using the selected images. Select approximately 60 zebrafish images in lateral poses, convert them to grayscale, and adjust their size to 512×512×1. Add random brightness adjustment (0-0.1) and random contrast adjustment (0-0.1) to the images to enhance the data and improve the model's generalization ability. Use LabelMe annotation software to annotate the regions of interest (cells or hearts) to obtain a mask (an image where a pixel value is 1 if it belongs to the segmentation region, and 0 otherwise). Divide the training set, test set, and validation set into a 4:1:1 ratio. During training, set the model optimizer to Adam, batch size to 2, iterations to 30, learning rate to 0.001, and loss function to Dice loss function, the calculation formula of which is:

[0085]

[0086] Where `pred` is the set of predicted values ​​and `true` is the set of true values. During training, the batch size is set to 2 (this can be adjusted based on the training results), and the model parameters are initialized randomly.

[0087] Step (4) yields a zebrafish image cell segmentation model (unet cell segmentation model) for cell segmentation and a zebrafish image heart segmentation model (unet heart segmentation model) for heart segmentation.

[0088] (5) Using the cell segmentation model trained in step (4), segment each frame of the erythrocyte fluorescent zebrafish video, track the cells after segmentation, and count the flow velocity of erythrocytes in the bloodstream. The specific steps are as follows:

[0089] (51) Perform grayscale processing and size adjustment on the image using the same method as in the model training stage. Input the processed image into the trained UET cell segmentation model, perform an opening operation on the output (operation box size 3×3) to eliminate small particle noise, and then perform an erosion operation (convolution kernel size 3×3) to prevent cell adhesion.

[0090] (52) Extract the contours of the segmented cells to obtain the bounding boxes of the contours, and input them as the target detection results into the Kalman filter. The state vector of the Kalman filter is a 7-dimensional vector:

[0091]

[0092] Where u and v represent the horizontal and vertical pixel positions of the target center, respectively, while s and r represent the size and aspect ratio of the target cell's bounding box (the aspect ratio is a constant).

[0093] (53) Predict the new position of each target in the current frame using a Kalman filter, estimate its bounding box shape, and then perform data association. Calculate the allocation cost matrix (calculated as 1-IoU) based on the intersection-over-union (IoU) ratio between each currently detected bounding box and the predicted bounding boxes of existing target cells. Match the target bounding box with the currently detected bounding box using the Hungarian matching algorithm. If the target bounding box matches the detected cell bounding box, update the observation state of the target cell in the Kalman filter and the corresponding trajectory information. If the IoU between a detected cell bounding box and all existing target cell bounding boxes is small (i.e., less than IoU), the allocation cost matrix is ​​calculated. min In this embodiment, IoU is set. min If the value is 0.2, a new tracking target will be established based on it. If a tracked target cell does not have a matching cell bounding box within a certain time (2 frames in this example), it is considered to have disappeared from the field of view and will no longer be tracked. The recorded cell motion trajectory is plotted on the image.

[0094] (54) Based on the tracking trajectory, calculate the average movement speed of each tracked cell during the tracking period. The calculation formula is as follows:

[0095]

[0096] Where s is the distance the tracking cell moves from the start frame to the end frame, fps is the video recording frame rate, and N is the number of tracking frames.

[0097] (6) Using the UET heart segmentation model trained in step (4), segment each frame of the fluorescent zebrafish video of the heart to obtain the heart area waveform. Analyze the waveform to obtain heart rate and ejection fraction to assess cardiovascular function. The specific steps are as follows:

[0098] (61) Extract the zebrafish heart contour obtained by the UET model segmentation, calculate the area of ​​the zebrafish heart region in each frame, and store it in a local CSV file.

[0099] (62) Read the CSV data and plot the waveform of the heart area over time. Calculate the peaks using the Automatic Multiscale Peak Finder (AMPD) algorithm, multiply the waveform by -1, and calculate the troughs using the same algorithm.

[0100] (63) Calculate the number of troughs within one minute to obtain the heart rate. When calculating the ejection fraction, use the following formula:

[0101]

[0102] Wherein, EDV is the end-diastolic volume of the ventricle, and ESV is the end-systolic volume of the ventricle. Since two-dimensional video cannot measure ventricular volume, researchers usually use area as an estimate (that is, the peak and trough values ​​calculated in step 3).

[0103] (7) Output the calculated blood flow red blood cell velocity, ejection fraction, and heart rate data.

[0104] In this embodiment, the structure of the zebrafish image classification ResNet50 model is shown in the attached figure. Figure 2 As shown, it consists of 7 parts: Part 1 consists of convolutional layers (7×7 kernel size), batch normalization layers, ReLU activation function layers, and max pooling layers (3×3 kernel size). Parts 2, 3, 4, and 5 consist of residual blocks, containing 3, 4, 6, and 3 residual blocks respectively. A residual block is a neural network structure that adds the input before several layers (here, several layers refer to 2 1×1 convolutional layers, 2 batch normalization layers, 2 ReLU activation function layers, and 1 3×3 convolutional layer) to the output after several layers and then outputs it through the ReLU activation function. This can prevent gradient vanishing and model degradation. The structure of each residual block varies slightly depending on the size of the input and output. If the third dimension of the input and output are equal, then the input before several layers and the output after several layers are directly added. Otherwise, the original input before several layers is convolved (1×1 kernel size), batch normalized, and then added to the result after several layers. After the calculations in the first five parts, the output size is 7×7×2048. The sixth part is an average pooling layer that transforms it into a feature vector. The final seventh part consists of a fully connected layer and a softmax layer, which outputs the probability of each image classification.

[0105] In this embodiment, the structure of the UNET semantic segmentation convolutional neural network used for zebrafish image segmentation is shown in the attached figure. Figure 3As shown. In this embodiment, the preprocessed zebrafish image is a 512×512 grayscale image input to the network, and the final result is a segmentation mask image (the pixel value of the region to be segmented is 1, otherwise it is 0). The UNET neural network is a symmetrical encoder-decoder structure. The left side of the network is the encoder, which consists of 4 downsampling operations. The downsampling operation consists of 2 convolutions (3×3 kernel size, after each convolution, it is passed through a batch normalization layer and a ReLU activation function layer) and max pooling (2×2 kernel size). The right side of the network is the decoder, which consists of 4 upsampling operations. The upsampling operation consists of 2 convolutions (3×3 kernel size) and upconvolution (2×2 kernel size). The image is processed by the encoder to obtain a 1024-dimensional feature vector. In the decoder stage, each upsampling operation also concatenates the feature map generated in the symmetrical downsampling stage to the generated feature map.

[0106] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for intelligent identification and evaluation of cardiovascular function of zebrafish, characterized in that, The application comprises the following steps: Preprocess each frame image of the cell fluorescence zebrafish video and the heart fluorescence zebrafish video, and then screen out cell fluorescence time sequence images and heart fluorescence time sequence images with suitable postures by using the constructed zebrafish image recognition classification model respectively; preprocess the screened time sequence images, and separate the cells in the cell fluorescence time sequence images and the heart in the heart fluorescence time sequence images by using the constructed zebrafish image cell / heart segmentation model respectively; Then: track the separated cells, calculate the flow rate of the cells in the blood after counting the trajectory information of each cell, calculate the area of the separated heart, draw a waveform diagram of the change of the area of the heart with time, obtain the ejection fraction and heart rate data, and finally output the information data of the cell flow rate, the heart rate and the ejection fraction to evaluate the cardiovascular function.

2. The method according to claim 1, wherein, When the zebrafish image recognition classification model is constructed, the following steps are included: (A-1) Collect cell fluorescence zebrafish images / heart fluorescence zebrafish images respectively, adjust the size of the images and perform data enhancement, and complete the preprocessing of the images; (A-2) Manually classify the preprocessed images, use the images with classification information as a data set to train the classification model, and obtain zebrafish image recognition classification models for classifying cell fluorescence zebrafish images and heart fluorescence zebrafish images respectively.

3. The method according to claim 2, wherein, The zebrafish image recognition classification model adopts a resnet50 neural network model; in the training process, a stochastic gradient descent method is used as the optimizer for model training, and a cross-entropy loss function is used as the loss function; the model parameters initialized by the model trained on the imagenet data set are used as the initialization model parameters of the resnet50 neural network model.

4. The method according to claim 1, wherein, When the zebrafish image cell / heart segmentation model is constructed, the following steps are included: (B-1) Adjust the size of the cell fluorescence time sequence images / heart fluorescence time sequence images screened by the zebrafish image recognition classification model and perform data enhancement, and complete the preprocessing of the images; (B-2) Label the regions of interest in the preprocessed images to obtain masks, use the images and the corresponding masks as a data set to train the segmentation model, and obtain zebrafish image cell segmentation models and zebrafish image heart segmentation models for cell segmentation and heart segmentation of cell fluorescence zebrafish images and heart fluorescence zebrafish images respectively.

5. The method according to claim 4, wherein, The zebrafish image cell / heart segmentation model adopts a unet segmentation model; in the training process, an Adam optimizer is used to optimize the model parameters, and a dice loss function is used as the loss function.

6. The method according to claim 1, wherein, The specific steps of tracking the separated cells and calculating the flow rate of the cells in the blood after counting the trajectory information of each cell are as follows: (C-1) Perform morphological opening operation and morphological erosion operation on the images output by the zebrafish image cell segmentation model to eliminate small noise points on the images and separate the adhered cells; (C-2) Extract the contour of the cell region, calculate the bounding box of the contour, and use the bounding box as the bounding box of the cells in each frame image; (C-3) Based on the Kalman filter and Hungarian matching algorithm, the matching and tracking of each cell is completed to obtain the motion trajectory of each cell; (C-4) Based on the movement trajectory, calculate the average movement speed of each tracked cell during the tracking period, and thus obtain the cell flow velocity.

7. The method according to claim 6, wherein the method is characterized by, In step (C-3): (I) For a target cell, based on the state of the cell in the previous frame, use a Kalman filter to estimate the predicted bounding box of the target cell in the current frame. (II) Calculate the intersection-union ratio (IUU) of the predicted bounding box and the actual detected bounding box in the current frame, construct the allocation cost matrix, input the allocation cost matrix into the Hungarian matching algorithm, obtain the matching result, and update the trajectory: (i) If there is an actual detected bounding box that matches the predicted bounding box, update the state of the target cell with the information of the actual detected bounding box; (ii) If no cell bounding box that matches the predicted bounding box is detected within a set time, stop tracking; (iii) If the IUU of a detected cell bounding box and all existing targets is less than a set value, establish a new tracking target.

8. The method according to claim 1, wherein the method is characterized by, The process of calculating the area of ​​the segmented heart region, plotting the waveform of the heart area changing over time, and obtaining ejection fraction and heart rate data includes the following steps: (D-1) For the segmented heart region, extract the zebrafish heart outline and calculate the area of ​​the zebrafish heart region in each frame. (D-2) Draw a line graph showing the change in heart area over time; (D-3) Use the automatic multi-scale peak finding algorithm to find peaks and troughs on the line chart; (D-4) The heart rate is calculated based on the number of peaks and troughs; Ejection fraction is calculated based on peak and trough values.

9. The method according to claim 1 or 2, wherein, The cell fluorescence images and cardiac fluorescence images of the appropriate pose are images of zebrafish in a lateral pose.

10. The method according to claim 1, wherein the method is characterized by, Includes the following steps: (1) The camera records microscopic images of zebrafish, which are then transmitted to a computer and stored. (2) Perform image preprocessing on the acquired images; (3) Use the trained zebrafish image recognition and classification model to screen out the lateral pose cell fluorescence time series images and cardiac fluorescence time series images from a large number of preprocessed zebrafish images; (4) Image preprocessing was performed on the time-series images of cell fluorescence and cardiac fluorescence; (5) For each frame of the zebrafish cell fluorescence time-series image, the cell region in the image is segmented using the trained UET segmentation model to obtain the detection bounding box of each cell; The cell bounding box information detected in each frame is input into a cell tracker consisting of a Kalman filter and a Hungarian matching algorithm to perform cell tracking. After statistical analysis of the trajectory information of each cell, the cell flow velocity is calculated. (6) For each frame of the cardiac fluorescence time series image, the trained UET segmentation model is used to segment the cardiac region in the image, and the area of ​​the cardiac region is calculated. The waveform of the cardiac area changing over time is plotted, and the peaks and valleys of the cardiac area are counted by the automatic multi-scale peak finding algorithm. The ejection fraction and heart rate data are then analyzed. (7) Output information data on blood cell flow velocity, heart rate, and ejection fraction to assess cardiovascular function.

Images