A real-time measurement method for cardiac outflow tract wall radial strain based on neural network

Abstract

The present application belongs to the field of biomedical and computer vision technology, and discloses a kind of real-time measurement method of cardiac outflow tract wall radial strain based on neural network. Image is collected using SD-OCT imaging system to obtain structure diagram. OFT-SegNet network is used to extract the boundary of cardiac outflow tract in the structure diagram, then based on these boundary information, by measuring the length change of myocardial cells before and after contraction, comparing these length changes with the length before contraction, the final radial strain value can be obtained. The OFT-SegNet network designed by the method has small complexity, can be migrated to mobile devices, and has fast inference speed. The OFT-SegNet network designed by the invention proposes a feature fusion module, so that the network can better understand the structural features of the image, and improve the accuracy of segmentation and strain calculation. The method can obtain the cardiac outflow tract wall radial strain in real time at a faster speed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of biomedical and computer vision, and particularly relates to a real-time measurement method of cardiac outflow tract wall radial strain based on a neural network. BACKGROUND

[0002] Myocardial wall strain (MWS) of embryonic heart is an important parameter for quantifying the mechanical properties of cardiac tissue. MWS can be divided into morphogenic strain and mechanical strain. The morphogenic strain changes relatively slowly due to cell growth, division, movement, shape change and death. The mechanical strain changes quickly due to external load. Generally, the morphogenic strain can be affected by the mechanical strain or the stress related thereto. Since the cardiac outflow tract (OFT) connects the ventricle and arterial system in the early stage of heart development, many congenital heart diseases are related to abnormal changes of the OFT in the process of heart development. These abnormalities can cause developmental defects of heart valves and major blood vessels, such as transposition of the great arteries, and therefore, it is of great significance to measure the strain of the cardiac outflow tract for the research of congenital heart disease.

[0003] In order to explore embryonic development, chicken embryos are often used as experimental objects. It has long been recognized that chicken embryos are attractive experimental models for studying angiogenesis, heart formation and neurohumoral control of the heart. Studies have shown that chicken embryos, like zebrafish, can also be used to study developmental changes in local vascular tone and hemodynamic control. However, the early chicken embryo heart is small in volume, beats and difficult to measure. Optical coherence tomography (OCT) is a powerful non-invasive imaging technology with high temporal and spatial resolution, and the imaging depth in highly scattering biological tissue can reach 2 millimeters, which is very suitable for early stage chicken embryo heart development research. However, there are the following problems in the segmentation and extraction of the OFT: (1) the early chicken embryo OFT is small in volume, thin in wall and small in radial strain change, so accurate segmentation is needed to achieve accurate measurement; (2) the OFT is adhered to other surrounding tissues, resulting in a blurred boundary and being difficult to distinguish; (3) the blood flow in the OFT can produce a tailing effect, and the faster the blood flow speed, the greater the influence. Traditional image processing methods have difficulty in segmenting the blurred areas caused by adhesion or tailing effect. Therefore, the existing strain measurement basically measures a single radial direction to avoid these areas. This greatly reduces the measurement range. Therefore, the efficient and accurate segmentation of the OFT has become a bottleneck problem for strain measurement based on OCT.

[0004] With the wide application of deep learning network in the field of medical imaging, image segmentation methods based on convolutional neural network (CNN) can be used to automatically extract any target region within the imaging range. The development of these deep learning techniques makes it possible to measure the radial strain of the heart OFT in real time, especially in the key step of automatically extracting the boundary of the heart OFT. However, due to the high complexity and significant individual specificity of the structure around the OFT, the traditional segmentation technology based on deep learning has the problem of blurred boundary and missing part of the structure when applied to OFT boundary segmentation. In order to deal with those areas lacking obvious structural features in OFT image segmentation, the model needs to be able to predict the subtle structure and overall features that can only be identified by artificial comprehensive judgment. This requires the algorithm to not only focus on obvious image features, but also be able to capture and understand the complex relationship between local subtle information and overall image context.

[0005] Based on the above difficulties, how to use the method of deep learning to measure the wall radial strain of the cardiac outflow tract in real time needs further research. SUMMARY

[0006] To at least partially solve one of the technical problems existing in the prior art, the purpose of the present application is to propose a neural network-based real-time measurement method for cardiac outflow tract wall radial strain.

[0007] The technical solution adopted by the present application is: a neural network-based real-time measurement method for cardiac outflow tract wall radial strain, specifically as follows:

[0008] An SD-OCT imaging system is built to collect images and obtain a structure map;

[0009] The OFT-SegNet network is used to extract the boundary of the cardiac outflow tract in the structure map. Then, based on this boundary information, by measuring the length change of myocardial cells before and after contraction, comparing these length changes with the length before contraction, the final radial strain value can be obtained.

[0010] The OFT-SegNet network includes an encoder block, a feature fusion module, a decoder block, and a loss function;

[0011] The input of the OFT-SegNet network is a structure map, the features of images at different stages are extracted through the encoder block, and the features of images at different stages are input into the decoder; the feature fusion module is inserted in each layer of the encoder-decoder, and the final output feature map is generated through the feature fusion module. Finally, the information from the feature fusion module is gradually fused through the decoder, and the extracted boundary region is output at the last stage.

[0012] The encoder block mainly contains two parts, which are downsampling operation and convolution block composed of two consecutive convolution layers. MaxPooling operation is used to downsample the image, expand the receptive field, and convolution layer is used to extract the features of the image at different stages.

[0013] The feature fusion module includes a boundary enhancement branch and a boundary prediction branch.

[0014] The boundary enhancement branch uses the entire input sequence to calculate the weights of the image in the spatial and channel aspects respectively, and then multiplies the obtained weights with the input sequence. Let the input be X in ∈R C×H×W In order to reduce the computational amount of the model, the input X in is first subjected to a pooling operation with a pooling factor of 2; then the shape of the result after pooling is reshaped:

[0015] X r =Reshape(Maxpooling(X in )),

[0016] In the channel feature enhancement module part, X is multiplied by X r , and a Softmax layer is applied to calculate the channel feature map to obtain X cc ∈R 1×C×C At the same time, the obtained channel feature map X cc is multiplied by X r , and then an up-sampling calculation is performed to obtain the final channel feature enhancement output X c ∈R C×H×W The feature enhancement output of each channel is the weighted sum of all channel features and original features, which helps to improve the distinguishability of channel features.

[0017]

[0018] X c =Unpooling(Reshape(X r ×X cc )),

[0019] In the spatial feature enhancement branch part, X r is multiplied by X , and a Softmax layer is applied to calculate the spatial feature map to obtain X The more similar the feature representations of the two positions are, the greater the correlation between them is proved, and the input X r is multiplied by the obtained spatial feature map X ssMultiply the results, reshape the shape of the result, and then perform an upsampling calculation to obtain X. s ∈R C×H×W The feature obtained for each location is a weighted sum of the features of all locations and the original features. Therefore, it has a global context view and selectively aggregates the context based on the spatial feature enhancement map.

[0020]

[0021] X s =Unpooling(Reshape(X) r ×X ss )),

[0022] Finally, the output X of the channel feature enhancement module c The output X of the spatial feature enhancement module s Multiply each by its respective weighting coefficient to obtain the final output: X out ∈R C×H×W .

[0023] X out =α×X s +β×X c

[0024] The boundary prediction branch divides each input image into multiple sequences row by row. As the images are input into the network from top to bottom, the correlation between each sequence and other sequences is calculated, and a memory module is generated so that the network can remember the previous observations. These are then combined with the subsequent parts to obtain the final predicted output. Let the input be X. in ∈R C×H×W To obtain the sequence features of the image, the input is first subjected to a 1×1 convolution to obtain the sequence image X. q ∈R 1×H×W The resulting sequence map X q The dataset is divided into H sequences, each containing W dimensions. These H sequences are sequentially input into the prediction module. After processing, the prediction module generates H prediction sequences, each containing W dimensions. Finally, these prediction sequences are aggregated to obtain the final prediction sequence image X. p ∈R 1×H×W Finally, the final predicted sequence map X will be obtained. p Perform a 1×1 convolution to obtain the output sequence X. out ∈R C×H×W .

[0025] X q =Conv(X in ),

[0026] X out= Conv(X p ),

[0027] Let the mth input sequence be X m ∈R 1×W : First, in order to obtain the correlation of this sequence and the last sequence, the input is first subjected to a convolution, and the obtained value is subjected to a Sigmoid function to obtain the memory weight M w . Then the weight is multiplied by the memory sequence M m-1 produced by the last sequence to obtain the corresponding associated memory output M C .

[0028] M C = Sigmoid(Conv(X m )) x M m-1 ,

[0029] Then, in order to obtain the input of this sequence and the influence coefficient on the entire output, the input is first subjected to a convolution, and the obtained value is subjected to a Sigmoid function to obtain the weight coefficient X w of the input sequence, and then the weight coefficient of the input sequence is multiplied by the input X in after convolution to obtain the corresponding associated input X C .

[0030] X in = Conv(X m ),

[0031] X C = Sigmoid(Conv(X m )) x X in ,

[0032] The associated input X C and the associated memory output M C are added to obtain the memory sequence M m produced by this sequence, and the memory sequence is input to the next sequence.

[0033] M m = M C + X C ,

[0034] Finally, in order to obtain the influence coefficient of the memory sequence M m produced by this sequence on the entire output, the input is first subjected to a convolution, and the obtained value is subjected to a Sigmoid function to obtain the weight C m of the memory sequence M w . Then the weight C m of the memory sequence M wThe memory sequence M after Sigmoid m Multiply by the weighting coefficients to obtain the final output C of this sequence. m .

[0035] C m =γ×(Sigmoid(Conv(X) m ))×Sigmoid(M m ))

[0036] The decoder block mainly consists of three parts: an upsampling layer, a feature fusion module, and a convolutional block composed of two consecutive convolutional layers. The decoder is primarily used to progressively fuse information from the feature fusion module.

[0037] The OFT-SegNet network is used for boundary segmentation of the cardiac outflow tract. The OFT-SegNet network is trained using Dice loss L. Dice Weighted binary cross-entropy loss L cross-entropy The combination of these factors results in this loss acting at the encoder's output.

[0038] Loss total =L Dice +L cross-entropy

[0039]

[0040] The radial strain of the cardiac outflow tract wall is calculated as follows:

[0041] First, distance transformation and skeletonization operations are performed on the boundary map of the myocardial wall to obtain thickness information at different locations. In the distance transformation operation, the distance from each pixel in the binary image to the nearest boundary pixel is calculated, thus obtaining the thickness of the myocardial wall at different locations along the central axis of the image. Simultaneously, skeletonization is used to extract the central axis from the binary image. Next, the distance matrix obtained from the distance transformation operation is multiplied by the central axis matrix obtained from the skeletonization operation, thereby obtaining the thickness of the myocardial wall at different locations on the skeleton map. Then, 360 straight lines are drawn based on the centroid of the skeleton map, and the thickness of the myocardial wall in the current direction is obtained by calculating the pixel size at the intersection of the skeleton and the straight lines. Finally, the thickness of the myocardial wall in different directions is compared with the thickness of the myocardial wall in the same direction before contraction to obtain the radial strain values ​​of the myocardial wall in different directions.

[0042] The beneficial effects of this invention are:

[0043] 1. The method for real-time measurement of radial strain of the cardiac outflow tract wall proposed in this invention, combined with the structural diagram, can acquire the radial strain of the cardiac outflow tract wall in real time at a relatively fast speed.

[0044] 2. The OFT-SegNet network designed in the present application proposes a feature fusion module, so that the network can better understand the structural features of the image, and improve the accuracy of segmentation and strain calculation. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 Flow chart for real-time measurement of cardiac outflow tract wall radial strain based on neural network in the present application example.

[0046] Figure 2 Structural diagram of SD-OCT in the present application example.

[0047] Figure 3 Structural diagram of OFT-SegNet network in the present application example.

[0048] Figure 4 Schematic diagram of feature fusion module in the present application example.

[0049] Figure 5 Schematic diagram of boundary enhancement branch in the present application example.

[0050] Figure 6 Schematic diagram of boundary prediction branch in the present application example.

[0051] Figure 7 Schematic diagram of radial strain calculation process in the present application example.

[0052] Figure 8 (a)- Figure 8 (h) Radial strain test results of chicken embryo heart at different beating cycle stages in the present application example 2D view.

[0053] Figure 9 Three-dimensional (3D) view of radial strain on OFT. (a) is the m-type structural image extracted from the OCT b-scan sequence with constant x-coordinate. Figure (b) shows the radial strain image corresponding to (a). Figure (c) shows the change of OFT boundary segmented in 3D view over time. (d) is the calculated radial strain change corresponding to (c). DETAILED DESCRIPTION

[0054] Embodiments of the present application are described below in detail with reference to the accompanying drawings, examples of which are shown in the drawings, wherein the same or similar notations represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as a limitation of the present application. For the step numbers in the following embodiments, they are only set for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] In the description of the present application, it should be understood that the orientation description, such as the orientation or position relationship indicated by up, down, front, back, left, right, etc. is based on the orientation or position relationship shown in the drawings, and is only for the convenience of describing the present application and simplifying the description, and therefore cannot be understood as a limitation of the present application.

[0056] In the description of the present application, the meaning of several is one or more, and the meaning of multiple is more than two, greater than, less than, more than, etc. are understood as not including the number, and above, below, etc. are understood as including the number. If it is described as first, second, it is only used for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features or the order of indicated technical features.

[0057] In the description of the present application, unless otherwise explicitly limited, the words such as setting, installing, connecting, etc. should be understood in a broad sense, and those skilled in the art can reasonably determine the specific meaning of the above words in the present application in combination with the specific content of the technical solution.

[0058] Congenital heart disease is the most common birth defect in newborns and children, which emphasizes the importance of heart development. In the early stages of development, the biomechanical environment, especially the radial strain, plays a crucial role in the morphological changes of the heart. The outflow tract (OFT) is an important part of the embryonic heart, and a large part of congenital heart defects originates from the OFT. However, it is still challenging to measure radial strain in real time in the OFT of animal models.

[0059] Using SD-OCT to collect images and extracting OFT boundaries through OFT-SegNet, finally measuring OFT wall radial strain in real time, including the following steps:

[0060] Step 1: Set up the SD-OCT imaging system: a broadband light source emits broadband light, which is coupled into a fiber-based Michelson interferometer through a fiber-optic circulator. The optical interference signal between the backscattered light from the probe and reference arms is transmitted to a spectrometer. Through the spectrometer, the interference spectrum is imaged into a line-scan camera (CCD), and the Doppler angle θ is quantified using a calibrated split head. This process ultimately obtains several position cardiac OFT structure maps.

[0061] The broadband light source is a broadband superluminescent diode (SLD) with a center wavelength of 1310 nm and a bandwidth of 52 nm, which can provide an axial resolution of about 14 μm when illuminating a sample exposed to air. The probe includes a collimator, an X-Y 2D galvanometer scanning system, and an objective lens (focal length 50 mm, lateral resolution 16 μm); the spectrometer consists of a collimating mirror (f = 5 mm), a 1145 line / mm transmission grating, and a Fourier lens (f = 100 mm); and the line-scan camera is a 1024-pixel infrared InGaAs.

[0062] Step 2: Build the OFT-SegNet network, which includes an encoder block, a feature fusion module, a decoder block, and a loss function.

[0063] The encoder mainly includes two parts, which are down-sampling operation and two consecutive Conv2D-BN-ReLU with a convolution kernel size of 3x3.

[0064] The feature fusion module includes a boundary enhancement branch and a boundary prediction branch. The boundary enhancement branch combines the mechanisms of channel attention and spatial attention. The boundary prediction branch draws on the application of long short-term memory network (LSTM) in natural language prediction and other fields. This module can use the positional relationship of blood vessels in different states to predict the edge blur or missing parts in the image due to the tailing effect.

[0065] The decoder includes an up-sampling layer and a convolution block composed of two convolution layers.

[0066] Step 3: Set the loss function for network training, which uses the sum of the Dice loss and the weighted binary cross-entropy loss.

[0067] Step 4: Preprocess and resize the structure map collected in Step 1 to a uniform size. In this example, images with a size of 512x512 can be used, but are not limited to this size. Use Amira and other labeling tools to perform pixel-level manual labeling, and the labeled images are used as segmentation labels, which are binary images.

[0068] Step 5: The original image and segmentation label are packaged into a dataset, and the network is iteratively trained until it converges, obtaining the trained OFT-SegNet network.

[0069] Step 6: Load the trained OFT-SegNet network weights and input the cardiac OFT structure image.

[0070] Step 7: First, perform distance transformation and skeletonization operations on the OFT boundary region extracted in step 6 to obtain the thickness information of the myocardial wall at different positions. Compare the thickness of the myocardial wall in different directions with the thickness of the myocardial wall in the same direction before contraction to obtain the radial strain value of the myocardial wall in different directions.

[0071] As shown in Figure 1 , the embodiment provides a real-time measurement method for cardiac outflow tract wall radial strain based on OFT-SegNet, which specifically includes the following steps:

[0072] S1, build an SD-OCT imaging system, as shown in Figure 2 : a broadband light source emits broadband light, which is coupled into a fiber-based Michelson interferometer through a fiber circulator. The optical interference signal between the backscattered light from the probe and the reference arm is transmitted to a spectrometer. Through the spectrometer, the interference spectrum is imaged into a line scan camera (CCD).

[0073] The broadband light source is a broadband superluminescent diode (SLD) with a center wavelength of 1310 nm and a bandwidth of 52 nm, which can provide an axial resolution of ~14 μm in air; the probe includes a collimator, an X-Y 2D galvanometer scanning system, and an objective lens (focal length 50 mm, lateral resolution 16 μm); the spectrometer is composed of a collimating mirror (f=5 mm), a 1145 line / mm transmission grating, and a Fourier lens (f=100 mm); the line scan camera is a 1024-pixel infrared InGaAs.

[0074] In SD-OCT, an axial information is collected in parallel, and the backscattering information of the sample is encoded in the interference spectrum, with different frequencies corresponding to different depths. Thus, by performing Fourier transform on the interference spectrum in wave number space, the backscattering distribution of the sample at different depths can be obtained:

[0075]

[0076] where A is the Fourier transform of the spectral light source S(k), represents convolution operation; is the coherence function curve, and the half-width of the curve is the coherence length, located near z=0; is the backscattering amplitude distribution of the sample at different depths, a(z) and a *(-z) symmetric about z = 0; Fourier transform of the coherence of the sample at different depths.

[0077] The process eventually obtains the cardiac OFT structure map of several positions.

[0078] S2, constructing an OFT-SegNet network, as Figure 3 The OFT-SegNet network includes an encoder block, a feature fusion module, a decoder block, and a loss function.

[0079] S2.1, similar to the U-Net structure, the encoder part mainly contains two parts, which are down-sampling operation and convolution block composed of two consecutive convolution layers. MaxPooling operation is used for down-sampling operation of the image, and convolution layer is used to extract the features of the image at different stages.

[0080] S2.2, a new module called feature fusion module is included in each encoder-decoder layer. Figure 4 The architecture of the feature fusion module is shown, which consists of two branches, including the boundary enhancement branch and the boundary prediction branch. The feature fusion module can be embedded in the connection of each layer of the encoder and decoder of the segmentation network. The feature fusion module embedded in the i-th layer fuses the boundary feature enhancement map generated by the boundary enhancement branch and the predicted sequence map generated by the boundary prediction branch. The final output feature map is generated. Alpha and beta are two trainable parameters that adaptively contribute to the boundary enhancement branch and the boundary prediction branch to drive the model to focus on more relevant representative parts in the input data.

[0081] S2.2.1, in order to better process the structure boundary information in the chicken embryo myocardial OCT image and improve the accuracy and robustness of segmentation, we introduce the SDA block at the connection part of the encoder and decoder of the model, as Figure 5The SDA block combines the mechanisms of channel attention and spatial attention. For chick embryo myocardial OCT images, the structure boundaries often have complex features and changes, so we need the network to dynamically learn the importance of different channels to better capture the relevance between different features in the image. The channel attention mechanism allows the network to adaptively adjust the importance of the channels according to the image content, so as to more accurately extract the feature information of the structure boundary. In addition, the structure boundary in the chick embryo myocardial OCT image is usually a local area, so the network needs to selectively focus on a specific area in the image to more accurately extract the structure boundary information. The spatial attention mechanism enables the network to focus on the local area of the image, thereby better capturing the detailed features of the boundary. By combining channel attention and spatial attention, the network better understands the structural features of the image and more effectively utilizes this information in the segmentation process, thereby significantly improving the segmentation performance and accuracy of the chick embryo myocardial OCT image.

[0082] S2.2.2, To solve the problem of blurred or missing boundaries in the image, we propose a boundary prediction module, inspired by the application of long short-term memory network (LSTM) in natural language prediction and other fields. This module can use the positional relationship of blood vessels in different states to predict the blurred or missing parts of the image due to the tailing effect. Although the blood vessel wall of the chick embryo heart is actually a complete structure, due to the tailing effect, it appears partially missing in the image. Considering the relative stability of the blood vessel wall structure of the chick embryo heart, we choose to introduce LSTM for prediction. By using the positional relationship of the blood vessels in the image, we can accurately predict the boundary position of the blood vessel wall.

[0083] Figure 6(a) The architecture of the boundary prediction module is shown. To address the problem of boundary blur or absence in chicken embryo heart images, we need to predict the location of the blood vessels based on their positional relationships. First, we use a convolutional layer to extract the low-dimensional features of the image. Then, we treat each row of the input image as an input sequence and divide the image into n input sequences, each containing information about the blood vessels at a certain location. In the case of boundary blur or absence, we can use the blood vessel location information from the previous input sequence to predict the blood vessel location information for the current input sequence. Next, we input each of the n input sequences into the prediction module, which generates the corresponding prediction results. In the prediction module, we make full use of the characteristics of the long short-term memory network (LSTM) to effectively process sequence data and retain previous information in different input sequences. Subsequently, we integrate the n prediction result sequences to obtain the final prediction results. This processing method can improve the prediction accuracy of the boundary blur or absence area based on the different positional relationships of the blood vessels. Finally, we use a convolutional layer to restore the image to its original size. Through this series of processing steps, we can predict the boundary blur or absence area in the image based on the known blood vessel location information, fill in the blank areas caused by flow voids, and improve the accuracy and completeness of the myocardial wall boundary.

[0084] Figure 6 (b) The architecture of the prediction module is shown. To enable the network to decide which information to retain, we designed a gate unit consisting of a convolutional block and a Sigmoid layer. This gate unit can output a value between 0 and 1, describing how much information each sequence should retain. When processing chicken embryo heart images, we can dynamically adjust the gate unit based on the degree of boundary blur to improve the prediction accuracy of the boundary area. By calculating the input gate, memory gate, and output gate, we can dynamically adjust the learning process of the network based on the characteristics of the input sequence, thereby obtaining more accurate prediction results. By utilizing previous information, we can gradually improve the prediction accuracy of the boundary blur area. The final prediction results can better reflect the characteristics of the chicken embryo heart images, improving the reliability and effectiveness of the prediction.

[0085] S2.3, The decoder block mainly contains three parts, which are upsampling layer, feature fusion module and convolution block composed of two consecutive convolution layers. The decoder is mainly used to gradually fuse the information provided by the feature fusion module.

[0086] S2.4, Set the loss function for network training, OFT-SegNet network training uses the combination of Dice loss L Dice and weighted binary cross-entropy loss L cross-entropy , which acts on the output end of the encoder;

[0087] Losstotal =L Dice +L cross-entropy

[0088]

[0089] S3. Load the pre-trained OFT-SegNet network weights, input the heart OFT structure map to extract the blood flow region in real time, and the boundary information of OFT.

[0090] S4. Based on OFT boundary information, by measuring the length changes of cardiomyocytes before and after contraction, and comparing these length changes with the length before contraction, the final radial strain value can be obtained. For example... Figure 7 As shown, we propose a semi-automatic computer program to calculate the radial strain of the myocardial wall in a conventional 2D image. The program takes a boundary map of the myocardial wall as input. First, we perform a distance transformation and skeletonization operation on the boundary map to obtain thickness information at different locations of the myocardial wall. In the distance transformation operation, we calculate the distance from each pixel in the binary image to the nearest boundary pixel, thus obtaining the thickness of the myocardial wall at different locations along the central axis of the image. Simultaneously, the skeletonization operation is used to extract the central axis from the binary image. Next, we perform a dot product operation between the distance matrix obtained from the distance transformation operation and the central axis matrix obtained from the skeletonization operation, thereby obtaining the thickness of the myocardial wall at different locations on the skeleton map. Then, we draw 360 straight lines based on the centroid of the skeleton map, and by calculating the pixel size at the intersection of the skeleton and the straight lines, we obtain the thickness of the myocardial wall in the current direction. Finally, we compare the thickness of the myocardial wall in different directions with the thickness of the myocardial wall in the same direction before contraction to obtain the radial strain values ​​of the myocardial wall in different directions. To more intuitively display the changes in the myocardial wall, we will present the calculated radial strain values ​​as a pseudo-color image.

[0091] The embodiments of the real-time measurement method of the cardiac outflow tract wall radial strain based on the neural network can be applied to any device with data processing capability, which can be a device or apparatus such as a computer. The device embodiments can be implemented by software, or by hardware or a combination of software and hardware. For the device embodiments, since they basically correspond to the method embodiments, the related parts are described with reference to the parts of the method embodiments. The device embodiments described above are only illustrative, and the units described as separate components can or can not be physically separated, and the components displayed as units can or can not be physical units, that is, they can be located in one place or distributed on multiple network units. Some or all of the modules can be selected to achieve the purpose of the present application according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0092] Supplementary experiment: Since the chicken embryo at the Hamburger-Hamilton (HH) stage 18 is relatively easy to obtain and has a fast development speed compared with most other animal models, the chicken embryo heart at HH stage 18 is used in this example to perform additional experiments to prove the method. The preparation of the chicken embryo follows the standard procedure: fertilized white Leghorn eggs are cultured in a rotating incubator at 38 degrees Celsius and 85% humidity for about 3 days. After opening the eggshell at the air sac end, a small piece of inner shell membrane is carefully removed. Then the egg is placed in a specially designed acrylic box. During image acquisition, the temperature in the acrylic box is maintained at 37.5 degrees Celsius by a heating blanket. Figure 8 Radial strain changes of the chicken embryo heart used in the examples of the present application at different stages of the beating cycle. Figure 9 3D view of the radial strain test results of the chicken embryo heart used in the examples of the present application at different stages of the beating cycle. (a) is the structure of the M section, (b) is the calculated radial strain corresponding to (a), (c) is the 3D view of the OFT boundary change over time, and (d) is the calculated radial strain change corresponding to (c).

Claims

1. A method for real-time measurement of cardiac outflow tract wall radial strain based on neural networks, characterized in that, The specific process is as follows: An SD-OCT imaging system is built to collect images and obtain structural images; The OFT-SegNet network is used to extract the boundary of the cardiac outflow tract in the structural image; then, based on the boundary information, the length change of the myocardial cells before and after contraction is measured, and the length change is compared with the length before contraction, so that the final radial strain value is obtained; The OFT-SegNet network comprises an encoder block, a feature fusion module, a decoder block and a loss function; the input of the OFT-SegNet network is a structural image, the features of images at different stages are extracted through the encoder block, and the features of images at different stages are input into the decoder; the feature fusion module is inserted in each layer of the encoder-decoder, and the final output feature mapping is generated through the feature fusion module; finally, the information from the feature fusion module is gradually fused through the decoder, and the extracted boundary region is output at the last stage; The cardiac outflow tract wall radial strain is calculated as follows: A semi-automatic computer program is designed to calculate the radial strain of the myocardial wall in a traditional 2D image; the program takes the boundary map of the myocardial wall as input; first, the distance transformation operation and skeletonization operation are performed on the boundary map of the myocardial wall to obtain the thickness information of the myocardial wall at different positions; in the distance transformation operation, the distance from each pixel point in the binary image to the nearest boundary pixel point is calculated, so that the thickness of the myocardial wall at different positions can be obtained at the central axis of the image; at the same time, the skeletonization operation is used to extract the central axis from the binary image; next, the distance matrix obtained by the distance transformation operation is multiplied with the central axis matrix obtained by the skeletonization operation, so that the thickness of the myocardial wall at different positions is obtained on the skeleton map; then, based on the center of gravity of the skeleton map, 360 straight lines are drawn, and the size of the pixels at the intersection of the skeleton and the straight line is calculated, so that the thickness of the myocardial wall in the current direction is obtained; finally, the thickness of the myocardial wall in different directions is compared with the thickness of the myocardial wall in the same direction before contraction, and the radial strain value of the myocardial wall in different directions is obtained.

2. The method of claim 1, wherein the method is a real-time measurement method of cardiac outflow tract wall radial strain based on a neural network. The encoder part mainly comprises two parts, namely a downsampling operation and a convolution block composed of two consecutive convolution layers; the MaxPooling operation is used for downsampling operation of the image, the receiving domain is expanded, and the convolution layer is used to extract the features of images at different stages and input the feature fusion module.

3. The method of claim 2, wherein the neural network is trained using a plurality of training data sets, each training data set comprising a plurality of cardiac outflow tract wall radial strain measurements and a corresponding plurality of cardiac outflow tract wall radial strain values. The feature fusion module comprises a boundary enhancement branch and a boundary prediction branch. The boundary enhancement branch combines the mechanisms of channel attention and spatial attention; for the chick embryo myocardial OCT image, the structural boundary often has complex features and changes, so the network needs to dynamically learn the importance of different channels to better capture the correlation between different features in the image. The channel attention mechanism allows the network to adaptively adjust the importance of the channels according to the image content, so as to more accurately extract the feature information of the structure boundary; in addition, the structure boundary in the chick embryo myocardial OCT image is usually a local area, so the network needs to selectively focus on a specific area in the image to more accurately extract the structure boundary information; the spatial attention mechanism enables the network to selectively focus on the local area of the image, thereby better capturing the detailed features of the boundary. By combining channel attention and spatial attention, the network better understands the structural features of the image and more effectively utilizes these information in the segmentation process, thereby significantly improving the segmentation performance and accuracy of the chick embryo myocardial OCT image. The boundary prediction branch aims to solve the problem of fuzzy or missing boundaries in the chick embryo heart image. The blood vessel position relationship is used for prediction. First, the low-dimensional features of the image are extracted using a convolutional layer. Then, each row of the input image is treated as an input sequence, and the image is divided into n input sequences, each of which contains information about the position of the blood vessel. In the case of fuzzy or missing boundaries, the blood vessel position information in the previous input sequence is used to predict the blood vessel position information in the current input sequence. Next, the n input sequences are input into the prediction module one by one, which will generate the corresponding prediction results. In the prediction module, the characteristics of the long short-term memory network (LSTM) are fully utilized to effectively process sequence data and retain previous information in different input sequences. Then, the n prediction result sequences are integrated to obtain the final prediction result. This processing method can improve the prediction accuracy of fuzzy or missing boundary areas according to the different position relationships of the blood vessels. Finally, the convolutional layer is used again to restore the image to its original size. Through a series of processing steps, the fuzzy or missing boundary areas in the image can be predicted based on the known blood vessel position information, filling in the blank areas caused by the flow void problem, and improving the accuracy and completeness of the myocardial wall boundary.

4. The method of claim 3, wherein the neural network is trained using a plurality of training data sets, each training data set comprising a plurality of cardiac outflow tract wall radial strain measurements and a corresponding plurality of cardiac outflow tract wall radial strain values. The decoder block mainly includes three parts, namely the upsampling layer, the feature fusion module, and the convolution block composed of two consecutive convolutional layers. The decoder is mainly used to gradually fuse the information provided by the feature fusion module.

5. The method of claim 4, wherein, The OFT-SegNet network performs boundary segmentation of the cardiac outflow tract, and the OFT-SegNet network training uses a combination of Dice loss L Dice and weighted binary cross-entropy loss L cross-entropy acting on the output of the encoder. Loss total = L Dice + L cross-entropy

Images