Method of tracking elongated objects in image data and image processing system

Abstract

Methods and systems for tracking elongated objects in image data are disclosed. Methods and systems are operable, for example, for neurite tracking. A turn prediction is inferred using an artificial intelligence model. The turn prediction is processed to determine a next point of the series of points (60) on the centerline (50) of the elongate object.

Description

Technical Field

[0001] The present invention relates to image processing methods and systems.Embodiments of the present invention relate to methods and systems for tracking elongated objects, such as neurites, in image data representing a three-dimensional tissue volume. Background Art

[0002] Dense mapping of neuronal circuits and their synaptic connections to enhance understanding of computation in the brain is a major goal of high-resolution connectomics. 3 3D-EM datasets of this size are feasible, but axonal and spinal neck reconstructions remain considerable challenges.

[0003] Manual annotation of 3D image data can provide low error rates, but is often prohibitively time-consuming. Therefore, there is a need for computer-assisted or computer-implemented techniques that can track elongated objects in image data representing three-dimensional (3D) tissue volumes. By reconstructing neurites along sparse trajectories of their centerlines ("skeletonization"), annotation speed is increased by about 50 times (see, e.g., M. Helmsteder et al., "High-Accuracy Neurite 593 Reconstruction for High-Throughput Neuroanatomy", Nature Neuroscience 14(8): 594 1081–88, 2011). The approach is based on a recent large-scale reconstruction (see, e.g., K. Eichler et al., “The Complete Connectome of a 579Learning and Memory Centre in an Insect Brain.” Nature, 548(7666):175–82, 2017). Skeleton reconstruction has been combined with automatically derived segmentations to produce volume reconstructions (see, e.g., Helmsteder et al., “Connectomic Reconstruction of the Inner Plexiform Layer in the Mouse Retina.”, 2013, Nature 500(7461):168–74; Dorkenwald et al., “Automated Synaptic Connectivity Inference for Volume Electron Microscopy.”, 2017, Nature Methods 14(4):435–42; and Staffler et al., “SynEM, Automated Synapse Detection for Connectomics.”, 2017, ELife 6(July):e26414).

[0004] Additional speedup is achieved by skeletonizing neurites using a user interface in which the annotator is presented with an egocentric view of the 3D-EM volume (“fly mode”) (see, e.g., Boergens et al., “WebKnosso: Efficient Online 3D Data Annotation for Connectomics.” Nature Methods 14(7):691–94, 2017). Other approaches aim to reduce the number of locations that require human annotation through automatic reconstruction of neurites and subsequent detection of reconstruction errors (SM Plaza, “Focused Proofreading to Reconstruct Neural Connectomes from EM Images at Scale,” Springer, 2016, 249–58; Cham.; Y. Meirrovitch et al., “A Multi-Pass Approach to Large 620 Scale Connectomics,” ArXiv E-Print, December, 2016, http: / / arxiv.org / abs / 1612.02120; D. Rolnick et al., ArXiv E-Print, December, 2017). E-Print. https: / / arxiv.org / pdf / 1705.10882.pdf, “Morphological Error Detection in 3D Segmentations.”, J. Zung et al., “An Error Detection and Correction Framework for Connectomics,” NIPS, 6818-29, 2017; K. Dmitriev et al., British Machine Vision Conference (BMVC), 2018, http: / / bmvc2018.“Efficient Correction for EM Connectomics with Skeletal Representation”, published in 2019, IEEE Transactions on Computer Science, 2013, 36(5), 115(4), 123(3), 16(5), 18(6), 19(9), 20(11), 21(12), 22(13), 23(14), 24(15), 25(16), 26(17), 27(18), 28(19), 29(20), 31(21), 32(22), 33(23), 34(24), 35(25), 36(26), 37(27), 38(28), 39(30), 40(41), 41(5), 42(6), 43(7), 44(8), 45(9), 46(11), 47(12), 48(13), 49(14), 50(15), 51(16), 52(17), 53(17), 54(18), 55(19), 56(11), 57(11), 58(11), 59(11), 60(11), 61(11), 62(11), 63(11), 64(11), 65(11), 67(11), 68(11), 69(11), 70(11), 71(11), 72(11), 73(11), 74(11), 75(11), 76(11), 77(11), 78(11), 79(11).

[0005] Additional exemplary techniques for tracking objects are disclosed in: Jelmer M. Wolterrink et al., "Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier", MEDICAL IMAGE ANALYSIS, OXFORD UNIVERSITY PRESS, OXOFRD, GB (Medical Image Analysis, Oxford University Press, Oxford, UK), vol. 51, October 22, 2018, pp. 46-60, ISSN: 1361-8415, DOI: 10.1016 / J.MEDIA.2018.10.005; Rouchen Gao et al., 2021 IEEE 18TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL "Joint Coronary Centerline Extraction And Lumen Segmentation From Ccta Using Cnntracker And VascularGraph Convolutional Network" by Tianhong Dai et al. in Proceedings of Machine Learning Research, 2019, pp. 130-150, DOI: 10.1109 / ISBI4821120219433764. http: / / proceedings.mlr.press / v102 / dai19a / dai19a.pdf "Deep Reinforcement Learning for Subpixel Neural Tracking" published in the journal IEEE Transactions on Computer Science. Summary of the invention

[0006] There remains a need for enhanced image processing techniques. In particular, there remains a need for techniques that can track elongated objects (eg, neurites) in a computationally efficient manner and / or with a low error rate.

[0007] According to embodiments of the invention, a method and a system are provided as described in the independent claims. The dependent claims define preferred embodiments.

[0008] A method for tracking an elongated object in image data, the image data being biological or medical image data, the method may include: determining, by an image processing system, positions of a series of points along the elongated object in a three-dimensional 3D space to track the elongated object in 3D in the image data. The series of points may be determined so that they are approximately located along a centerline of the elongated object.

[0009] Determining the positions of a series of points may involve an iterative process.

[0010] The iterative process can include using an artificial intelligence (AI) model to infer a steering prediction that depends on (and can indicate) the direction in which the elongated object (e.g., a centerline or an approximation of the centerline) extends from a previously determined point along the elongated object (e.g., along the centerline or an approximation of the centerline), the AI ​​model comprising an input layer and an output layer, the input layer being operable to receive an AI model input based on pixel or voxel values ​​of image data, the output layer being operable to provide an AI model output that is a steering prediction or indicates a steering prediction in each iteration of the iterative process.

[0011] The iterative process may include using at least the steering prediction to calculate a next point in the series of points that is subsequent to a previously determined point in the series of points.

[0012] The iterative process may include iteratively repeating the inference step and the calculation step until a termination criterion is met, thereby determining a series of points.

[0013] According to a preferred embodiment, the AI ​​model may include a convolutional neural network (CNN), the convolutional neural network including an input layer, multiple convolutional layers, one or more dropout layers, and an output layer, or a recurrent neural network (RNN), wherein the AI ​​model input to the input layer in an iteration of the iterative process represents a sub-volume of the tissue volume represented by the image data, wherein the AI ​​model input includes pixel or voxel values ​​representing the sub-volume of the image data, or is obtained by multivariate interpolation of pixel or voxel values ​​representing the sub-volume of the image data, wherein the orientation of the sub-volume depends on the steering prediction determined in a previous iteration of the iterative process.

[0014] According to a preferred embodiment, the steering prediction may depend on a second order derivative of the elongated object, for example a centre line of the elongated object.

[0015] According to another preferred embodiment, the AI ​​model can be a convolutional neural network (CNN) comprising an input layer, multiple convolutional layers, one or more dropout layers, and an output layer, or a recurrent neural network (RNN), wherein the AI ​​model input to the input layer in an iteration of the iterative process represents a sub-volume of the tissue volume represented by the image data, the steering prediction is a curvature value relative to a reference system associated with the sub-volume so that it produces a position offset, which is used to iteratively determine a series of points located on the centerline of the slender object, and in each iteration after the previous iteration, the corresponding sub-volume being processed is determined to have a rotational orientation determined by the extension direction of the slender object, which extension direction is determined by the steering prediction obtained in the previous iteration.

[0016] The method may comprise outputting the result of the image processing via a graphical user interface or via a data interface.

[0017] Determining a next point in the series of points may include integrating the steering prediction.

[0018] The steering prediction may depend on the second order derivative of the centerline of the elongated object.

[0019] The AI ​​model input to the input layer in an iteration of the iterative process may represent a sub-volume of the tissue volume represented by the image data.

[0020] The subvolume may depend on a previous turn prediction determined in a previous iteration of the iterative process and / or a point that has been determined in a previous iteration of the iterative process.

[0021] The orientation of the subvolume may depend on previous steering predictions and / or may depend on the uncertainty of the AI ​​model's predictions at multiple orientations.

[0022] The orientation of the sub-volume may depend on the direction in which the elongated object extends at or within the sub-volume.

[0023] The orientation of the sub-volume may be approximately aligned with the direction in which the elongated object extends at or within the sub-volume.

[0024] The center of the subvolume may be offset relative to a previously determined point by an offset, further optionally wherein the offset depends on a steering prediction determined in a previous iteration.

[0025] The method may include performing multivariate interpolation (e.g., averaging) of pixel or voxel values ​​of the image data to determine the AI ​​model input.

[0026] Multivariate interpolation may include weighted averaging of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0027] Multivariate interpolation can include trilinear projection.

[0028] The method may include performing an initialization of an iterative process.

[0029] Initialization may depend on whether an initial orientation is available a priori.

[0030] When the initial orientation is not available, the method may include determining the initial orientation for an initial iteration of the iterative process.

[0031] Determining the initial orientation may include determining uncertainty estimates of the AI ​​model predictions for each orientation, and selecting an orientation with the smallest uncertainty estimate from among the orientations.

[0032] Uncertainty estimates can be determined using the Monte Carlo dropout method.

[0033] The AI ​​model may include a convolutional neural network (CNN).

[0034] The AI ​​model may include a recurrent neural network (RNN).

[0035] The AI ​​model may include: an input layer, multiple convolutional layers, one or more dropout layers, and an output layer.

[0036] The output layer can be a linear layer.

[0037] The multiple convolutional layers may include one or several strided convolutional layers.

[0038] The AI ​​model may be or may include a fully convolutional network.

[0039] The AI ​​model may include one or more of the following: pooling layers, attention modules, self-attention modules, dilated convolutions, normalization layers, batch normalization layers, transformers, and recurrent layers.

[0040] An AI model may include one or more nonlinearities.

[0041] The AI ​​model may include an exponential linear unit (ELU).

[0042] The AI ​​model may include a rectified linear unit (ReLU).

[0043] The AI ​​model may include a scaled exponential linear unit (SELU).

[0044] The AI ​​model may include a Gaussian Error Linear Unit (GELU).

[0045] The image data may represent a volumetric representation of tissue.

[0046] The image data may include a three-dimensional (3D) image or a set of two-dimensional (2D) images.

[0047] The method may also include acquiring image data.

[0048] Electron microscopy can be used to acquire image data.

[0049] Image data may be acquired using an optical microscope (eg, a light microscope).

[0050] The image data may be acquired using positron emission tomography (PET), magnetic resonance (MR) imaging, X-ray tomography, or another medical imaging modality.

[0051] The elongated object may be a neurite.

[0052] The elongated object may be a blood vessel.

[0053] The elongated object may have a surface that extends in a partially cylindrical manner about a centerline.

[0054] Methods can be used for axon tracing.

[0055] The elongated object may be an axon.

[0056] The elongate object may be a spinal neck.

[0057] The method can be performed multiple times in parallel and / or sequentially to trace multiple axons and / or dendrites.

[0058] The method may include training an AI model.

[0059] AI models can be trained using supervised learning.

[0060] Training the AI ​​model may include gradient-based updates of AI model parameters.

[0061] Training the AI ​​model may include training the AI ​​model to conform to a flight strategy that causes the iterative process to converge to an elongated object (e.g., along the centerline or an approximation of the centerline) when starting from an initial position off the centerline in the image data.

[0062] The convergence distance of the flight strategy determines how quickly the iterative process converges back to the centerline.

[0063] The convergence distance can be a dynamic parameter that changes during AI model training and / or inference.

[0064] During AI model training, the convergence distance can be set as a function of the distance from the physiological boundary.

[0065] The convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0066] The method may include outputting one or more of: a visual representation of a series of points or a centerline; a machine-readable representation of a series of points or a centerline; and a control signal dependent on the series of points positioned along the centerline; and an uncertainty in the predicted control signal; and a representation of a stopping criterion for ending the iterative process.

[0067] Outputting may be performed via a user interface such as a graphical user interface or via a data interface. A medical imaging method or a medical image processing method, comprising: receiving medical image data representing a tissue volume and comprising a 3D image or a set of 2D images containing neural tissue, performing an image processing method to track an elongated object in the medical image data, and outputting one or more of: a visual representation of a series of points or a centerline; a machine readable representation of the series of points or the centerline; a control signal dependent on the series of points located along the centerline; a machine learning (ML) model having an architecture dependent on the series of points located along the centerline.

[0068] The connection group method includes: performing a method for tracking elongated objects.

[0069] A method for analyzing a biopsy sample may include: acquiring image data of the biopsy sample; processing the image data using a method for processing image data; and outputting a result of the image data processing via an interface.

[0070] Methods of analyzing a biopsy sample can be performed to determine a portion of the connectome.

[0071] A biopsy sample is a sample that has not been implanted into an animal or human.

[0072] A method for generating signal processing logic for controlling a physical asset may include: performing a method for processing image data, selecting or generating a machine learning (ML) model architecture using output of an elongated object tracking method by at least one computing system, training a ML model having the ML model architecture, and deploying the trained ML model as signal processing logic to a control device for execution.

[0073] An image processing system for tracking an elongate object in image data may include an interface operable to receive image data, the image data being biological or medical image data.

[0074] The image processing system may include at least one processing device operable to determine positions of a series of points along an elongated object (e.g., along a centerline or an approximation of a centerline) in three-dimensional 3D space to track the elongated object in 3D in the image data, wherein determining the positions of the series of points may include an iterative process.

[0075] The iterative process may include using an artificial intelligence (AI) model to infer a steering prediction that depends on (and may indicate) the direction in which the slender object extends from a previously determined point along the slender object, the AI ​​model comprising an input layer and an output layer, the input layer being operable to receive an AI model input based on pixel or voxel values ​​of image data, the output layer being operable to provide an AI model output that is a steering prediction or indicates a steering prediction in each iteration of the iterative process.

[0076] The iterative process may include using the steering prediction to calculate a next point in the series of points that is after a previously determined point in the series of points.

[0077] The iterative process may include iteratively repeating the inference step and the calculation step until a termination criterion is met, thereby determining a series of points.

[0078] According to a preferred embodiment, the image processing system can be operated such that the AI ​​model may include a convolutional neural network (CNN), the convolutional neural network (CNN) including an input layer, multiple convolutional layers, one or more dropout layers, and an output layer, or a recurrent neural network (RNN), wherein the AI ​​model input to the input layer in an iteration of the iterative process represents a sub-volume of the tissue volume represented by the image data, wherein the AI ​​model input includes pixel or voxel values ​​representing the sub-volume of the image data, or is obtained by performing multivariate interpolation of the pixel or voxel values ​​representing the sub-volume of the image data, wherein the orientation of the sub-volume depends on the steering prediction determined in a previous iteration of the iterative process.

[0079] According to a preferred embodiment, the steering prediction may depend on a second order derivative of the elongated object, for example a centre line of the elongated object.

[0080] According to another preferred embodiment, the image processing system can be operated so that the AI ​​model can be a convolutional neural network (CNN) comprising an input layer, multiple convolutional layers, one or more dropout layers, and an output layer, or a recurrent neural network (RNN), wherein the AI ​​model input to the input layer in an iteration of the iterative process represents a sub-volume of the tissue volume represented by the image data, the steering prediction is a curvature value relative to a reference system associated with the sub-volume, so that it produces a position offset for iteratively determining a series of points located on the centerline of the slender object, and in each iteration following the previous iteration, determining the corresponding sub-volume being processed to have a rotational orientation determined by the extension direction of the slender object, and the extension direction is determined by the steering prediction obtained in the previous iteration.

[0081] The image processing system may be operable to perform the method according to the embodiments.

[0082] The image processing system may include an image acquisition system operable to capture image data.

[0083] The image acquisition system is operable to perform 3D electron microscopy or light microscopy.

[0084] The image processing system may be operable to output the results of the image processing via a graphical user interface or via a data interface.

[0085] The image processing system may be operative such that determining the next point in the series of points may comprise integrating the steering prediction.

[0086] The image processing system may be operated such that the AI ​​model input to the input layer in an iteration of the iterative process may represent a sub-volume of the tissue volume represented by the image data.

[0087] The image processing system may be operated such that the sub-volume may depend on a previous steering prediction determined in a previous iteration of the iterative process and / or a point that has been determined in a previous iteration of the iterative process.

[0088] The image processing system may be operated such that the orientation of the sub-volume may depend on a previous steering prediction.

[0089] The image processing system may be operative such that the orientation of the sub-volume may depend on the direction in which the elongate object extends at or within the sub-volume.

[0090] The image processing system may be operated such that the orientation of the sub-volume may be approximately aligned with the direction in which the elongate object extends at or within the sub-volume.

[0091] The image processing system may be operated such that the centre of the sub-volume may be offset by an offset relative to a previously determined point.

[0092] The image processing system may be operated such that the amount of the offset may depend on the steering prediction determined in a previous iteration.

[0093] The image processing system can be used to multivariately interpolate (e.g., average) pixel or voxel values ​​of image data to determine AI model inputs.

[0094] The image processing system can be operable such that multivariate interpolation can include weighted averaging of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0095] The image processing system may be operable to perform the projection to determine pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0096] The image processing system may be operable to perform an initialization of the iterative process.

[0097] The image processing system may be operated such that initialization may depend on whether an initial steering prediction is available a priori.

[0098] The image processing system may be operative such that, when the initial orientation is not available, the image processing system determines the initial orientation for an initial iteration of the iterative process.

[0099] The image processing system may be operative such that determining the initial orientation may comprise determining uncertainty estimates for each orientation and selecting the orientation from the various orientations for which the uncertainty estimate is minimized.

[0100] The image processing system may be operative such that the uncertainty estimate may be determined using a Monte Carlo dropout method.

[0101] The AI ​​model may include a convolutional neural network (CNN).

[0102] The AI ​​model may include a recurrent neural network (RNN).

[0103] An AI model may include an input layer, multiple convolutional layers, one or more dropout layers, and an output layer.

[0104] The output layer can be a linear layer.

[0105] The multiple convolutional layers may include one or several strided convolutional layers.

[0106] The AI ​​model may be or may include a fully convolutional network.

[0107] The AI ​​model may include one or more of the following: pooling layers, attention modules, self-attention modules, dilated convolutions, normalization layers, batch normalization layers, transformers, and recurrent layers.

[0108] The AI ​​model may include an exponential linear unit (ELU).

[0109] The AI ​​model may include a rectified linear unit (ReLU).

[0110] The AI ​​model may include a scaled exponential linear unit (SELU).

[0111] The AI ​​model may include a Gaussian Error Linear Unit (GELU).

[0112] The image data may represent a volumetric representation of tissue.

[0113] The image data may include a 3D image or a collection of 2D images.

[0114] The image processing system is operable to acquire image data.

[0115] Electron microscopy can be used to acquire image data.

[0116] Image data may be acquired using an optical microscope, such as a light microscope.

[0117] The image data may be acquired using positron emission tomography (PET), magnetic resonance (MR) imaging, or another medical imaging modality.

[0118] The elongated object may be a neurite.

[0119] The elongated object may be a blood vessel.

[0120] The elongated object may have a surface that extends in a partially cylindrical manner about a centerline.

[0121] The image processing system is operable to train the AI ​​model.

[0122] AI models can be trained using supervised learning.

[0123] The image processing system may be operated such that training the AI ​​model may include: gradient-based updating of AI model parameters.

[0124] The image processing system may be operated such that training the AI ​​model may include training the AI ​​model to conform to a flight strategy that causes the iterative process to converge to the centerline when starting from an initial position in the image data that is off the centerline.

[0125] The image processing system may be operated such that the convergence distance of the flight strategy may determine the speed at which the iterative process converges back to the centerline.

[0126] The image processing system can be operated so that the convergence distance can be a dynamic parameter that varies during AI model training.

[0127] The image processing system can be operated so that the convergence distance can be set as a function of the distance from the physiological boundary during AI model training.

[0128] The image processing system may be operated such that the convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0129] The image processing system may be operable to output one or more of: a visual representation of a series of points or a centerline; a machine-readable representation of the series of points or the centerline; and a control signal dependent upon the series of points being positioned along the centerline.

[0130] The image processing system may include a user interface, such as a graphical user interface, or a data interface to perform output.

[0131] Also disclosed is a training method for training an AI model, the AI ​​model comprising an input layer and an output layer, wherein the input layer is operable to receive an AI model input based on pixel or voxel values ​​of image data representing a tissue volume, and the output layer is operable to provide an AI model output that is or indicates a steering prediction, wherein the steering prediction depends on (and can indicate) the direction along which the slender object (e.g., along a centerline or an approximation of a centerline) extends.

[0132] The training method may include training the AI ​​model with the goal of predicting the direction along which the centerline of the elongated object extends.

[0133] The training method may include training the AI ​​model with the goal of outputting a derivative (e.g., a first, second, or higher order derivative) of the direction along which the centerline of the elongated object extends.

[0134] The training method can be a gradient-based supervised learning method.

[0135] In this training method, the AI ​​model can be trained using supervised learning.

[0136] Training methods may include gradient-based updates of AI model parameters.

[0137] The training method may include training the AI ​​model to conform to a flight strategy that causes the iterative process to converge to the centerline when starting from or reaching a location away from the centerline in the image data.

[0138] The convergence distance of the flight strategy determines how quickly the iterative process converges back to the centerline.

[0139] The convergence distance can be a dynamic parameter that changes during AI model training and / or inference.

[0140] The convergence distance may depend on the distance to the obstacle.

[0141] The barrier may be a physiological boundary, such as a cell boundary.

[0142] During AI model training, the convergence distance can be set as a function of the distance from the physiological boundary.

[0143] The convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0144] The AI ​​model inputs to the input layer may represent sub-volumes of the tissue volume represented by the training data. The training method may include multivariate interpolation of pixel or voxel values ​​of the image data to determine the AI ​​model inputs.

[0145] Multivariate interpolation may include weighted averaging of pixel or voxel values ​​to generate an AI model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0146] Among the training methods, the AI ​​model may include a convolutional neural network (CNN).

[0147] In the training method, the AI ​​model may include a recurrent neural network (RNN).

[0148] In the training method, the AI ​​model may include an input layer, multiple convolutional layers, one or more dropout layers, and an output layer.

[0149] In the training method, the output layer can be a linear layer.

[0150] In the training method, the multiple convolutional layers may include one or several strided convolutional layers.

[0151] In the training method, the AI ​​model can be or include a fully convolutional network.

[0152] In the training method, the AI ​​model may include one or more of the following: pooling layer, attention module, self-attention module, void convolution, normalization layer, batch normalization layer, converter, recurrent layer.

[0153] In the training method, the AI ​​model may include one or more nonlinearities.

[0154] In the training method, the AI ​​model may include an exponential linear unit (ELU).

[0155] In the training method, the AI ​​model may include a rectified linear unit (ReLU).

[0156] In the training method, the AI ​​model may include a scaled exponential linear unit (SELU).

[0157] In the training method, the AI ​​model may include a Gaussian Error Linear Unit (GELU).

[0158] In the training method, the image data may represent a volumetric representation of the tissue.

[0159] In the training method, the image data may include 3D images or a collection of 2D images.

[0160] In the training method, the image data may be electron microscope (EM) images.

[0161] In the training method, the image data may be 3D EM images.

[0162] In the training method, the image data may be tomographic images.

[0163] In the training method, the image data may be optical microscopy (eg, light microscopy) data.

[0164] In the training method, the image data may be positron emission tomography (PET), magnetic resonance (MR) imaging, or other medical image data.

[0165] In the training method, the elongated object on which the AI ​​model is trained can be a neurite.

[0166] In this training method, the elongated object on which the AI ​​model is trained may be a blood vessel.

[0167] In this training method, the elongated object on which the AI ​​model is trained may have a surface that extends around a centerline in a partially cylindrical manner.

[0168] According to another embodiment, a method of tracking an object in image data is provided. The method includes determining a series of points along the object to track the object in the image data.

[0169] Determining a series of points may include an iterative process. The iterative process may include:

[0170] inferring a steering prediction using an artificial intelligence (AI) model, the AI ​​model comprising an input layer and an output layer, the input layer being operable to receive an AI model input based on pixel or voxel values ​​of the image data, the output layer being operable to provide an AI model output being or indicating the steering prediction at each iteration of the iterative process;

[0171] using at least the turn prediction to calculate a next point in the series of points, the next point being after a previously determined point in the series of points; and

[0172] The inference step and the computation step are repeated iteratively until a termination criterion is met.

[0173] The image data may be N-dimensional image data, where N is an integer. N may be at least 2, at least 3, or at least 4.

[0174] Objects may be tracked in ND dimensions, where D is an integer equal to or greater than zero and less than N.

[0175] One of the dimensions of the image data may correspond to a time axis.

[0176] Determining the next point in the series of points may include integrating the steering prediction.

[0177] Integrating the steering prediction may include integrating the curvature in the Bishop reference frame.

[0178] The steering prediction is related to the second-order derivative of the centerline of the object.

[0179] The AI ​​model input to the input layer in an iteration of the iterative process may represent a sub-volume of image data.

[0180] The sub-volume may depend on a position and / or orientation and / or a turning prediction determined in one or several previous iterations of the iterative process.

[0181] The orientation of the subvolume may depend on previous steering predictions.

[0182] The center of the subvolume may be offset by an offset relative to a previously determined point.

[0183] The amount of the offset may depend on the steering prediction determined in a previous iteration.

[0184] The method may also include multivariate interpolation of pixel or voxel values ​​of the image data to determine the AI ​​model input.

[0185] Multivariate interpolation may include weighted averaging of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0186] Multivariate interpolation can include multilinear projection.

[0187] The method may also include performing an initialization of the iterative process.

[0188] Initialization may depend on whether an initial orientation is available.

[0189] When the initial direction is not available, the method may include determining the initial direction for an initial iteration of the iterative process.

[0190] Determining the initial directions may include determining uncertainty estimates for the predictions of the respective directions.

[0191] Determining the initial direction may include selecting, from the various directions, a direction and a steering prediction for which an uncertainty estimate in a corresponding prediction is minimized.

[0192] Uncertainty estimates can be determined using the Monte Carlo dropout method.

[0193] AI models can include convolutional neural networks (CNN) or recurrent neural networks (RNN).

[0194] The AI ​​model may include an input layer, multiple convolutional layers, one or more dropout layers, and an output layer.

[0195] The AI ​​model may include one or more of the following: pooling layers, attention modules, self-attention modules, dilated convolutions, normalization layers, batch normalization layers, transformers, and recurrent layers.

[0196] One, several or all of the following may apply: the output layer is a linear layer; the multiple convolutional layers include one or more strided convolutional layers; the AI ​​model is or includes a fully convolutional network; the AI ​​model includes one or more nonlinearities; the AI ​​model includes an exponential linear unit (ELU); the AI ​​model includes a rectified linear unit (ReLU); the AI ​​model includes a scaled exponential linear unit (SELU); the AI ​​model includes a Gaussian error linear unit (GELU).

[0197] The object may be an object extending along the time axis of the image data.

[0198] The object may be an object in a video.

[0199] The object may be an object to be tracked through a series of image frames.

[0200] The subject may be a biological subject, in particular a physiological subject.

[0201] The subject may be a human or animal neurite.

[0202] The subject may be a human or animal blood vessel.

[0203] Objects can be infrastructure assets, especially utilities, systems.

[0204] The object may be a pipe of a fresh water or sewage system.

[0205] The object could be a pipe in an industrial plant.

[0206] An object may have a surface that extends in a partially cylindrical manner around a central axis of the object.

[0207] The method may also include training the AI ​​model.

[0208] AI models can be trained using supervised learning.

[0209] Training the AI ​​model may include gradient-based updates of AI model parameters.

[0210] Training the AI ​​model may include training the AI ​​model to conform to a flight strategy that causes the iterative process to converge to the object when starting from or reaching a location away from the object in the image data.

[0211] The convergence distance of the flight strategy determines how quickly the iterative process converges back to the object.

[0212] The convergence distance can be a dynamic parameter that can vary during AI model training.

[0213] The convergence distance can be set during AI model training as a function of the distance from the boundary represented by the image data.

[0214] The convergence distance may be a monotonically increasing function of the distance from the boundary.

[0215] The image processing method may include receiving and / or acquiring image data and performing an object tracking method.

[0216] The image processing method may include outputting one or more of: a visual representation of a series of points or a centerline; a machine-readable representation of the series of points or the centerline; a control signal, the control signal being dependent on the series of points located along the centerline; a machine learning (ML) model having an architecture that is dependent on the series of points located along the centerline.

[0217] The image data may comprise a time series of time-sequential image frames.

[0218] Tracking the object may include tracking the physical object along a time axis in a space-time coordinate system.

[0219] A control method comprises an object tracking method or an image processing method, and further comprises: controlling at least one controllable asset based on a result of the object tracking method.

[0220] Also disclosed is a processing system comprising an interface for receiving image data and at least one circuit (eg, a set of processors or other integrated circuits) operable to perform an object tracking method or an image processing method.

[0221] Also disclosed is a system that includes at least one controllable asset (eg, a machine or actuator) and a processing system, wherein the controllable asset responds to control commands or control signals generated by the processing system.

[0222] Various effects and advantages are obtained by the method and system according to embodiments of the present invention. The method and system allow tracking of elongated objects with low error rates. The method and system facilitate automatic analysis of images for purposes such as connectomes, for example, automatic analysis of images of neural tissue.

[0223] The method and system allow neurites to be tracked with an accuracy comparable to or better than conventional automatic or semi-automatic techniques and close to that of human annotators. The method and system can be applied not only to dendrite tracking, but also to tracking, for example, axons and spinal cord necks, which are generally more difficult to track than dendrites.

[0224] According to additional aspects of the present invention, an image processing method and an image processing system are also disclosed, which are operable to perform local realignment, which may include the following steps being performed to perform local realignment of slices of image data along a slice direction: determining slice-to-slice offset vectors for multiple slices along the slice direction; identifying valid slice-to-slice offset vectors between slices; and using the valid slice-to-slice offset vectors to generate a locally realigned sub-volume.

[0225] The locally realigned subvolumes may be fed to an AI model input, e.g., to an input layer of a CNN or RNN, or to filters or other preprocessing that preprocesses the realigned subvolumes before they are processed by the CNN or RNN.

[0226] Determining the slice-to-slice offset vector may include determining the autocorrelation and cross-correlation of the slices.

[0227] Determining the slice-to-slice offset vectors and / or their validity may include determining a mean and a standard deviation of the associated weighted offset vectors based on a first threshold criterion.

[0228] Identifying a valid slice-to-slice offset vector may include verifying that the determined autocorrelation satisfies a first criterion (eg, has a peak at an expected shift (eg, at zero shift)).

[0229] Identifying a valid slice-to-slice offset vector may include verifying that the determined autocorrelation peak has a standard deviation that satisfies a second criterion (such as having a standard deviation that satisfies a second threshold criterion, eg, less than or equal to a certain number of pixels).

[0230] Identifying a valid slice-to-slice offset vector may include verifying that a combination of two consecutive autocorrelation and / or cross-correlation peaks satisfies a third criterion (eg, satisfies a third threshold criterion, which may optionally depend on surrounding autocorrelations and / or cross-correlations).

[0231] Identifying a valid slice-to-slice offset vector may comprise verifying that the determined cross-correlation satisfies a first cross-correlation criterion which depends on the autocorrelation of the two involved slices (such as the mentioned autocorrelation criterion being valid for both slices).

[0232] Identifying a valid slice-to-slice offset vector may include verifying that the determined cross-correlation normalized by the expected cross-correlation value of the full correlation strength satisfies a second cross-correlation criterion (e.g., the peak value of the cross-correlation normalized by the expected cross-correlation value of the full correlation strength satisfies another threshold criterion, e.g., is equal to or greater than a fourth threshold).

[0233] Identifying a valid slice-to-slice offset vector may include verifying that a standard deviation of the offset vector satisfies a third cross-correlation criterion (such as the standard deviation is equal to or less than a fifth threshold).

[0234] Using the valid offset vector may include identifying one or more slices to be realigned even if the slice-to-slice offset vector is not considered valid, and in this case calculating the offset vector based on valid offset vectors associated with a neighborhood of these one or more slices.

[0235] Using the valid offset vectors may include generating local realignment and trilinear interpolation sub-volumes.

[0236] Generating the locally realigned and trilinearly interpolated subvolumes may include performing a floor or ceil operation.

[0237] Generating the locally realigned and trilinearly interpolated subvolumes may include performing trilinear interpolation as a sequence of linear interpolations along a basis axis.

[0238] The linear interpolation sequence may include four linear interpolations along the first dimension, two linear interpolations along the second dimension, and one linear interpolation along the third dimension.

[0239] The locally realigned sub-volumes can be processed to track elongated objects such as blood vessels or neurites in 3D.

[0240] The local realignment techniques may be used in connection with (eg, as part of) methods of tracking elongated objects, but are not limited thereto. These local realignment techniques solve the problem of referencing slices to a common reference frame in a particularly efficient manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0241] The embodiments will be described in detail with reference to the accompanying drawings, in which the same or similar components are denoted by the same or similar reference numerals.

[0242] Figure 1 is a block diagram representation of a system according to an embodiment.

[0243] Figure 2 An elongated object in 3D image data is shown.

[0244] Figure 3 An elongated object in 3D image data is shown.

[0245] Figure 4 It is a diagram of a series of points for explaining the operation of the system and method according to the embodiment.

[0246] Figure 5 is a flow chart of a method according to an embodiment.

[0247] Figure 6 is a diagram of a sub-volume of an imaged tissue volume used to explain the operation of systems and methods according to embodiments.

[0248] Figure 7 is a view of the voxel grid.

[0249] Figure 8 is a diagram illustrating determining an artificial intelligence (AI) model input.

[0250] Fig. 9 An AI model of a system and method according to an embodiment is shown.

[0251] Fig.10 An AI model of a system and method according to an embodiment is shown.

[0252] Fig.11 is a flow chart of a method according to an embodiment.

[0253] Fig.12 is a flow chart of a method according to an embodiment.

[0254] Fig.13 An AI model of the system and method according to an embodiment is shown and its operation is illustrated.

[0255] Fig.14 Another AI model according to the system and method of the embodiment is shown and its operation is illustrated.

[0256] Fig.15 is a flow chart of a method according to an embodiment.

[0257] Fig.16 is a diagram showing flight strategies that may be employed during AI model training.

[0258] Fig.17 is a flow chart of a method according to an embodiment.

[0259] Fig.18 is a flow chart of a method according to an embodiment.

[0260] Fig.19 The operation of systems and methods according to embodiments is shown.

[0261] Fig. 20 A local coordinate frame that may be employed by methods and systems according to embodiments is shown.

[0262] Fig.21 The relationship between error rate and training iterations of the method and system according to an embodiment is shown.

[0263] Fig. 22 is a flow chart of a method according to an embodiment.

[0264] Fig.23 is a diagram showing applications for which embodiments of the present invention are suitable due to their low reconstruction error rate and large distance between reconstruction errors.

[0265] Fig.24 is a diagram showing improvements obtained by the embodiment compared with the conventional technology.

[0266] Fig.25 is a diagram showing improvements obtained by the embodiment compared with the conventional technology.

[0267] Fig.26 is another diagram showing the improvement obtained by the embodiment compared with the conventional technology.

[0268] Fig. 27 is a flow chart of a method according to an embodiment.

[0269] Fig.28 is a block diagram of a representation of a system according to an embodiment. DETAILED DESCRIPTION

[0270] The embodiments will be described in detail with reference to the accompanying drawings. In the accompanying drawings, the same or similar components are represented by the same or similar reference numerals.

[0271] Although embodiments will be described in detail with reference to specific physiological objects, such as neurites, axons, and spinal cord necks, the technology disclosed herein is not limited thereto. Methods and systems as disclosed herein can be used to process image data to track various physiological objects, such as blood vessels or other elongated lumen walls. Methods and systems of operation as disclosed herein can also be used to generate annotated image data from raw image data.

[0272] Although the embodiments will be described in detail with reference to specific imaging techniques or specific image data, such as three-dimensional electron microscopy (3D EM) imaging and 3D EM images, the methods and systems can also be used to process images obtained using other imaging techniques, such as light microscopy, magnetic resonance imaging, X-ray holographic nanotomography, etc.

[0273] Slender Object Tracking

[0274] The systems and methods disclosed herein are operable to track elongated objects (e.g., neurites, but not limited thereto) in image data. The image data may be 3D image data, or may be image data in a space-time coordinate system having a time axis as one of its axes. The image data may be biological or medical image data. The image data may include a series of frames that may be taken in time sequence. The techniques disclosed herein may also be applied to other image data.

[0275] The system and method may determine a series of points predicted to be located on the centerline of an elongated object, thereby tracking the object. The system and method may be operable to process voxel values ​​or pixel values ​​of image data without requiring prior object identification in the image data.

[0276] The systems and methods disclosed herein can perform object tracking by: (a) selecting a subvolume of image data, (b) determining information about the direction along which the elongated object extends in the subvolume, and (c) processing the direction to determine further points located along the central axis. These steps can be repeated iteratively.

[0277] The subvolumes processed in an iteration may have an orientation that depends on the direction of extension of the object determined in the previous iteration. Thus, this direction steers the analysis process, similar to autonomous driving or flying techniques. The direction in which the central axis of the elongated object extends is therefore also referred to herein as the "flight direction". This direction determines the path along which the techniques disclosed herein operate.

[0278] The change in flight direction is determined at least by a turn prediction. The turn prediction may define or otherwise relate to a derivative (e.g., a first or second or higher order derivative) of the flight direction. As used herein, the term "turn prediction" refers to a quantity that depends on how the elongated object extends from a previously determined point along the elongated object and / or determines how (particularly at which subvolume orientation and / or position) the iterative techniques disclosed herein perform.

[0279] The orientation of the subvolume analyzed next may be set according to the flight direction and / or the turn prediction.

[0280] The determination of the steering prediction may use an AI model. The AI ​​model may have an input layer that is operable to receive voxel values ​​determined based on the image data. The AI ​​model input may be obtained from the image data, or more generally, may be obtained by interpolating adjacent voxels of the image data to accommodate arbitrary positions and orientations of the subvolume. The AI ​​model may have an output layer that is operable to output the steering prediction. To improve accuracy and stability, the AI ​​model may be an AI model trained not only at positions located along the centerline of the elongated object in the training image data, but also at positions located away from the centerline of the elongated object in the training image data. Thereby, convergence back to the centerline can also be ensured during inference.

[0281] The determination of the new point along the centerline may be performed using an integration of the steering prediction. For example, when the steering prediction involves higher order derivatives of the centerline direction of the elongated object, the integration may include multiple subsequent integrations.

[0282] The systems and methods are independent of affinity predictions for individual voxels and do not require explicit volume segmentation of image data. Instead, the systems and methods are operable to directly solve the problem of following the centerline of elongated objects such as neuronal lines. The systems and methods facilitate large-scale connectome reconstruction. Since they are also computationally efficient, they help reduce resource consumption for acquiring connectomes from animal and human brains (in healthy and diseased states), screening for circuit diseases in clinical settings, and biopsy analysis of pathological tissues.

[0283] The system and method provide a method for automatic axon reconstruction in 3D EM data by training an AI model on simulated axon flight pattern tracing. The system and method can employ a convolutional neural network (CNN). Details of CNN are described, for example, in "Backpropagation Applied to Handwritten Zip Code Recognition" by Y. LeCun et al., Neural Computation 1(4):541–51, 1989.

[0284] The system and method offer various differences and effects compared to the CNN-based technique of “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks.” published by M. Januszewski et al. in Nature Methods, 2018 https: / / doi.org / 10.1038 / s41592-018-0049-4.

[0285] The system and method determine the (approximate) centerline of an elongated object such as a neurite and reconstruct the centerline by iterative skeletonization. Subvolume selection (e.g., subvolume orientation) and problem formulation ensure that the analysis is aligned with the elongated axis. To maintain alignment, only the deviation of the centerline from a straight line (along the subvolume orientation) needs to be predicted.

[0286] That is, the systems and methods disclosed herein analyze a subvolume, oriented in a manner that takes into account the local orientation of the elongated object, in each of several iterations of an iterative process.

[0287] If the centerline is straight, the turn prediction is zero. If the centerline deviates from a straight line, the turn prediction can capture this deviation by means of, for example, a parabolic approximation of the neurite at the current position.

[0288] By comparison, the subvolumes of the flood filling network (FFN) in "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." published by M. Januszewski et al. in Nature Methods in 2018 https: / / doi.org / 10.1038 / s41592-018-0049-4 are oriented along the cardinal axes of the image dataset and therefore do not capture the main axis / centerline orientation of the neurites. These traditional techniques do not provide a mechanism by which the main axis or centerline will be captured or used for reconstruction. Instead, FFN predicts the affinity of each voxel, which increases the computational complexity and computational cost. Therefore, FFN is computationally more expensive than the techniques disclosed in this article. FFN is limited to integer coordinates, while the techniques disclosed in this article provide a continuous problem formulation for multivariate interpolation of image data and a spatially varying coordinate frame, which is more easily applied to continuous coordinates of input and / or output.

[0289] For the techniques disclosed herein, the AI ​​model is trained to predict the local continuity of centerlines, for example from 3D-EM subvolumes centered and aligned around neurites. This allows for direct optimization of error rate, a metric where traditional automated methods have made little progress, and also outperforms voxel-based metrics in extracting connectomes from 3D-EM volumes much larger than typical neurite diameters.

[0290] During inference, the systems and methods are operable to integrate turn predictions into new positions and orientations such that the recurring application produces trajectories of visited positions similar to skeletons generated by human annotations.

[0291] The directionality and continuity of elongated objects (such as neurites) can be directly incorporated into the problem formulation.

[0292] Tests have shown that the techniques disclosed herein produce low error rates even for lower resolution data and significantly reduce computational costs compared to previous methods. In addition, the methods and systems avoid any need for segmentation and can operate on 3D image data without prior segmentation.

[0293] Figure 1 is a block diagram of a system 10 operable to track elongated objects (eg, neurites) in 3D image data (eg, 3D-EM image data).

[0294] The system includes an imaging system 20. The imaging system 20 may be a 3D EM imaging system. The imaging system 20 may include a source 21. The source 21 may include an electron source that may be operated to, for example, output an electron beam. The imaging system 20 may include a detector 22. The detector 22 may be operated to detect electrons that have been scattered, reflected, or refracted by a physiological object within a sample (e.g., a biopsy sample). The imaging system 20 may include an image alignment component 23 for image alignment prior to reconstruction.

[0295] The imaging system 20 can be operated to perform local realignment. The local realignment can be performed by the image alignment component 23, or the imaging system 20 can be operated to perform a second local image realignment component in addition to the image alignment component 23. The imaging system 20 (and in particular its local image realignment component) can be operated to perform local realignment and dealignment for mapping to a global image coordinate system. For illustration, the imaging system 20 can be operated to perform the method of "Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment." published by PHLi et al. in Preprint.https: / / doi.org / 10.1101 / 605634, in particular in combination with the preprint Figure 3 Describe the technology.

[0296] Local realignment may be performed using techniques specifically described in more detail below.

[0297] The imaging system 20 may include an image reconstruction circuit 24. The image reconstruction circuit 24 can be used to calculate voxels of 3D image data representing the sample. The 3D image data 40 can be provided to the image processing system 30. Instead of or in addition to the 3D image data, other forms of volumetric tissue representations can be used, such as a series of two-dimensional (2D) images or projections. An example of such a representation is X-ray holographic nano-tomography (which can be performed using, for example, the technique of "Dense neuronal reconstruction through X-ray holographic nano-tomography" published by AT Kuan et al. in Nature Neuroscience Volume 23, pp. 1637–1643 (2020)).

[0298] The image processing system 30 includes one or more processing circuits 31. The processing circuit 31 may include one or more circuits, including integrated circuits, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), controllers, processors, superconducting circuits, quantum bit (qubit) processing circuits, but is not limited thereto.

[0299] The processing circuit 31 can be operated to perform elongated object tracking to follow the elongated object (e.g., a neurite, but not limited to this) through a volume representation 40 of the tissue. Starting from a position on the centerline of the elongated object, AI model inputs that depend on (and can represent) sub-volumes of the image data 40 can be fed to the AI ​​model 32 respectively. The AI ​​model 32 is trained to output the direction in which the elongated object extends from a known point in response to the image data. This direction determines the direction in which the tracking is continued, and is therefore also referred to as a steering prediction. Based on the steering prediction, the next point along the centerline (adjacent to the known point) is calculated. This can be accomplished by integration performed by a steering prediction process 33. The steering prediction determined by the AI ​​model and the new point along the centerline determined by the steering prediction process can be repeated to produce a series of points located on the centerline.

[0300] Segmentation of the image data 40 is not required for the systems or methods to operate in the manner disclosed herein.

[0301] Respectively determining points located along the centerline in 3D includes determining the location of the point within the 3D volume represented by the image data 40. Determining the point includes determining information that allows all three coordinates of the point to be specified in the volume represented by the image data 40. Determining the point may include determining three Cartesian coordinates of the point, it being understood that many equivalent formulas may be used to specify the location of a point in 3D.

[0302] The image processing system 30 can output the results of the image processing. The image processing system 30 may include a graphical user interface (GUI) to output a graphical representation of a series of points and / or center lines of the elongated object. The GUI can be operated to output a visual representation of the center line of the elongated object. Optionally or additionally, the image processing system 30 may include an output interface 34. The output interface 34 can be operated to output machine-readable data depending on the results of the elongated object tracking. The machine-readable data may include control signals depending on the results of the elongated object tracking. The machine-readable data may include control logic that depends on the results of the elongated object tracking, such as when the results of the neurite tracking are used for neuroscience-inspired machine learning (ML) model design or training.

[0303] The image processing system 30 may also be operable to control the GUI 30 to allow the user to input a starting point for object tracking. The image processing system 30 may be operable to control the GUI 30 to allow the user to optionally input a starting direction for object tracking. As explained in more detail herein, knowledge of the initial direction is optional. The initial direction may be automatically determined using uncertainty estimates obtained, for example, from a Monte Carlo exit method.

[0304] Although slender object tracking is further illustrated below in conjunction with the tracking of one slender object, the system and method may be operable to track multiple slender objects. The technology is also applicable to multiple slender objects and / or multiple slender portions of an object and / or multiple portions of an slender object. This can be done in a parallel or sequential manner. For illustration, multiple instances of the image processing technology disclosed herein can be instantiated to simultaneously track multiple slender objects in parallel. Alternatively or additionally, the system and method can be looped multiple times for different slender objects by the technology disclosed herein. This can be done until a 3D trajectory is determined for each desired slender object. For illustration, the 3D trajectory of multiple (e.g., more than 100 or more than 1000) neurites, axons and / or neural spinal necks can be tracked. The method disclosed herein is applicable to connection groups, but is not limited to this.

[0305] Figure 2 A volume 40 represented by image data and an elongated object 41 extending therein are shown. The systems and methods disclosed herein are operable to track the centerline of the elongated object 41. The systems and methods may use subvolume selection related to steering prediction, AI model processing of image data in the subvolume for updating steering prediction, and integration of steering predictions that follow the centerline of the elongated object 41. The systems and methods may employ these techniques for neurite tracing, but are not limited thereto.

[0306] Figure 3 The centerline 50 of the elongated object is shown. The technology disclosed herein is not only applicable to neurites, but also more generally to other objects, such as blood vessels, respiratory tract lumens, etc. The technology is also applicable to trajectories in a space-time coordinate system (wherein the physical object tracing the elongated path along the time axis is not necessarily physically elongated). The technology is also applicable to multiple elongated objects and / or multiple elongated portions of an object and / or multiple portions of an elongated object.

[0307] The methods and systems disclosed herein can be used to track an object having a partially cylindrical shape (e.g., the centerline 50 of an elongated object). The term "partially cylindrical" is used to refer to a geometric shape whose outer wall can be partially approximated by cylindrical segments 42, 43. The orientation of the individual cylindrical segments 42, 43 can be tilted relative to each other in 3D space. The diameter of the cylindrical segments 42, 43 can vary along the centerline 50 (usually this is also the case with neurites).

[0308] Figure 461 on the elongated object to be tracked. Alternatively, automatic object recognition may be employed to identify the starting point 61 on the object. An initial flight direction 70 specifying the direction of the elongated object at the starting point 61 may be received by user input, or may be automatically estimated by the system using, for example, prediction uncertainties (such as may be obtained from, but not limited to, Monte Carlo exit methods) that specify the uncertainty of the prediction estimates for different directions.

[0309] The system and method are operable to

[0310] - identifying a first sub-volume located around the starting point 61 (but typically having a first center slightly offset from the starting point),

[0311] - determining a first steering prediction 71 by processing image information from the first subvolume, and

[0312] - Process at least the first turn prediction 71 (eg using integration) to determine

[0313] o a position offset 72 which gives a first point 62 after the starting point 61,

[0314] o New subvolume orientation (not shown) is used in subsequent iterations.

[0315] These steps may utilize a starting point 61 and a starting direction 70 as input.

[0316] The process can then be repeated in an iterative manner. For illustration, the system and method can then operate to

[0317] - identifying a second sub-volume located around the first point 62 (but typically having a second center slightly offset from the first point),

[0318] - determining a second steering prediction by processing image information from the second subvolume, and

[0319] - Determine a second point 63 following the first point 62 by processing at least the second steering prediction (eg using integration) and determining a position offset 73 and a new sub-volume orientation (not shown).

[0320] This process may be repeated in a loop to determine, by looping extrapolation, a series of points 60 located on the centerline 50. In this way, additional steering predictions may be determined, resulting in additional position offsets 74, 75, 79, and additional points 64, 65 of the series 60.

[0321] In an iteration of the iterative process, a sub-volume 76 of the volume represented by the image data 40 is determined. The sub-volume 76 contains the current point 65 determined in the previous iteration. The sub-volume 76 has an orientation that depends on the previous steering prediction determined in the previous iteration. The image data of the sub-volume 76 is processed by the AI ​​model to determine the steering prediction 78 for the current iteration of the iterative process. Using the steering prediction 78 for the current iteration, the next point and orientation after the current point 65 is determined, for example by integration (e.g., by adding a vector proportional to the instantaneous flight direction 77 to the current point 65 to determine the next point, and by adding a vector proportional to the steering prediction to the instantaneous flight direction / sub-volume orientation 77 to determine the next flight direction / sub-volume orientation, or alternatively by using more complex extrapolation techniques to extrapolate from a series of previously determined steering predictions). As previously described, analysis of the image data contained in the sub-volume 76 provides the steering prediction 78, which is integrated to determine the next position offset 79 to continue tracking the elongated object.

[0322] Thus, the systems and methods disclosed herein can perform processing of sub-volumes of volume 40 to infer the local direction of centerline 50, which corresponds to the turn prediction. Each sub-volume can be centered on the current point determined in the previous iteration in two directions transverse to the instantaneous flight direction / sub-volume orientation. Each sub-volume may have a center that is offset from the previous point determined in the previous iteration, and the direction of the offset is parallel to the instantaneous flight direction. For illustration, sub-volume 76 can be positioned so that point 65 determined in the previous iteration is closer to an end face of sub-volume 76, which is positioned from point 65 in a direction opposite to instantaneous flight direction 77, rather than closer to the other end face of sub-volume 76, which is positioned from point 65 in a direction opposite to instantaneous flight direction 77.

[0323] Thus, the methods and systems disclosed herein employ an approach where the tracking process is automatically steered along a centerline, where the field of view (i.e., subvolume) analyzed at each step is guided along an estimated tangent to the centerline. Continuity of elongated objects is automatically established in the process. The process is similar to tracking a desired path in autonomous driving (but in 3D).

[0324] Figure 5 is a flow chart of a method 80 . The method 80 may be automatically performed by an image processing system. The method 80 determines a series of points 60 along a centerline 50 .

[0325] In step 81, the AI ​​model input is determined based on the image data and the current point. In the first iteration, the current point is the starting point 61 which may be set by user input. In each subsequent iteration, the current point may be a point 62-65 on the centerline determined in the previous iteration.

[0326] Determining the AI ​​model input in step 81 may determine the AI ​​model input as image data from a subvolume 76 of the volume 40. The subvolume 76 may contain a current point, which may be the result of a previous iteration. The subvolume 76 may have an orientation that depends on a previous steering prediction and the subvolume orientation. The subvolume 76 may (and typically is) tilted relative to an edge of the volume 40. In a first iteration, the subvolume orientation may be determined based on user input, based on prediction uncertainty calculated at different orientations, as a random orientation, or otherwise. In each subsequent iteration, the current position and subvolume orientation are determined by the AI ​​model in the previous iteration of the method 80.

[0327] Determining the AI ​​model input at step 81 may include performing an averaging (eg, weighted averaging) or other interpolation technique on the voxel or pixel values ​​of the image data 40, as will be explained in greater detail herein.

[0328] At step 82, the subvolume of the image is processed. This may include processing the AI ​​model input with the AI ​​model. The output layer of the AI ​​model may output an updated steering prediction.

[0329] At step 83, at least the steering prediction obtained at step 82 is processed to determine the next point along centerline 50 and the next orientation of the subvolume. This may include vector addition or integration of the previous (and optionally several previously determined) steering predictions, positions and orientations.

[0330] At step 84, it is determined whether a termination criterion is met. The termination criterion may be any one or any combination of the following, but is not limited thereto: intersection of the subvolume and / or the next position to be analyzed with the boundary of the image data 40 or a user-defined boundary; a threshold criterion for the number of iterations or cumulative path length; no identifiable continuation of elongated objects in the subvolume; intersection of positions within an object defined by other methods or path length accumulation (e.g., a tracked neurite intersects a previously tracked neurite). If the termination criterion is not met, the method may return to step 81. Otherwise, the results of the elongated object tracking may be output via a GUI or data interface, or may be used in other ways at step 85.

[0331] Figure 5 The technique can be used as part of a more complex process or method, or can be embedded in some higher-order logic. To illustrate, a method is disclosed that automatically detects a missing continuation that is automatically reconstructed, and initiates a process in response to detecting the missing continuation at that location. Figure 5 technology and run Figure 5 until it finds another automatically reconstructed component. From that point, the techniques disclosed herein (and as described in reference Figure 5Explained) can then be run in the backward direction to reconfirm that it ends at the same starting position.

[0332] therefore, Figure 5 The techniques can be used as building blocks for more complex reconstruction methods.

[0333] Figure 6 1 shows a position-dependent selection of subvolumes for analysis in the systems and methods disclosed herein. The points 62 along the centerline 50 and the instantaneous flight direction 71 determined in the iteration of the loop inference process are used to determine which portion of the volume 40 is to be processed next by the AI ​​model to continue tracking the elongated object. The subvolume 91 can have a cubic shape, but is not limited to this. The four edges of the subvolume 91 can be parallel to the instantaneous flight direction 71. The four edges parallel to the turning prediction 71 can be constantly selected, or selected to be shorter, longer, or the same length as other edges of the subvolume 91 depending on previous AI model predictions. The size of the subvolume 91 depends on the object to be tracked.

[0334] Point 62 may be contained within subvolume 91. Subvolume 91 may be positioned with its center 92 offset from point 62 along instantaneous flight direction 73. That is, as viewed from point 62, the subvolume may be positioned to extend a greater distance along instantaneous flight direction 73 than a distance in a direction opposite to the instantaneous flight direction.

[0335] The orientation of the subvolume is variable, depending on the previous flight direction and the steering prediction determined in the previous iteration.

[0336] For illustration, the point 65 along the centerline 50 and the instantaneous flight direction 77 determined in an iteration of the loop inference process are used to determine which portion of the volume 40 to be processed by the AI ​​model next to continue tracking the elongated object. The subvolume 93 may have a cubic shape, but is not limited thereto. The four edges of the subvolume 93 may be parallel to the instantaneous flight direction 77. The four edges parallel to the instantaneous flight direction 77 may have shorter, longer, or the same length as the other edges of the subvolume 93, and may be selected as a constant or dependent on the AI ​​model prediction from the previous iteration.

[0337] Point 65 may be contained within subvolume 93. Subvolume 93 may be positioned so that its center 94 is offset from point 65 by an offset along instantaneous flight direction 77. That is, as seen from point 65, the subvolume may be positioned to extend a greater distance along instantaneous flight direction 77 than a distance opposite to the turn prediction.

[0338] The orientations of the edges of the subvolumes 91, 93 that are not parallel to the instantaneous flight direction 73, 77 are not closely related. When selecting the subvolumes 91, 93, during training and / or during inference, random rotations 95, 96 around the instantaneous flight direction 73, 77 may be introduced. This further enhances the robustness of the techniques disclosed herein.

[0339] The random rotation 95, 96 may be replaced or combined with a method that makes the AI ​​model (approximately) equal to that rotation. This has the effect that feeding a rotation input produces a steering prediction of a corresponding rotation. This may further increase the robustness of the techniques disclosed herein.

[0340] The edges of the subvolumes 91, 93 parallel to the steering direction may be selected to be constant, or, depending on the previous AI model prediction, shorter, longer, or the same length as the edges of the subvolumes 91, 93 not parallel to the steering direction. This further enhances the robustness of the techniques disclosed herein.

[0341] The dependence of the subvolumes 91, 93 on previous turn predictions, previous flight directions, and previous points has the effect that the voxels of the subvolumes 91, 93 will generally not coincide with the voxels of the original image data 40. Interpolation methods may be used to calculate voxel values ​​within the subvolumes 91, 93 for arbitrary orientations and positions of the subvolumes 91, 93, such as Figure 7 and 8 shown.

[0342] Figure 7 and 8 The voxels of the image data 40 are shown with solid circles (e.g., the centers of the voxels). The voxels of the subvolumes 91, 93 are shown with open circles and may be anisotropic in general with respect to physical length scales. The voxels of the image data 40 define a grid 100. The voxels of the subvolumes define a second grid 105, which may be anisotropic. The second coordinate axes 106 of the second grid 105 may be rotated in 3D and / or offset in 3D relative to the coordinate axes 105 of the grid 100 on which the voxels of the original image data 40 are defined.

[0343] Interpolation techniques may be used to determine the voxel value at a location on the second grid 105 (eg, as a grayscale or a color value with various color channels), which is then one of the AI ​​model inputs.

[0344] The projection may be used to determine the voxel value on the second grid 105. To illustrate, Figure 8 As shown, projection (such as trilinear projection) may be used to project the image data onto the plane of the grid 105 .

[0345] Other techniques may be used. For illustration, a point on the second grid 105 (i.e., a voxel center of a subvolume 91, 93) may be located within a cube defined by a plurality (e.g., eight) of voxel centers 103 of the original image data 40. A weighted average may be performed to calculate a voxel value for a point 106 of the subvolume 91, 93 based on the voxel values ​​of the original image data. For illustration, the voxel value of the point 106 may be calculated as a weighted average of the voxel values ​​of eight voxel centers 103 located around the point on the second grid 105. Other techniques may be used to interpolate image data to the grid 105 defined for the subvolume to determine the input to the AI ​​model, such as trilinear projection (as described above).

[0346] As described above, and as discussed in more detail below, local realignment may be applied. Figure 7 In the 2D image plane (given by Figure 7 A local realignment is applied by applying a (possibly non-integer) x and / or y offset (which can be obtained from the cross-correlation) of the voxels at the black dot positions (shown in the black dot plane). Thus, trilinear interpolation (when the voxels at the black dot positions are subjected to local realignment) is used to generate Figure 7 The voxel value at the white point position in ) can be calculated as follows: two bilinear interpolations for each white point position determined using the upper and lower truncation on the z coordinate, respectively, where different x / y offsets may be applied to the black point position, and then the two intermediate values ​​of the bilinear interpolation are combined by another linear interpolation along the z axis.

[0347] Regarding the configuration of the AI ​​model, an AI model including a convolutional layer has been determined to provide good performance. For illustration, the AI ​​model may be or may include a CNN.

[0348] Fig. 9 and 10 An AI model that may be used in the systems and methods disclosed herein is shown.

[0349] The AI ​​model 110 includes an input layer 111 that is operable to receive AI model inputs based on pixel or voxel values ​​of image data. The AI ​​model inputs may include or may be image data within the subvolumes 91, 93, as determined from voxel values ​​of the image data 40. The voxel values ​​may be converted from integers to floating point representations and normalized prior to being processed by the AI ​​model. Normalization may be applied to limit AI model input values ​​to a specific range and / or change the statistical properties of the AI ​​model input values, such as to achieve an average input value of zero and a standard deviation of the input values ​​of one.

[0350] The AI ​​model 110 includes an output layer 112 that is operable to provide an AI model output that is or is indicative of the steering predictions 71 - 75 .

[0351] The AI ​​model 110 includes a hidden layer 113. The hidden layer 113 may include several convolutional layers 114, 115. The hidden layer 113 may include a strided convolutional layer and / or a fully connected layer. The stride may be different for different convolutional layers.

[0352] The hidden layers 113 may include several convolutional layers 114 , dropout and / or reshaping layers 116 , and one or several fully connected layers 115 .

[0353] The hidden layer 113 may include a linear layer as the last layer.

[0354] Various activation functions can be used. For illustration, both exponential linear units (ELU) and rectified linear units (ReLU) have been successfully used in the systems and methods disclosed herein. A variety of other activation functions can be used in addition or alternatively. These activation functions may include any one or any combination of proportional exponential linear units (SELU), Gaussian error linear units (GELU) activation functions, but are not limited to this. A variety of other artificial neural network layers can be used in addition or alternatively. These layers may include any one or any combination of fully connected layers, pooling layers (e.g., max pooling), (self) attention modules, dilated convolutions, (batch) normalization layers, converters, recurrent layers (e.g., long short-term memory (LSTM), gated recurrent units (GRU)), but are not limited to this.

[0355] To further enhance efficiency and robustness, the AI ​​model may be trained to provide information about steering predictions in a spatially varying reference frame along an elongated curve. To illustrate, the AI ​​model may be trained (during a training phase) and may be operated during inference to provide information about steering predictions in a (local) Bishop frame, for example by outputting two Bishop curvatures (see below for more details).

[0356] Fig.11 is a flow chart of the method 120. The method 120 may be automatically performed by the image processing system 30. The method 120 may be performed to determine a series of points 60 along the centerline 50 by recursive inference.

[0357] At step 121, a subvolume is selected for processing. The subvolume may have a position that depends on a previously determined point along the centerline (which may be specified by user input in a first iteration or be a result of a previous iteration in a later iteration). The subvolume may have an orientation that depends on a previous subvolume orientation and a previously determined turn prediction (which may be specified by user input or automatically determined in a first iteration, and which may be obtained from a previous iteration in a subsequent iteration).

[0358] At step 122, the AI ​​model operates on the image data within the sub-volume. Interpolation of the image data voxels may be performed to determine voxel values ​​in a voxel grid of the corresponding sub-volume for any orientation and / or position of the sub-volume.

[0359] At step 123, the AI ​​model may provide an updated steering prediction as an AI model output. The updated steering prediction may be specified relative to a local reference frame, for example, a curvature value relative to a reference frame associated with the subvolume (e.g., a Bishop coordinate system). The AI ​​model may provide additional outputs, such as information about the distance to obstacles.

[0360] At step 124, a new point along the centerline may be determined based on the turn prediction from step 123. To illustrate, an integration may be performed to determine the new point along the centerline.

[0361] Fig.12 1 is a flow chart of method 125. Method 125 may be automatically performed by image processing system 30. Method 125 may be performed to determine a series of points 60 along centerline 50 by looping inference. Method 125 may include reference Fig.11 Steps 121-124 explained. The method 125 may further comprise an additional step 126 of determining a rotation about the instantaneous flight direction, which is used to specify the subvolumes in subsequent iterations. Figure 6 Exemplary rotations 95 , 96 are shown in .

[0362] Fig.13 The AI ​​model architecture and data processing that may be employed in the methods and systems disclosed herein are shown.In each iteration, a subvolume 131 centered at the current position (in a direction perpendicular to the current flight direction) is analyzed.

[0363] The AI ​​model 130 may include an input layer operable to receive K×N×M voxel values ​​of image data in the subvolume 131. The number of voxels K and N in a direction transverse to the most recent turn prediction may exceed the number of voxels M parallel to the most recent turn prediction. For illustration, K=N=96 and M=16 may be used. In addition, the physical size of a single voxel of the subvolume may be selected anisotropically and independently of the physical size of a single voxel in the image data 40.

[0364] The values ​​of K and N may be set depending on the object being tracked. To illustrate, for an object that is known or expected to have an elliptical cross-section in a plane transverse to the central axis, it may be useful to set K and N to different values.

[0365] exist Fig.13In the implementation of , the AI ​​model includes seven strided convolutional layers conv1 to conv7. These layers do not have to be identical. For illustration, the three convolutional layers conv1 to conv3 can have 5×5×3 convolutions using stride 2×2×1, and the four convolutional layers conv4 to conv7 can have 3×3×2 convolutions using stride 1×1×1.

[0366] exist Fig.13 In the implementation, the AI ​​model includes a dropout layer, and its dropout rate can be 0.5.

[0367] exist Fig.13 In the implementation, the AI ​​model includes three fully connected layers fc1-fc3 and one linear layer.

[0368] The AI ​​model output 132 may be a steering prediction, optionally determined in a local reference frame (eg, as Bishop curvature).

[0369] The process includes integration 133 to determine the next point along the centerline based on the AI ​​model output 132. The subvolume analyzed in subsequent iterations can be set to have no additional random rotation ( Fig.13 ) or with additional random rotations 134.

[0370] Fig.14 Another implementation of an AI model 138 that can be used in the methods and systems disclosed herein is shown. The parameters 139 of the convolutional layers conv1 to conv7 are compared to Fig.13 The AI ​​model can be modified. Alternatively or additionally, the parameters of the fully connected layers fc1-3 are compared to Fig.13 The AI ​​model can be modified. For illustration, Fig.14 In one embodiment, the parameter value before or after the vertical bar shown in Fig.14 The vertical bar in the figure indicates the parameter value 139' and the parameter value 139.

[0371] The system and method can be used for neurite tracing, but is not limited to this. The system and method allow neurite reconstruction to be performed from a 3D-EM volume as a centerline reconstruction task, where the neurite is represented by a series of access points. A CNN is trained on the task of predicting local neurite continuity from a 3D-EM subvolume centered and aligned with the neurite, similar to human annotation flight mode, see, for example, Boergens et al., 2017, Nature Methods (Natural Methods) 14 (7): 691–94, published in "WebKnossos: Efficient Online 3D Data Annotation for Connectomics.").

[0372] Integration of the predicted neurite continuity yields new positions and orientations, which are used to generate subsequent CNN inputs. Iterative application of this process transforms starting positions and orientations into neurite skeleton reconstructions using only 3D-EM data without intermediate steps such as volume segmentation.

[0373] for Fig.13 The input to the AI ​​model of the CNN consists of 96×96×16 voxels (Vx) of a 3D-EM subvolume 131 centered and aligned on neurites. The subvolume can cover, for example, a field of view of approximately ∼1×1×0.7 μm for an axon (see Fig.13 ).

[0374] The third dimension (Z) corresponds to the current flight direction of the process, and the current position can be taken as the center of the 4Z plane.

[0375] Therefore, the field of view is asymmetric along the Z direction, with more contextual information available in the forward flight direction than in the reverse flight direction. This allows for better steering towards the “exit” within the axonal varicosity.

[0376] EM data can be projected onto the plane of axon alignment by trilinear interpolation.

[0377] Note that by using only 3D EM subvolumes as input, it is possible to extract continuous relevant information about neurites from a single input. Alternatively, 2D EM images can also be fed to a recurrent neural network, which can internally track the image sequence and thus also estimate the continuity of neurites.

[0378] for Fig.13The CNN architecture uses seven 3D strided convolutional layers, followed by a dropout layer (dropout rate: 0.5), three fully connected layers, and a final linear layer that estimates two steering commands and the distance to the membrane. The two steering values ​​output by the output layer as steering predictions can represent the Bishop curvature described in more detail below. Due to the use of nonlinear ELUs (see, for example, DA-Clevert et al., "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).", 2015, Iclr. 566 https: / / doi.org / 10.3233 / 978-1-61499-672-9-1760).

[0379] For normalization of the network's input and output, the Bishop frame and associated Bishop curvature can be used, as will be described in the technical section below. The Bishop frame is a local orthogonal coordinate system spanned by a tangent vector and two normal vectors, and obeys the rotation minimization property that does not allow any distortion around the tangent vector. The projection plane of the neurite arrangement input to the AI ​​model can be defined by the Bishop normal vector, and the flight direction of the network can correspond to the tangent vector.

[0380] The evolution of the unit vectors in the Bishop frame is coupled via the Bishop curvature, one curve per normal vector direction.

[0381] The Bishop curvature can be the output of the output layer of the AI ​​model for steering. Each of the two Bishop curvatures is similar to the signed curvature of the corresponding normal vector direction. Signed curvature has also been used as a steering command for image-based road following, although only for one steering direction corresponding to a planar curve (see, for example, KM. Bojarski et al., "End to End Learning for Self-Driving Cars.", published in ArXiv E-Print. https: / / arxiv.org / pdf / 1604.07316.pdf in 2016).

[0382] The task of predicting neurite continuity in the form of Bishop curvature can be interpreted as fitting a parabola to the centerline of the neurite. The curvature vector, calculated as the sum of the Bishop curvatures multiplied by their corresponding Bishop normal vectors, determines the direction and magnitude of the curvature of this parabola.

[0383] Therefore, the Bishop's curvature output by the AI ​​model output layer can be integrated to determine the next point in a series of points along the centerline (the series is a priori unknown and is determined to track the centerline).

[0384] train

[0385] The operation of the trained AI model has been explained above. The systems and methods disclosed herein are operable to train the AI ​​model for its intended use.

[0386] Fig.15 is a flow chart of a method 140 including AI model training 141 and operation of the trained AI model during inference 142. Using the trained AI model at step 142 may be implemented using any of the techniques described herein for inferring a direction in which an elongated object extends.

[0387] The training at step 141 may be supervised training. A collection of annotated 3D-EM images may be used in the training. The training may include gradient-based updates of AI model parameters. Techniques known to those skilled in the art may be used to update the model parameters, such as gradient descent techniques.

[0388] The training of step 141 may include training the AI ​​model to conform to a flight strategy that causes the AI ​​model to converge back to the centerline 50 even when starting from or reaching a position that is off the centerline. To achieve this, the training step 141 may include (also) training the AI ​​model in a scenario where the AI ​​model starts from a position that is off the centerline 50 and / or during training the predictions of the AI ​​model may be integrated and fed back, as is done in inference, to potentially also produce positions that are off the centerline. A flight strategy may be implemented that forces convergence back to the centerline 50 even when the initial subvolume (i) starts from a position that is off the centerline and / or (ii) is not aligned with the direction of the elongated object. Thus, training may also be performed using subvolumes where none of the voxel planes of the subvolumes 91, 93 have a center that coincides with a position along the centerline 50. At least a portion of the subvolume may be intentionally selected so that, for example, the 4th plane (for Fig.13 The model in , from the center of the 16 planes of the subvolume) has an offset from a point on the centerline determined by the annotated data or in a previous iteration. The offset can be oriented transverse to the previously determined extension direction (i.e., in the Bishop framework, transverse to the steering prediction).

[0389] The training can be implemented in a way that uses one or several training parameters that are variable during the training. This allows different strategies to be implemented during the training.

[0390] When using this training technique, the results of the slender object model training are even further improved.

[0391] In the training of step 141, the convergence distance of the flight strategy can determine the speed at which the iterative process converges back to the centerline. The convergence distance can be a dynamic parameter that changes during the AI ​​model training. During the AI ​​model training, the convergence distance can be set as a function of the distance from the physiological boundary. The convergence distance can be a monotonically increasing function of the distance from the physiological boundary, thereby achieving a more aggressive return to the centerline at a position closer to the physiological boundary (e.g., a cell boundary).

[0392] As an alternative to supervised learning with hand-crafted flight policies, reinforcement learning can also be used to directly learn a flight policy by defining a reward function that encourages stable tracking of slender objects.

[0393] As a specific example, Fig.13The CNN architecture can be trained on the neurite following task using TensorFlow (M. Abadi et al., “TensorFlow: A System for Large-Scale Machine Learning.”, 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI'16), 2016, 265–537 84).AI models can be trained using RMSProp in mini-batches of 128 (see, e.g., T. Tielman and G. Hintony, “Neural Networks for Machine Learning Lecture 6a: Overview of Mini-batch Gradient Descent.”, Lecture Notes 2015, http: / / www.cs.toronto.edu / ~tijmen / csc321 / slides / lecture_slides_lec6.pdf, 2015), with momentum (see, e.g., Rumelhart et al., “Learning Representations by Back-Propagating Errors,” Nature 323(6088), 1986; I. Sutskever et al., “On the Importance of Initialization and Momentum in Deep Learning. "), or Adam (DP Kingma and JL Ba, "Adam: A Method for Stochastic Optimization.", In 3rd International Conference on Learning Representations, ICLR 2015-Conference 607 Track Proceedings, 2015 https: / / arxiv.org / abs / 1412.6980) to minimize the mean squared error of the neural network training parameters plus an L2 regularization loss term.

[0394] Training and subsequent testing have shown that by also training on positions that are off-centerline, performance during inference can be improved. During inference, the current position and orientation depends on past AI model decisions. Therefore, even small errors in the network’s past predictions of steering can cause the network to move away from the neurite centerline.

[0395] In order to enhance the stability of path following (i.e. tracking), the AI ​​model can also be trained in eccentric positions and deviation directions, and the steering prediction is adjusted accordingly, resulting in a return to the centerline of the neurite. In order to sample the eccentric position within the neurite, the neurite diameter and / or shape must be at least roughly known during training. Volume reconstruction of the neurite can be used to determine the eccentric position within the neurite. The mapping from a specific deviation position, deviation direction state to an adaptive steering is also called a flight strategy.

[0396] A greedy flight strategy can be used that is based only on local information available to the AI ​​model within a limited field of view, as described in more detail herein. The distance of convergence back to the centerline is a free parameter of this flight strategy. Different convergence distances can be used for eccentric positions within the synaptosome. When getting closer to "hitting" the cell membrane, more aggressive turns with shorter convergence distances can be used to stay within the neurite, while for larger distances to the membrane, gentler turns are tolerable and better maintain the neurite arrangement of the input field of view. Therefore, a dynamic convergence distance can be used, which can be set according to (e.g., equal to) the distance to the cell membrane along the flight direction during training. This gives rise to the concept of obstacle avoidance, namely the cell membrane, and performs better than using a constant value of the convergence distance empirically during training.

[0397] Fig.16 A volume 40 of 3D training data is shown. A neurite 151 has been annotated in the 3D training data. The AI ​​model is also trained using subvolumes, such as an example subvolume 156 surrounding a point such as example point 152, where example point 152 (which is located at the center of a voxel plane of subvolume 156) deviates from the centerline of neurite 151. During training, the flight direction 153 can be (significantly) different from the extension direction of the neurite within subvolume 156. One or more training parameters, such as the convergence distance that determines the speed at which the process converges back to the centerline of the neurite, can be changed dynamically. For illustration, the convergence distance can be set according to a distance 155 from a cell membrane 154, which is only shown schematically. Distance 155 can be measured along the instantaneous flight direction.

[0398] Performance on error rate

[0399] To evaluate performance, image processing systems and methods were evaluated on a validation set and two test sets of linear axon branches by running loop inference starting from both sides. As disclosed in more detail herein, reaching a set of thresholds on the distance or angle relative to the ground truth trace triggers a reset of the ground truth trace. When the network reaches the other end of the axon branch, the trace is stopped.

[0400] Each steering error that results in a reset corresponds to a merging error of entering the wrong process and a splitting error due to discontinuing the neurite of interest. Each reset is counted as two errors. The error rate is the number of errors divided by the path length of the ground truth skeleton. This metric allows model selection based on a validation set in the case of multiple trained AI models, as well as a rough comparison with other automated methods that report the distance between errors, i.e., the inverse error rate, and allows comparison with human error rates.

[0401] The image processing system and method were trained on a 3D-EM dataset of mouse primary somatosensory cortex 4 obtained using serial block-face scanning electron microscopy (SBEM, "Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure." by W. Denk and H. Horstmann, 2004, edited by Kristen M. Harris. PLoS Biology 2 (11)). The dataset has a resolution of 61.8 × 94.8 × 92.6 μm 3 The dimensions are 11.24×11.24×28nm 3 The voxel size of .

[0402] For model selection, image processing systems and methods were evaluated on a validation set consisting of 13 axons with a path length of 1.4 mm. For the validation set, axon branches with a path length of less than 5 μ row were excluded to better heuristically detect errors. The training and validation set axons were random subsets of non-overlapping axons bred with fragments of presynaptic classification obtained using the techniques disclosed in the following, "SynEM, Automated Synapse Detection for Connectomics." published by B. Staffler et al. in ELife 6 (July) in 2017: e26414. Averaged over all models and training iterations, the random inference mode outperformed its normal mode counterpart by 33% (range: 12-52%).

[0403] The AI ​​model identified as having particularly good performance on the validation set used only EM data as input, had an ELU activation function, was trained for 700,000 training iterations, and produced 16.4 errors / mm based on heuristic error detection. Manual inspection of the tracking of the inventive technique produced 1 false positive and 0 false negative resets (96% precision, 100% recall). The true error rate was 15.7 errors / mm.

[0404] When the selected AI model was applied to a set of 59 somatic propagating axons (path length 6.2 mm), it produced 29.8 errors / mm in 185 resets, including 9 false negative resets and 5 false positive resets (95% precision and 95% recall). The true error rate was 30.4 errors / mm.

[0405] Since somatically propagated axons are known to tend to have larger diameters, a random set of 10 axons with a path length of 1.6 mm, which was previously used to quantify human error rates (see, e.g., Boergens et al., 2017, “WebKnossos: Efficient Online 3D Data Annotation for Connectomics.” Nature Methods 14(7): 691–94, 2017, was also tested. On this test set, the image processing systems and methods disclosed herein produced 58.1 errors / mm by heuristic error detection. Manual inspection revealed 4 false positive and 2 false negative resets (96% precision, 98% recall), resulting in a true error rate of 55.6 errors / mm.

[0406] Possible further training and testing implementations

[0407] Exemplary training data and evaluation techniques have been described above.Possible further 3D-EM datasets of neural tissue are provided next, and within these training data, validation and test sets that may be used in association with the methods and systems disclosed herein.

[0408] The method and system can be 92.6×61.8×94.8μm 3The SBEM dataset (“Serial block-face scanning electron microscopy to reconstruct three-dimensional tissue nanostructure” by W. Denk and H. Horstmann, 2004, PLoS Biology, 2, e329. doi:10.1371 / journal.pbio.0020329) was trained and / or used on layer 4 primary somatosensory cortex from 28-day-old mice (previously densely reconstructed by A. Motta et al. (2019)). This tissue is typically whole-mount stained (“Wiring specificity in the direction-selectivity circuit of the retina” by KL Briggman et al., 2011, Nature, 471, 183-188, doi:10.1038 / nature09818) and is sized at 11.24x11.24nm. 2and imaged at a nominal cut thickness of 28 nm. Training and validation axons can be sampled from a set of axons propagated from presynaptic classified segments obtained using SynEM (B. Staffler et al., “SynEM, automated synapse detection for connectomics”, 2017, Elife, 6, e26414.doi:10.7554 / eLife.26414) and skeletons traced by annotators. To obtain volume reconstructions of training axons, over-segmented fragments obtained using SegEM with parameters set to whole-cell segmentation of the cortical dataset (M. Berning et al., 2015, “SegEM: Efficient Image Analysis for High-Resolution Connectomics”, Neuron, 87, 1193-1206. doi:10.1016 / j.neuron.2015.09.003) can be picked up and combined. The volume masks can also be used to iteratively refine the interpolated skeleton tracing to produce better centerline approximations. A set of 14 axons with a path length of 1.2 mm can be used for training, and up to 700,000 weight updates corresponding to about 260 epochs are performed. The validation set may consist of 13 axons with a path length of 1.4 mm, where branches smaller than 5 μm are excluded for better heuristic error detection. The third group of 10 paths with a length of 1.7 mm and a distance from the bounding box of (2.5 μm) 3 Axons propagated in , which can be used as a test set. These axons are the same as those used previously to evaluate human and semi-automatic segmentation (KM Boergens et al., 2017, “webKnossos: efficient online 3D data annotation for connectomics”, Nature Methods, 14, 691-694, doi:10.1038 / nmeth.4331 and Motta et al., 2019, supra).

[0409] To achieve automatic attachment of spine heads on this dataset, an AI model trained on Axon (e.g. Fig.14and explained in more detail in the paper) are evaluated on a random subset of 50 spine heads previously attached by human annotators, and also on the set of spine heads previously used as a test set in the ibid. paper by Motta et al., 2019.

[0410] Next, the AI ​​model of the method and system disclosed in this article can be trained, tested, and applied to 1.3x1.3x0.25mm 3 A subset of the dataset, taken from the barrel cortex of 28-day-old mice, was stained following the protocol “Large-volume en-bloc staining for electron microscopy-based connectomics” by Y. Hua et al., 2015, Nature Communications, 6, 7923. doi: 10.1038 / ncomms8923, with minor modifications, sectioned at 35 nm using ATUM (KJ Hayworth et al., “Automating the collection of ultrathin serial sections for large volume TEM reconstructions”, Microscopy and Microanalysis: Cambridge University Press, 2006), and imaged at 4 × 4 nm using a multibeam scanning electron microscope. 2 (ALEberle et al., “High-resolution, high-throughput imaging with a multibeam scanning electron microscope”, Journal of Microscopy, 259, 114-120. doi:10.1111 / jmi.12224, 2015).

[0411] The 8×8×35 nm can be performed using conventional methods or techniques such as those based thereon. 3The resolution (downsampled by a factor of 2 in xy) was applied to the segmentation and clustering of the dataset; briefly, a 3D U-Net (Ronneberger et al., “U-net: Convolutional networks for biomedical image segmentation”, Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), Springer, Cham, 2015) was used to predict each spindle voxel affinity according to Lee, K., Zung, J., Li, P., Jain, V., & Seung, HS, “Superhuman Accuracy on the SNEMI3D Connectomics Challenge”, arXiv e-prints, 2017, from which a watershed-based over-segmentation was generated. Over-segmented segments can then be combined using hierarchical clustering as proposed by J. Funke et al., 2019, IEEE Trans Pattern Anal Mach Intell, 41(7), 1669-1680. doi:10.1109 / TPAMI.2018.2835450, "Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction". In addition, neurite type prediction, blood vessel and nucleus detection can be incorporated into the clustering to further reduce merging errors. Neurite type prediction is also used for spine head detection, which is the basis for spine head attachment of the methods and systems disclosed herein.

[0412] To train the AI ​​model of the methods and systems disclosed herein on axons, a training set can be obtained based on a set of 10 somatically propagated axons from layer 4, for which the annotator can pick up fragments from the over-segmentation to obtain a volume reconstruction. Kimimaro (“Kimimaro: Skeletonize densely labeled 3D image segmentations” published by W. Silversmith et al. in 2021 in the GitHub repository (v3.0.0ed.) can be used, followed by subsampling and B-spline interpolation to extract the centerline skeleton from the volume reconstruction. This can produce a training set with a total axon path length of 21 mm. As an axon validation set, the size (2μ small) from layer 4 propagation was 3 A random 10 of the 20 axons in the bounding box of (50 μm) 3 The bounding box is traced, resulting in a 1 mm path length. For the axon test set, another one within layer 4 (1.5 μm) 3 Bounding boxes were densely annotated and a random subset of 5 axons could be included in the (150 μm) 3 The subvolume Si150L4 (accessible at https: / / wklink.org / 7122) was traced, resulting in a path length of 1.7 mm.

[0413] A separate AI model according to the methods and systems disclosed herein can be trained on the spine appendage. For this training and validation set, a set of 20 (5 μm) 3 Bounding box set generation. A subset of approximately 1000 spine heads can be annotated by annotators from spine head through spine neck to dendrite trunk in 4 × 4 × 35 nm. 3 Here, we again use Kimimaro (W. Silversmith et al., 2021, supra) to extract centerline skeletons from the stereo masks for training, producing 2 mm spine neck tracking. In addition, a random subset of 76 spine heads with 0.2 mm path length from 20 bounding boxes can be skeleton tracked to serve as a validation set. Another randomly chosen (5 μm) 3 Evaluation was performed on a bounding box (Si11L3, accessible at https: / / wklink.org / 2458) containing 91 densely annotated spine head to dendritic trunk skeleton traces.

[0414] To evaluate the error correction of another state-of-the-art segmentation that can be performed using the methods and systems disclosed herein, the methods and systems disclosed herein can be applied to the recently published mm 3 Scaled multiSEM dataset, using voxel size 4×4×33nm 3 (A. Shapson-Coe et al., “A connectomic study of a petascale fragment of human cerebral cortex”, published in bioRxiv, 2021, 2021.2005.2029.446289.doi:10.1101 / 2021.05.29.446289) segmented and segmented using a flood filling network (FFN; M. Januszewski et al., “High-precision automated reconstruction of neurons with flood-filling networks”, published in Nature Methods, 2018.doi:10.1038 / s41592-018-0049-4). Specifically, the training and evaluation of the technology can focus on (150μm) 3 A bounding box containing 6.5 mm of axon path length of the provided ground truth skeleton trajectory. The ground truth skeleton trajectory published in this box can be used to assess FFN on somatic axons. To fine-tune the processing of the methods and systems disclosed herein for this dataset, the ground truth skeleton trajectory of densely populated axons can be obtained by centered (15 μm) 3 The sampling size within the bounding box is (2.5μm) 3 We generate a bounding box of 5 axons, annotate all processes in that bounding box, and then sample a random subset of 5 axons that can be found throughout the entire (150 μm) 3 The bounding box is tracked, resulting in a path length of 1.25 mm. Here, the skeleton annotation has been completed with high precision along the centerline, so that no post-processing is required. The volumetric neurite mask required for training can be generated by picking cf from the segment of the C3 FFN segmentation (the above-mentioned article published by Shapson-Coe et al. in 2021). Then, based on the validation set of axon training from the mouse cortical ATUM-multiSEM dataset, the best model checkpoint of the model used by the method and system according to the embodiment is used as initialization. The model can be trained with gradient updates for approximately two million more times until it converges in terms of the reset-based error rate on the training axons.

[0415] For the evaluation of FFN and the methods and systems disclosed herein (see e.g. Fig.26 and accompanying description), size (1.5 μm) 3 Another bounding box can be annotated for the entire (150 μm) 3 Bounding box traced a random subset of 5 axons, yielding a path length of 1.4 mm.

[0416] Additionally, the methods and systems disclosed herein may be trained or used for any or a combination of the following:

[0417] During inference, artificially generated data that mimics the appearance of the elongated objects to be tracked

[0418] X-ray, in particular X-ray holographic nano-tomography, light microscopy or magnetic resonance imaging of one or more elongated objects.

[0419] The amount of training data for path lengths required for the methods and systems disclosed herein to work as described may generally depend on the resolution and visual complexity of the data, the variability in the morphology and appearance of the elongated objects to be tracked, the algorithm and artificial neural network architecture used for training, and the amount, type, and order of image data used for training.

[0420] Performance Compared to Other Image Processing Methods

[0421] The performance of the image processing systems and methods disclosed herein is also compared with other techniques.

[0422] We studied to what extent the techniques disclosed herein can replace human annotations and which connectome analyses will thus become fully automated. To this end, we started with a dense connectome analysis of a piece of mammalian neocortex (“Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex.” by A. Motta et al., 2019, Science, Vol 366. Issue 6469. p.eaay3134), in which, based on automatic segmentation, human annotators had been asked to solve a set of automatically identified problem locations, consuming a total of 4,000 work hours. To resolve split errors, manual queries were placed at the end of the object, asking to continue if possible, and stopping the task when another automatically reconstructed object was reached. To resolve merging errors, cross configurations had been detected and queries had been placed at the exits, asking to continue appropriately to one of the other cross exits. By starting with the techniques disclosed herein, the techniques disclosed herein are used to replace these human annotations, querying in a manner similar to that of human annotators. In fact, 76% of the end tasks and 78% of the cross tasks are correctly tracked by the disclosed technique (humans: 74% and 94%, respectively). In addition, the disclosed technique of running from the endpoint back to the beginning gives an error signal that allows avoiding most of the merge errors, resulting in only 4% of the queries, while the new technique disclosed introduces a merge error for the end tasks and 1% for the cross tasks, which is comparable to human annotations.

[0423] While this performance suggests that the techniques disclosed herein can faithfully replace human annotation, we wanted to quantify this directly as a measure of reconstruction success using connectome analysis. Some connectome analyses require higher reconstruction accuracy than others. In particular, the following were considered: (1) paired same-axon-same-dendrite synapse analysis aimed at measuring the learning score of the connectome (Motta et al., supra); (2) spine rate analysis used to identify interneuron dendrites; (3) axon type analysis based on the distribution of synaptic targets of axons. Three types of connectomes were used for comparison: (I) a connectome obtained by fully automated reconstruction in (Motta et al., supra) before any centralized human annotation; (II) a connectome in (Motta et al., supra) including 4,000 hours of human annotation; and (III) a connectome obtained by combining the fully automated reconstruction in (Motta et al., supra) with the techniques disclosed herein, resulting in a fully automated and automatically proofread connectome. These three types of connectome analyses (I), (II), and (III) were performed. While paired synapse analysis already produced similar results in the automated state before corrections were implemented by the techniques of the present invention, the correct spine rates of apical dendrites and the true fraction of excitatory axons defined by their spine head preferences required manual correction or correction implemented by the techniques of the present invention. Furthermore, corrections implemented by the techniques of the present invention restored axon target specificity of inhibitory axons on apical and smooth dendrites, which was undetectable in the automated state before the techniques disclosed herein and manual corrections. Thus, in fact, when utilizing automated connectome metrics limited by the types of connectome problems that can be automatically analyzed, error correction using the techniques disclosed herein will shift automated analysis performance from simpler connectome problems to more complex connectome problems.

[0424] The techniques disclosed herein were also applied to data sets from an automated tape collection ultramicrotome (ATUM) / multi-SEM imaging method, which is significantly larger and has different image resolution and contrast. The techniques disclosed herein were tested for their usability for seed axon reconstruction: Reconstruction of the axon began at the cell body. After the initial segmentation and aggregation steps, the termini were queried using the techniques disclosed herein in the following manner: axon termini were detected, and the techniques disclosed herein tracked these until new agglomerates of 2.5 μm in length were found or any one of a number of stopping criteria was met. To avoid merging errors, new agglomerates were added only when the techniques disclosed herein verified reverse tracking and met a number of type and angle criteria. If the axon training technique disclosed herein failed, a second inventive tracker trained on the spinal neck was used. Performing 20 iterations for each axon, we found that the techniques disclosed herein were able to extend axon recall from 7% to 37% (median: from 6% to 44%; precision decreased from >99% to 74%) for local connectome analysis with a volume size of approximately 200-300 μm on one side corresponding to local intralaminar connectivity in mouse cortex.

[0425] Similar to thin axons, thin spine necks can pose difficulties for current reconstruction pipelines. However, in order to assign synapses on the spine heads to the correct cells, the spine necks need to be reconstructed with high accuracy. Therefore, it was tested whether the technology disclosed in this article can improve the precision / recall of spine head attachment from global clustering (100% / 43%) and local clustering (100% / 75%, a strategy similar to (Motta et al., supra)). To this end, the technology disclosed in this article begins to track the two candidate directions with the lowest uncertainty in the prediction among the detected spine heads. The maximum distance tracked by the inventive technology disclosed in this article is 7μm, or until a dendritic aggregation of sufficient size is reached. The spine heads are then added to the dendritic aggregates using the candidate order, type, and attachment information. This increased the spine head attachment recall from 75% to 94% while maintaining a high accuracy of 97%.

[0426] The present technique also provides enhancements over techniques such as flood-filling networks (M. Januszewski et al., “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks.”, Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4, 2018). Application of the present technique to terminals automatically detected from flood-filling network (FFN) segmentation resolved 60% of splits on a random set of axons traced throughout the subvolume, reducing the split rate from 65-70 splits / mm to 28-31 splits / mm while introducing only a few new merging errors (merging rate increased from 3-4 merges / mm to 5-6 merges / mm). Notably, compared to limiting to soma-proximal axons, the application of the present technique to terminals that are not necessarily connected to the mm 3 Evaluation of random axons of the soma within the scaled volume better reflects the reconstruction quality. Evaluation of FFNs on axons connected to the soma restricted to the volume (as previously used to demonstrate reconstruction performance, Januszewski et al., 2018) underestimates the splitting and merging errors of random axons by 3-4 times (15-22 splits / mm, 0.2-1.1 merges / mm).

[0427] As shown in these results, the technology disclosed herein can be used as a fully automatic annotation technology for a wide variety of situations, and produces fully orthogonal data that can improve all currently available automatic segmentations. This is particularly significant for the segmentation obtained using the most advanced flood filling network (FNN). The computational complexity of the technology disclosed herein is advantageously lower than other methods, and the direct end-to-end strategy of the technology of the present invention can allow for optimization of additional accuracy and computational efficiency, which will be the next key challenge in the connection group of future EB-level data sets (A. Motta et al., CurrOpNeurobiol. Volume 55, April 2019, pp. 180-187, "Big data in nanoscale connectomics, and the greed for training labels").

[0428] Efficiency compared to other image processing methods

[0429] To estimate the computational cost, we benchmarked the disclosed techniques on approximately 128,000 end tasks from the first set of end detections in “Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex.” (A. Motta et al., 2019, Science, Vol. 366. Issue 6469. p.eaay3134). The step factor was set to f=5, and the analytically derived equations of the Bishop framework along the parabola were used to integrate the turn predictions. The computation was performed on a single Tesla V100 with 32 cores (Intel(R) Xeon(R) Gold 6130 CPU, 2 sockets), The total runtime on a single node with a GPU (PCIE-16GB) and less than 128GB of RAM is 13.6 hours for the reconstruction of approximately 2.1 meters of axons (including backward validation tracing, a total of 64 million CNN inferences), resulting in a reconstruction speed of about 160 mm / hour (~1300 steps / second, average step size of 33nm). Based on runs of this and previous inventive techniques using f=1 and integration using the forward Euler method, we infer that the total runtime for axon and spine neck tracing (4.3 meters) in automatic error correction of the reconstructed state from (Motta et al., 2019) on a single node is 27.9 hours, and the initial direction prediction (on ~138,000 spine heads) is 7.6 hours, with a total of 35 GPU node hours required for previous human intervention. For the flood-fill network, 6.964 GPU node hours multiplied by 1000 (NVIDIA P100) GPU (Januszewski et al., 2018). For local shape descriptors, the AcRLSD architecture (“Local Shape Descriptors for Neuron Segmentation.” by Sheridan et al., 2021, bioRxiv, 2021.01.18.427039v1.doi:10.1101 / 2021.01.18.427039v1) took a total of 10.5 hours on 24 NVIDIA V100 GPU nodes, and watershed and clustering took 7.7 CPU node hours on 100 cores. For (Motta et al., 2019), the dense reconstruction including segmentation, clustering, type and synapse prediction, and human skeleton reconstruction processing took 101 hours on 24 nodes, each with 16 CPU cores.For all methods, the number of CPU and GPU hours were multiplied by the ratio of the respective dataset size (Motta et al., 2019), resulting in the computational cost estimates provided in the table below.

[0430]

[0431]

[0432] Table: Calculation cost estimates

[0433] Operation of image processing systems and methods in combination with other techniques (e.g., as an error corrector)

[0434] The processing of 3D EM data or other volumetric image data using the trained AI model disclosed herein can be combined with other reconstruction techniques. For illustration, the techniques disclosed herein can be used to perform connectome reconstruction (or other 3D elongated object tracking) to replace human annotation associated with conventional annotation or tracking techniques. The techniques disclosed herein enable fully automatic connectome reconstruction of the volume of interest when used as an error corrector.

[0435] In embodiments, the image processing systems and methods disclosed herein may be used in conjunction with other reconstruction techniques. For purposes of illustration,

[0436] - can run an automated rebuild process,

[0437] - In response to detecting a missing continuation of an automatic reconstruction, the iterative technique disclosed herein may start at that location and may run until it finds another automatically reconstructed component. From that point, and

[0438] - optionally, in response to finding another automatically reconstructed component, the iterative technique disclosed herein can be run in a backward direction to reconfirm that it ends at the same starting position (for verification purposes), and

[0439] - The skeleton reconstruction of the process defined above can be used for error-corrected automatic reconstruction, optionally in an iterative manner.

[0440] In addition, for illustrative purposes,

[0441] - can run an automated rebuild process,

[0442] - in response to detecting an incorrect continuation or merging error of the automatic reconstruction, the iterative technique disclosed herein may start at several locations around the error and may run until it leaves a bounding box around the error location, and

[0443] - optionally, in response to leaving the bounding box around the error location, the iterative technique disclosed herein can be run in a backward direction to reconfirm that it ends at the same starting location (for verification purposes), and

[0444] - The skeleton reconstruction of the process defined above can be used for error-corrected automatic reconstruction, optionally in an iterative manner and also optionally interleaved with iterations of detection and correction of missing continuations.

[0445] Fig.17 is a flow chart of the method. The method can be automatically executed by an image processing system.

[0446] At step 161, an initial 3D image analysis is performed. An initial 3D image analysis may be performed to obtain a 3D connectome reconstruction. Techniques such as those described in "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." published by A. Motta et al. in 2019 in Science Vol 366. Issue 6469. p.eaay3134 or "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." published by M. Januszewski et al. in 2018 in Nature Methods https: / / doi.org / 10.1038 / s41592-018-0049-4, but not limited thereto, may be used to implement step 161.

[0447] At step 162, the image processing system or method disclosed herein may be used to act as an error corrector.The image processing system or method may be used for neurite (eg, axon) tracing in situations where other automated techniques are unable to annotate the image data.

[0448] To illustrate, using the techniques in "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex" by A. Motta et al., 2019, Science Vol 366. Issue 6469. p. eaay3134, the techniques disclosed herein were applied to all detected spine heads determined using the above techniques, a problem that previously required manual annotation (38% of all detections consumed approximately 900 work hours by the annotator). A precision / recall rate of 89% / 84% for attached spine heads was obtained, close to the precision determined by humans. The starting position detected from the spine head was used, which was also provided to the human annotator. The direction prediction of the technique disclosed herein was then performed. The cyclic inference mode of the technique disclosed herein was applied to the candidate direction with the lowest prediction uncertainty, and tracking was performed until the dendritic trunk defined by the dendritic mask was reached or the path length threshold was exceeded. It is noteworthy that the precision / recall reported above was achieved without retraining the techniques disclosed herein on the cervical spine. Even further improvements can be made by training on a dedicated training set from the cervical spine.

[0449] For the correction of split and merge errors in axon reconstruction, which previously consumed thousands of work hours of annotators, the starting position and direction from the technique of "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." published by A. Motta et al. in 2019 in Science Vol 366. Issue 6469. p.eaay3134 is provided to the method and system disclosed herein. For split error resolution, the technique disclosed herein is applied to small extensions, and each extension is verified by running the method in reverse. Only when the verification is successful does the tracking continue to the next extension until the end of the known axon or data set is reached. The resulting skeleton tracking can then be used similar to human annotation. For merge error resolution, the stopping criterion is based on the bounding box around the merge error, - again similar to humans - and if the verification in the backward direction produces the same skeleton reconstruction with a certain error tolerance, the tracking is accepted. Split and merge error resolution is run iteratively until the remaining error drops below a threshold. The final axon reconstruction for this fully automated approach yielded 12 split errors / mm and 16 merged errors / mm on a representative set of 10 axons (1.7 mm), which is comparable to and close to the error rate of human annotation (5 split errors / mm and 12 merged errors / mm) and outperforms conventional automatic techniques (Motta et al., 2019 before human annotation: 32 split errors / mm and 14 merged errors / mm).

[0450] Direction prediction

[0451] As discussed herein, image processing systems and methods are operable to track an elongated object by iteratively determining a local direction (referred to as a direction of flight) along which the elongated object extends.

[0452] The initial direction (i.e., the direction used to select the subvolume for analysis by the AI ​​model in the first iteration) can be specified by user input. The image processing system is operable to control the GUI so that the corresponding user input can be subsequently used by the slender object tracking technique disclosed herein. Alternatively, when the initial direction cannot be obtained in this way and is not otherwise known a priori, the initial direction can be set to a random direction (where the AI ​​model typically directs the process back to the center line in view of its training). Alternatively, a more targeted approach can be used, in which a Monte Carlo exit method is used on multiple candidate directions. The implementation of direction selection from the candidates can depend on the stability estimated by the image processing system and method. In the selection process, more robust predictions (i.e., directional predictions with less uncertainty) are given priority over predictions with higher uncertainty.

[0453] Fig.18 is a flow chart of method 165. Method 165 may be automatically performed by an image processing system or method.

[0454] At step 166, direction sampling is performed. A set of directions may be sampled, which may be equidistant or approximately equidistant on the surface of the unit sphere.

[0455] At step 167, uncertainty estimates may be calculated for each direction. The calculation of uncertainty estimates may be performed in a variety of ways. For illustration, uncertainty estimates for multiple directions may be determined using a process that includes running a Monte Carlo dropout method, calculating the covariance of a direction estimate (such as Bishop curvature) for each orientation, and determining uncertainty estimates based on the covariance matrix of the direction estimates.

[0456] At step 168 , one or more directions (such as an initial direction of loop extrapolation) may be determined based on the uncertainty estimate.

[0457] Operation & Effect

[0458] The image processing system and method addresses the need for a technique capable of performing automated axon tracing (while also lending itself to other applications).

[0459] The image processing system and method solves axon reconstruction in a manner similar to the task of on-the-fly reconstruction, thereby generating a skeleton with the aid of a sequence of visited points rather than a volumetric (voxel) reconstruction. The image processing system and method provide an automated method for predicting skeleton reconstruction in the field of connectomes directly from raw EM data.

[0460] The image processing system and method provide the possibility of end-to-end optimization.The image processing system and method focus on the essence of neurite reconstruction: following axons over long distances throughout a dataset.

[0461] Image processing systems and methods exploit the fact that neurite reconstruction and error metrics at the skeleton level are more robust than their voxel-based counterparts, which suffer from ambiguity in the affinities of individual voxels, for example in the presence of staining or imaging artifacts.

[0462] The image processing system and method provide a new approach for neurite tracing that is skeleton-based rather than voxel-based and shows good results.

[0463] Fig.19 and 20The operation of the image processing system and method is shown. The image processing system and method are operable to track neurites (e.g., axons) through a 3D volume represented by image data. Tracking here refers to determining a series of points in 3D or determining a continuous curve in 3D, which specifies the trajectory of the neurites in 3D. A series 171 of subvolumes 172-274 is analyzed using the technology disclosed herein. Multiple planes 175 (e.g., 16 planes) spaced apart along the flight direction each include image data determined from the original image data using techniques such as interpolation or projection (e.g., trilinear projection). Information such as the distance from the membrane can be determined and used for, for example, AI model training. In addition, the membrane probability 176 predicted by other methods can be fed to the AI ​​model as an input, but in experiments, this is only helpful in the initial training phase, and the final original EM data 175 provides enough information about the continuity of neurites to make the method successful.

[0464] Fig. 20 A local variable coordinate frame 180, such as a Bishop frame, that can be employed in the techniques disclosed herein is shown. Effectively, the image processing system and method operates by sequentially analyzing sub-volumes of a larger tissue volume, where the sub-volumes represent the field of view seen when flying along an already tracked portion of the centerline of an elongated object in 3D. The Bishop curvature vector 185 can be an output of the AI ​​model and can be integrated to determine a new point along the centerline.

[0465] Techniques for training the AI ​​model are also described in detail. As explained, training the AI ​​model at off-centerline locations and / or orientations provides enhanced stability. Even for complex tasks such as axon tracing or spinal neck applications, error rates as low as about 14 errors / mm are obtained, as described in conjunction with a comparison with conventional techniques above. Fig.21 As shown by curve 190 in , the error rate saturates as the number of training iterations increases. In the example, this article uses approximately 700,000 training iterations for performance evaluation. A larger training set may require more training iterations for saturation.

[0466] The image processing systems and methods disclosed herein can be used for various applications, such as connectome reconstruction ( Fig. 22 ), analysis of biopsy samples ( Fig. 27 ), and / or neuroscience-inspired artificial intelligence ( Fig.28 ).

[0467] Fig. 222 is a flow chart of method 200. Method 200 may be performed by or using an image processing system as disclosed herein. In step 201, 3D EM image acquisition is performed. 3D EM image acquisition may be performed on a biopsy sample. A biopsy sample is a sample that is not implanted back into a living animal or human after imaging.

[0468] In step 202, the trajectory of at least one neurite is determined. For example, by operating several instances of the image processing system in parallel, the trajectories of several neurites may be determined simultaneously. In addition, this step may include more complex logic for iterative reconstruction and may include Figure 5 The module 80 is used as a submodule.

[0469] In step 203, a termination criterion is checked. If the termination criterion is not met, the method returns to step 202. The method may continue until all axons of interest have been traced.

[0470] In this way, several axons may be traced in the volume.Neurite tracing may include tracing axons, for example determining axonal connectomes and / or dendrites.

[0471] At step 204, synapse information is calculated. This can be done using a synapse interface classifier known to those skilled in the art, without limitation.

[0472] At step 205, when the termination criteria are met (eg, when all axons of interest have been traced), the results may be output. The results may be output via a GUI or a data interface, but are not limited thereto.

[0473] Fig.23 is a graph 210 depicting the error rates required for various uses associated with connectome analysis. As shown in region 211, the low error rates and large distances between consecutive errors achieved by the methods and systems of the present invention enable these techniques to be applied to situations not previously easily addressed by conventional computerized techniques.

[0474] Fig.24 is a graph 220 depicting the evolution of the time required to annotate one millimeter as a function of the technology available in recent years. The technology in "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." published by Motta et al. in Science Vol 366. Issue 6469. p.eaay3134, 2019 provides improvements 221. A further, very significant reduction in the annotation consumption rate is provided by the methods and systems of the present invention disclosed herein 222.

[0475] Fig.25 is a graph 230 quantifying the rates of axonal splitting and merging of randomly seeded axons. Results 231 were obtained based on FFN reconstructions of human cortex (A. Shapson-Coe et al., "A connectomics study of a petascale fragment of human cerebral cortex." published in 2021 on BioRxiv, 2021.2005.2029.446289.doi: 10.1101 / 2021.05.29.446289 (2021)). The techniques disclosed herein provide significantly improved results 241 for the two segmentation states analyzed, in which the merging rate is increased. Note that the splitting rate requirement depends on the synaptic rate along the axon: for at least 5 synapses per axon, typical mouse and human axons require splitting rates of less than 40 / mm and 20 / mm, respectively. Therefore, moving automation to these states is desirable for more difficult connectome analyses. The methods and systems disclosed herein improve densely seeded axon reconstruction of large-scale data from mouse cortex. Compared with the present invention technology providing improved results 242, the results 232 were obtained using conventional techniques (undisclosed). For SBEM data, the methods and systems disclosed herein allow for complete replacement of the manual annotation of the analysis employed, providing an improvement to the results 233 obtained using the conventional techniques of Motta et al. supra. The improved results obtained by the methods and systems disclosed herein are shown in 243. The potential additional benefit of the further improvement of the factor 2 that can be obtained from human annotation (black dashed line) is independent of the analysis performed.

[0476] Fig.26 is another graph 260 that quantifies the rates of axon splitting and merging reported for randomly seeded axons. Fig.26 , the technique is applied to 3D EM data obtained using state-of-the-art high-throughput 3D EM imaging methods following multi-beam scanning electron microscopy (multiSEM). Data for mouse cortex and human cortex were obtained. Data 261, 262, 263 correspond to conventional techniques. Data 264 corresponds to data obtained using additional human annotations. Data 271, 272, 273 correspond to data obtained using the methods and systems disclosed herein.

[0477] After automated aggregation, the methods and systems disclosed herein were seeded at the end of automated detection of axon aggregates and used to join the split axon aggregates. As a result, the rate of axon splitting was reduced by 7-fold (42.7 to 6.0 per mm of axon path length), while the rate of merging was only moderately increased (3.3 to 4.5 per mm, as shown in Figure 2). Fig.26262 compared to the bottom data point 272). When directly comparing splitting rates at the same merging error rate (3.9 merging errors per mm axon path length) with higher aggregation, aggregation only resolves 25% of the splitting errors, resulting in 31.9 splits per mm, while the methods and systems disclosed herein resolve 79% of the splitting errors, resulting in 9.0 splits per mm. Thus, in this splitting / merging error range, the methods and systems disclosed herein improve splitting resolution to 3.1 times that of standard aggregation ( Fig.26 , data point 262 versus top data point 272).

[0478] The methods and systems disclosed herein are based on a study of a petascale fragment of human cerebral cortex (150 μm) from A. Shapson-Coe et al., published in 2021 on bioRxiv, 2021.2005.2029.446289.doi:10.1101 / 2021.05.29.446289 (2021). 3 The method and system disclosed in this paper were evaluated in subvolumes of 1000 × 1000 cells per minute and found that application of the method and system disclosed in this paper to axon terminals obtained from dense reconstruction of FFN segmentation (M. Januszewski et al., "High-precision automated reconstruction of neurons with flood-filling networks" published in Nat Methods. 2018 Aug; 15(8): 605-610. doi: 10.1038 / s41592-018-0049-4. Epub 2018 Jul 16. PMID: 30013046) resolved 57% of the splits on a set of random axons traced in the entire subvolume. This reduced the split rate from 65 splits / mm to 28 splits / mm, reaching the split length required for automated connectome analysis, while introducing few new merging errors (such as Fig.26As seen in the data point 261 (bottom), the merging rate increased from 1.7 merges / mm to 3.1 merges / mm compared to data point 271 (bottom), which is a tolerable merging rate for the kind of connectome analysis studied, for example, in the context of SBEM reconstructions of mouse cortex. When comparing the reduction in splitting error at the same merging error rate (C2: 1.7 merges / mm) between FFN and the methods and systems disclosed herein, it was found that FFN-based aggregation from C3 to C2 only resolved 8% of the splitting error of C3. In contrast, aggregation based on the methods and systems disclosed herein resolved 28% of the splitting error of C3 at the same merging error rate as C2, making it 3.5 times more effective than FFN in terms of splitting resolution ( Fig.26 , data point 261 (top) compared to 261 (bottom) and 271 (top).

[0479] In more detail, Fig.26Quantification of axon splitting and merging rates reported for randomly seeded axons is shown. For a flood-filled network (FFN)-based reconstruction of human cortex (A. Shapson-Coe et al., “A connectomics study of a petascale fragment of human cerebral cortex,” 2021, bioRxiv, 2021.2005.2029.446289.doi:10.1101 / 2021.05.29.446289 (2021)), two aggregation states (c2, c3) were analyzed. The change in FFN's clustering parameters from c3 to c2 resolved only 8% of c3 splits in dense axons (FFN data point 261), while the application of the first 40% of the validation trajectories obtained from c3 terminals with the highest certainty using the methods and systems disclosed herein resolved 28% of the split errors, with the same merge error rate as c2, corresponding to a 3.5-fold improved split resolution (data point 271). The application of all validation trajectories obtained from c2 terminals using the methods and systems disclosed herein further reduced the split rate from 65 splits / mm to 28 splits / mm, resolving 57% of c2 split errors, with only a slight increase of 1.4 merge errors per mm. For example, in human cortex, a splitting rate of less than 30 splits / mm (approximately ≥30 μm axon segments) is required to accurately distinguish excitatory from inhibitory axon segments (see S. Loomba et al., 2022, "Connectomic comparison of mouse and human cortex." Science, 2022 Jul 8;377(6602):eabo0924. doi:10.1126 / science.abo0924. Epub 2022 Jul 8. PMID: 35737810), whereas in mouse cortex, at the same splitting rate, ≥10 synapses associated with each axon segment can be analyzed for inhibitory target specificity (A. Motta et al., 2019, "Dense connectomic reconstruction in layer 4 of the somatosensory cortex," Science, eaay3134). cortex" "Dense connectomic reconstruction of layer 4 of somatosensory cortex"). Therefore, introducing automation into this field is essential for most connectomic analyses.For large-scale data from mouse cortex (Si150L4, available at https: / / wklink.org / 7122), the methods and systems disclosed herein improved dense seed axon reconstruction by 7 times, while only slightly increasing the merging error from 3.3 to 4.5 per mm. With a 90% aggregation threshold as a reference, partial application of the trajectories obtained using the methods and systems disclosed herein produced the same merging error rate, while reducing splits 3.1 times more effectively than aggregation. For SBEM data, the methods and systems disclosed herein allow for complete replacement of the human annotations of the employed analysis (A. Motta et al., "Dense connectomic reconstruction in layer 4 of the somatosensory cortex," Science, 2019, eaay3134). The additional benefit of a factor of 2 from manual annotation (black dashed line for data 264) is not relevant to the analysis performed in this study. For fully automated analysis of local circuits, an error rate below 10 per mm is ideal. Such error rates can be obtained using the techniques disclosed herein (data 272, 273).

[0480] In order to evaluate the aggregate splitting and merging errors before and after error correction performed by the methods and systems disclosed herein ( Fig.26 ), detected merge errors that further extended more than 2.2 μm from the ground truth, and manually verified that the heuristic method accurately detected merge errors. For the aggregation before correction performed by the method and system disclosed herein, each merge error was counted as 1 / 2 and divided by the ground truth path length to produce a merge error rate. This is because each merge error usually connects two neurites, and counting them as 1 error per neurite instead of 1 / 2 will overestimate the total amount of merge errors. For the sparse evaluation of error correction performed by the method and system disclosed herein, it is limited to aggregations that overlap with the ground truth, as done in the two multiSEM data sets evaluated, and the additional mergers introduced by the method and system disclosed herein are counted as 1 instead of 0.5, which explains that the mergers from the aggregation do not overlap with the ground truth and are therefore not observable in the sparse evaluation. For split errors, the group of aggregates is limited to those aggregates that are evaluated to overlap with the ground truth by more than 2.5 μm. The overlap threshold is introduced to avoid split errors being dominated by many small aggregates or fragments that are not aggregated along the axonal lamina. Despite this overlap length threshold, approximately 90% of the ground truth is still covered.

[0481] To test whether the correction based on the method and system disclosed herein can also reduce the split error rate when compared under the same circumstances as the merged error rate of the segmentation / aggregation of the two multiSEM datasets, the following strategy was used: only a subset of the trajectories obtained using the method and system disclosed herein was applied, thereby limiting the merged error rate; the prediction uncertainty was quantified for each step performed by the method and system disclosed herein using the Monte Carlo exit method, as previously used to identify the spinal neck direction. The maximum uncertainty was then taken for the step, and the forward and validation trajectories were combined with the minimum of the two uncertainty scores. Therefore, this method is based on the following concept: each direction of the validation trajectory is first scored according to the position with the maximum uncertainty, and then, since the forward and backward trajectories produce the same flight path, the minimum of these two uncertainty scores is taken to quantify the uncertainty of the entire trajectory. Then, the application of the validation trajectory obtained using the method and system disclosed herein was followed up with percentiles of uncertainty up to 0.2, 0.4, 0.6, 0.8, resulting in the partial correction performed by the method and system disclosed herein. The split and merge error rates of these were evaluated, and those with the minimum split error rate that had the same merged error rate as the segmentation / aggregation were added as Fig.26 The midpoint within the split-merge error plane of . Specifically, for the mouse cortex multiSEM dataset, the top 80% validation traces obtained using the methods and systems disclosed herein were applied to the aggregation with an 85% aggregation threshold, resulting in the same merging error rate as the aggregation with a 90% aggregation threshold. Similarly, for the human cortex multiSEM dataset, the top 40% validation traces obtained using the methods and systems disclosed herein were applied to the FFN c3 aggregation, resulting in the same merging error rate as the FFN c2 aggregation (A. Shapson-Coe et al., “A connectomic study of a petascale fragment of human cerebral cortex” published in 2021 on bioRxiv, 2021.2005.2029.446289.doi: 10.1101 / 2021.05.29.446289 (2021)).

[0482] Fig. 27 240. The method 240 may be performed by or using an image processing system as disclosed herein. In step 241, 3D EM image acquisition is performed. The 3D EM image acquisition may be performed on a biopsy sample. A biopsy sample is a sample that is not implanted back into a living animal or human after imaging.

[0483] At step 241, an image acquisition is performed on the biopsy sample. The image acquisition may be a 3D EM image acquisition on the biopsy sample that will be discarded after imaging.

[0484] At step 242, the trajectory of at least one neurite is determined. For example, by operating several instances of the image processing system in parallel, the trajectories of several neurites may be determined simultaneously.

[0485] At step 243, the results of the tracking are output. The results may be output via a GUI (eg, by outputting a visual representation of portions of the connection group via an optical output device) or a data interface, but is not limited thereto.

[0486] The image processing systems and methods disclosed herein may be used in the process of generating, training, and / or operating processing logic (eg, signal processing logic) for controlling at least one physical asset (eg, a machine).

[0487] Fig.28 A system 250 is shown comprising an image processing system operating as disclosed herein. The image processing system may be operable to determine (partial) information about a connectome and / or other structural information of a nervous system of a human or other animal.

[0488] System 250 includes a physical asset 252 (eg, a machine) controlled by a controller 251. Controller 251 executes a machine learning (ML) model to process signal inputs (which may include sensor inputs from sensor measurements) and generate control signals that affect the operation of physical asset 252.

[0489] The ML model executed by the controller 251 may be generated by the control logic generation module 36 of the processing system 30. The control logic generation module 36 of the processing system 30 may use the results of the connection group analysis to at least one of: (i) select or generate an ML model having an ML model architecture that depends on the connection group; (ii) train an ML model having an ML model architecture that depends on the connection group; and / or (iii) deploy the trained ML model that depends on the connection group to the controller 251 as signal processing logic for execution. The controller 251 may execute the ML model that depends on the connection group determined by the processing system 30 for controlling the physical asset 252. The controller 251 may also feed information back to the processing system 30. For illustration, the operation of the controller 251 and / or the asset 252 may be monitored during field use. The processing system 30 may be caused to repeat and / or modify object tracking during field use of the controller 251 and / or the asset 252. In this way, the control logic may be updated during field use.

[0490] Alternatively or additionally, the controller 251 may be coupled to the image acquisition system 20 to control its operation. This may be done in response to the results of the image processing performed by the image processing system 30.

[0491] technology

[0492] Various aspects of the technology that can be implemented in an image processing system and method will be discussed.It should be understood that alternative implementations can be used in an image processing system and method.

[0493] EM image dataset

[0494] The image processing system and method were trained and tested on a SBEM dataset of layer 4 primary somatosensory cortex from 28-day-old mice. The tissue was routinely stained whole mount (KL Briggman et al., "Wiring Specificity in the Direction-Selectivity Circuit of the Retina", Nature, 471(7337):183-88, 2011) and imaged at 11.24×11.24 nm. 2 and a nominal cut thickness of 28 nm. Volume reconstructions of the training set axons were generated using oversegmentation. Oversegmentation was obtained using the technique of Berning et al., "SegEM: Efficient Image Analysis for High-Resolution Connectomics" (Neuron, 87(6): 1193-1206, 2015), with parameters set for whole-cell segmentation of the cortical dataset.

[0495] The image processing systems and methods were also trained and tested on a dataset of barrel cortex from mice acquired with a multibeam scanning electron microscope. The tissue was stained according to the protocol of "Large-Volume En-Bloc 599 Staining for Electron Microscopy-Based Connectomics" published by Y. Hua et al., Nature Communications, 6(August):7923, 2015, with minor modifications and at 4×4 nm. 2 and imaged at a nominal cutting thickness of 35 nm.

[0496] Also with a voxel size of 4×4×33nm3 Image processing systems and methods were evaluated on the multiSEM dataset (A. Shapson-Coe et al., “A connectomic study of a petascale fragment of human cerebral cortex,” BioRxiv, 2021.2005.2029.446289.doi: 10.1101 / 2021.05.29.446289 (2021)), using flood-filling networks for segmentation and aggregation (M. Januszewski et al., “High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks,” Nature Methods, 2018, https: / / doi.org / 10.1038 / s41592-018-0049-4).

[0497] Interpolate the skeleton

[0498] Interpolation can be used to obtain a continuous representation and / or centerline nodes with a desired step size for neurite branches (eg, axon branches) from nodes that are sparsely placed along the central axis.

[0499] The interpolation can be implemented as a 4th order B-spline interpolation (L. Piegl and W. Tiller, The NURBS Book. 2nd ed., Springer Science & Business Media, 1996). For illustration, the curve can be parameterized to obtain a curvature adaptive step size of

[0500]

[0501] Where κ is the curvature, f is the step size factor, d is the default step size if the curvature is zero and f = 1, and p is the physical size of the projection plane. The default step size d can be chosen to be equal to the minimum length of a voxel. This size may depend on the imaging system and may be, for example, 11.24 nm for a particular SBEM data set used in some examples herein.

[0502] The adaptive step size ensures that no voxels with a radius greater than half the size of the projection plane of the CNN's current position are skipped. During inference, the same adaptive step size can be optionally used for the inventive techniques disclosed herein to determine the step size of the integration step for determining subsequent positions and orientations.

[0503] Bishop Framework

[0504] The Bishop frame (RL Bishop, "There Is More than One Way to Frame a Curve", The American Mathematical Monthly, 82(3):246-51, 1975) consists of three orthogonal vectors, the tangent vectors to the curve (also called flight vectors) and two vectors and Bishop Frames and Curves The evolution equation of is as follows:

[0505]

[0506] Where s is the curve The arc length of the curve is used as the parameterized curve parameters, so that Can be a curve Similar equations can be formulated for arbitrary parameterization up to a certain factor. Here, k1 and k2 are respectively related to these normal vectors and The associated Bishop curvature. In the technology disclosed herein, the Bishop curvature corresponds to the steering prediction of the AI ​​model. This curvature vector is defined as curve The curvature κ is The invariance of the curvature vector under certain operations (e.g., flipping and rotation of two normal vectors around the tangent vector, and the corresponding transformation of the Bishop curvature) is used in association with the AI ​​model in the technology disclosed in this article. For illustration, online data augmentation can be performed. The flight direction can also be flipped, which corresponds to time reversal symmetry. On the centerline, this time reversal and tangent vector flipping does not affect the Bishop curvature, while for the flight strategies discussed below, this symmetry can be broken.

[0507] Bishop Frames for Curves The Bishop framework does not require κ≠0 for parameterized curves in three-dimensional Euclidean space. The initial condition, together with the Bishop curvature, uniquely defines the centerline curve This makes Bishop curvature particularly suitable as an AI model output, since this allows one to reconstruct the curve via the evolution equations described above. Since the Bishop framework also exists for parameterized curves in N-dimensional Euclidean space with N ≥ 3, the same mathematical framework can be used and the techniques disclosed herein are similarly applicable to N-dimensional data. Furthermore, parameterized curves in two-dimensional Euclidean space can be considered as a special case of parameterized curves in three-dimensional Euclidean space, so that the methods described here are also applicable.

[0508] Flight strategy

[0509] The flight strategy that can be used during AI model training (used to force convergence back to the centerline when the AI ​​model is applied to a position that is off the centerline) can be derived by considering the following factors: (a) having a corresponding Bishop framework and and the curvature vector The known center line curve The second-order Taylor series expansion of and (b) with a position off the center line And the corresponding Bishop framework as well as and and the curvature vector A second-order Taylor series expansion of the unknown trajectory that converges back to the center line:

[0510]

[0511] For positions off center and the associated deviation direction and the axon centerline The closest position associated with the corresponding centerline is aligned with the direction and the curvature vector The trajectory converging back to the center line can be formulated as minimizing the future distance It is defined by the following formula:

[0512]

[0513] In s c Minimize the distance (of parameter s) The optimization problem can be expressed as:

[0514]

[0515] in

[0516] Defining the projection operator

[0517]

[0518] in Denotes the outer product, and the solution to the minimization problem can be expressed as:

[0519]

[0520] Parameters c is a free parameter of the flight strategy. As described above, the parameter s can be chosen based on the distance of the current deviation from the centerline position from an obstacle, such as a membrane (e.g., a cell membrane) c Thus, obstacle (eg, membrane) avoidance techniques can be implemented during training. Since the above derivation is based on Taylor series expansion, convergence to the centerline is achieved by iteratively applying the above steering.

[0521] Direction prediction using Monte Carlo exit method

[0522] The techniques disclosed in this section can be used, for example, to perform direction prediction to obtain an initial turn prediction. N (e.g., N=256) equidistant or approximately equidistant orientations on the surface of a unit sphere can be computed. The Monte Carlo exit method can be run with M (e.g., M=128) samples. The Bishop curvature prediction from these samples The mean curvature of the Bishop curvature <κ> can be calculated for each orientation i and covariance cov i The uncertainty estimate for the orientation can be determined as the square root of the largest eigenvalue of the covariance matrix divided by the mean curvature,

[0523]

[0524] This quantity is similar to the coefficient of variation.

[0525] Using the cosine similarity of adjacent angles within 30 degrees, a weighted average of the uncertainty can be determined, e.g.

[0526]

[0527] With weight

[0528]

[0529] Weighted averaging provides enhanced stability.

[0530] The orientation with the smallest average uncertainty may be used as the first orientation candidate.

[0531] For the spine head, a second candidate may be selected that is at least 110° from the first candidate and has the smallest average uncertainty from among the remaining orientations.

[0532] Local realignment

[0533] As already discussed above, a local realignment may be performed in the method and / or by the imaging system 20. The local realignment may be performed by the image alignment component 23, or the imaging system 20 may be operable to perform a second local image realignment component in addition to the image alignment component 23.

[0534] The local realignment techniques will be described at least in part in conjunction with 3D EM data. It should be understood that these techniques may also be used in conjunction with other 3D image data or 2D image data sets.

[0535] The local realignment may include the following steps, which are performed to perform local realignment of slices of the image data along a slice direction: determining slice-to-slice offset vectors for multiple slices along the slice direction; identifying valid slice-to-slice offset vectors between the slices; and generating a locally realigned sub-volume using the valid slice-to-slice offset vectors.

[0536] The locally realigned subvolumes may be fed to an AI model input, e.g., to an input layer of a CNN or RNN, or to a filter or other preprocessing that preprocesses the realigned subvolumes before they are processed by the CNN or RNN.

[0537] Determining the slice-to-slice offset vector may include determining an autocorrelation and a cross-correlation of the slices.

[0538] Determining the slice-to-slice offset vectors and / or their validity may include determining a mean and a standard deviation of the associated weighted offset vectors based on a first threshold criterion.

[0539] Identifying a valid slice-to-slice offset vector may include verifying that the determined autocorrelation satisfies a first criterion, such as having a peak at an expected shift (eg, at zero shift).

[0540] Identifying a valid slice-to-slice offset vector may include verifying that the determined autocorrelation peak has a standard deviation that satisfies a second criterion (such as having a standard deviation that satisfies a second threshold criterion, eg, less than or equal to a certain number of pixels).

[0541] Identifying a valid slice-to-slice offset vector may include verifying that a combination of two consecutive autocorrelation and / or cross-correlation peaks satisfies a third criterion (eg, satisfies a third threshold criterion, where the third threshold criterion may optionally depend on surrounding autocorrelations and / or cross-correlations).

[0542] Identifying a valid slice-to-slice offset vector may comprise verifying that the determined cross-correlation satisfies a first cross-correlation criterion which depends on the autocorrelation of the two involved slices (such as the mentioned autocorrelation criterion being valid for both slices).

[0543] Identifying a valid slice-to-slice offset vector may include verifying that the determined cross-correlation normalized with the expected cross-correlation value of the full correlation strength satisfies a second cross-correlation criterion (e.g., the peak value of the cross-correlation normalized with the expected cross-correlation value of the full correlation strength satisfies another threshold criterion, e.g., is equal to or greater than a fourth threshold).

[0544] Identifying a valid slice-to-slice offset vector may include verifying that a standard deviation of the offset vector satisfies a third cross-correlation criterion (such as the standard deviation is equal to or less than a fifth threshold).

[0545] Using valid offset vectors may include identifying one or several slices to be realigned even if the slice-to-slice offset vectors are not considered valid, and in this case, calculating the offset vectors from valid offset vectors related to a neighborhood of these one or several slices.

[0546] Using the valid offset vectors may include generating local realignment and trilinear interpolation sub-volumes.

[0547] Generating the locally realigned and trilinearly interpolated subvolumes may include performing a lower truncation or an upper truncation operation.

[0548] Generating the locally realigned and trilinearly interpolated subvolumes may include performing trilinear interpolation as a sequence of linear interpolations along the basis axes.

[0549] The sequence of linear interpolations may include four linear interpolations along the first dimension, two linear interpolations along the second dimension, and one linear interpolation along the third dimension.

[0550] The locally realigned sub-volumes can be processed to track elongated objects, such as blood vessels or neurites, in 3D.

[0551] In order to efficiently reconstruct neurites that span discontinuities in globally aligned 3D image data, the methods and systems disclosed herein are operable to locally apply an alignment method on a potentially imperfect global alignment, followed by a neurite reconstruction method that applies a 3D subvolume that receives a local realignment, similar to the technique of "Automated Reconstruction of a Serial-Section EMDrosophila Brain with Flood-Filling Networks and Local Realignment." published by PHLi et al. in Preprint. https: / / doi.org / 10.1101 / 605634. The reconstruction within the subvolume then needs to be de-aligned, i.e., projected back to the global reference frame (as discussed in the literature cited by PHLi et al.).

[0552] The following discussion focuses on local realignment of 2D slices across the slice axis, ignoring issues arising from in-plane stitching of 2D mosaics with overlapping fields of view. In a trade-off between accuracy and efficiency, slice-to-slice offset vectors for local realignment are precomputed on a grid over the entire dataset and bilinearly interpolated to produce an approximate per-voxel localized alignment, where the size of the grid used for precomputed offset vectors is a hyperparameter of the local realignment method.

[0553] Precompute slice-to-slice offset vectors for local realignment, from the 8×8 nm 2 Slice z of N × N grayscale EM data with resolution i 2D image data (where, for example, N=512 and the pixel values ​​are in the range [0,255]) can be averaged and divided by 255. Next, the slice (z i ,z i ),(z i ,z i+1 ) and (z i ,z i+2 ) can be determined to correspond to the pixel-level offset vector Furthermore, the mean and standard deviation of the correlation weighted offset vectors are calculated for those vectors that achieve a correlation of at least two thirds of the maximum correlation with respect to the noise level. Specifically, if the correlation exceeds σ i,j +2 / 3(max(π i,j ,σ i,j )-σ i,j ), then consider the offset vector, where the cross-correlation peak between slices i and j is π i,j, and the noise level (i.e., the standard error of the correlation expected under the assumption that pixel intensity values ​​within each slice are independent and identically distributed).

[0554] To filter out irregular offset vectors, several criteria can be used. For illustration, realignment can be achieved such that for a valid slice z i The following conditions must apply to the autocorrelation of : (1) The autocorrelation must peak at zero offset, i.e. and (2) must be sharp enough, with a standard deviation ≤ 2 pixels, and (3) for two consecutive (z i-1 ,z i ) and (z i ,z i+1 ) The total correlation strength value of the cross-correlation, using the expected cross-correlation value e i,j =N 2 std(z i )std(z j ) Normalized peak cross-correlation π i,j , is obtained from the surrounding three cross-correlations (z i-3 ,z i-2 ),(z i-2 ,z i-1 ) and (z i+1 ,z i+2 ), minus the same measure of , must be ≥ -0.095.

[0555] The realignment can be achieved so that in order to i ,z i+1 )'s cross-correlation offset vector To be valid, the following conditions must apply: (1) the autocorrelation condition must be true for both slices, (2) the peak cross-correlation normalized using their respective maximum values ​​must be ≥ 0.18, and (3) the offset vector standard deviation must be ≤ 10 pixels.

[0556] Realignment can be performed so that Invalid locations (according to the above criteria for valid offset vectors) are considered from slice (z) if the following criteria are met: (1) the autocorrelation condition (see above) must be true for both slices, and (2) the offset vector standard deviation must be ≤ 11 pixels. i ,z i+2 )'s offset vector

[0557] This realignment can be implemented so that in order to calculate the invalid offset vector otherwise A valid offset vector can be used and / or Without loss of generality, given a valid offset vector and / or and at least one valid offset vector and / or Set i = 2, the invalid offset vector is calculated as or or if and If both are valid, then it is calculated as the average of these two. Similarly, given a valid offset vector as well as These invalid offset vectors as well as Approximated as Remaining invalid offset vectors is set to

[0558] The offset vector determined as above can be pre-computed for each z-slice on a regular grid The regular grid has a grid spacing of 512 pixels along the in-plane xy axis. To approximate the local offset vector at any location in the global reference frame, bilinear interpolation can be employed. To generate a locally realigned and trilinearly interpolated subvolume that can be fed into an AI model (e.g., to a CNN or another AI model input), the current location can be taken as a reference to calculate the local offset vector, so that The slice at has an offset of 0, where s z = 35 nm is the voxel size along the slice axis, and This is the floor operation. To do this, first, the offset vector of the adjacent slice at the x / y position and All relevant slice-to-slice offset vectors for this subvolume can be used The slice-to-slice offset vector can then be aggregated across the z coordinates of adjacent slices using a cumulative sum, and the slice can be subtracted from the result This produces an offset vector relative to the current position The offset vector, and therefore in the slice An offset of 0 is produced.

[0559] To perform trilinear interpolation while taking into account the offset vector, the method and system can utilize the formula for trilinear interpolation as a sequence of linear interpolations along the principal axes: four linear interpolations along the first dimension, (2) two linear interpolations of the results from (1) along the second dimension, and (3) one linear interpolation of the results from (2) along the third dimension. Specifically, the interpolation of a single voxel can be calculated by first linearly interpolating along the x and y axes independently for the z values ​​of the lower and upper truncations that apply the corresponding offsets along x and y, and then linearly interpolating along the z axis.

[0560] Since the predictions from the CNN on the locally realigned subvolume are only valid within that subvolume, they need to be de-aligned to relate to the global coordinate system of the 3D EM dataset. The methods and systems disclosed herein are operable to achieve this de-alignment, including first integrating the predicted Bishop curvature as previously described into The starting z-slice will then be used as a reference for local realignment of the subvolumes To target z slice The aggregated offset vector is applied to to produce a position in the global coordinate system. In this process, the unit vectors across the Bishop frame can remain unchanged.

[0561] Alignment techniques can be used in association with reassembling 2D images into a consistent 3D image volume with a common global reference frame. This is called alignment. In the case of errors in any of the preparation, slicing, imaging, or reassembly steps, the resulting 3D image data set is typically inconsistent at all locations. The realignment techniques disclosed herein are technically advantageous for many imaging techniques disclosed herein. To illustrate, in order to image neural tissue using 3D EM, the neural tissue is physically decomposed (sliced) along one axis, imaged in 2D with an electron microscope, possibly as a mosaic with overlapping fields of view, and virtually reassembled to ideally produce a globally consistent 3D image data set. For this reassembly and / or after a rough global reassembly, the realignment techniques disclosed herein can be applied. The alignment techniques disclosed herein provide good consistency while avoiding excessive computational runtime. The alignment techniques disclosed herein are beneficial for many applications considered herein.

[0562] Although embodiments have been described in detail, various modifications and variations may be implemented in other embodiments.

[0563] To illustrate, the techniques disclosed herein may be applied to tracking a wide variety of elongated objects.

[0564] The techniques disclosed herein are not limited to tracking elongated objects and / or tracking in 3D, but may be applied to a variety of other use cases, such as tracking objects through a series of image frames (e.g., in a video), tracking objects in a space-time coordinate system, and / or performing image processing for a variety of control actions.

Claims

1. A method for tracking an elongated object (41) in image data (40; 170), the image data (40; 170) being biological or medical image data (40; 170), the method comprising the following steps performed by an image processing system (30): The positions of a series of points (60) along the elongated object (41) are determined in three-dimensional 3D space to track the elongated object (41) in 3D in the image data (40; 170), wherein: Determining the positions of the series of points (60) comprises an iterative process comprising: inferring a steering prediction using an artificial intelligence (AI) model (110; 130), the steering prediction being dependent on the direction of the elongated object (41), optionally the centerline (50) of the elongated object, extending from a previously determined point along the elongated object (41), the AI ​​model (110; 130) comprising an input layer and an output layer, the input layer being operable to receive an AI model input based on pixel or voxel values ​​of the image data (40; 170), the output layer being operable to provide an AI model (110; 130) output as or indicative of the steering prediction at each iteration of the iterative process; using at least the turn prediction, calculating a next point (62-65) in the series of points (60), the next point being subsequent to the previously determined point (61-64) in the series of points (60); and The inferring step and the calculating step are iteratively repeated until a termination criterion is met, thereby determining the positions of the series of points (60).

2. The method according to claim 1, wherein: The AI ​​model includes: A convolutional neural network (CNN), comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer, or Recurrent Neural Network (RNN); wherein the AI ​​model input input to the input layer in an iteration of the iterative process represents a sub-volume (91, 93; 172-174) of the tissue volume represented by the image data (40; 170), wherein the AI ​​model input comprises pixel or voxel values ​​representing the sub-volume of the image data (40; 170), or obtained by multivariate interpolation of pixel or voxel values ​​representing said sub-volume of said image data (40; 170); and Therein, the orientation of the subvolume (91, 93; 172-174) depends on the steering prediction determined in a previous iteration of the iterative process.

3. The method according to claim 2, wherein: The steering prediction depends on the second order derivative of the centerline (50) of the elongated object (41).

4. The method according to claim 1, wherein: The AI ​​model is A convolutional neural network (CNN), comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer, or Recurrent Neural Network (RNN); wherein the AI ​​model input input to the input layer in an iteration of the iterative process represents a sub-volume (91, 93; 172-174) of the tissue volume represented by the image data (40; 170); The steering prediction is a curvature value relative to a reference frame associated with the subvolume such that it produces a position offset for iteratively determining a series of points located on a centerline (50) of the elongated object; and In each iteration following a previous iteration, the respective subvolume being processed is determined to have a rotational orientation determined by a direction of extension of the elongated object determined by the steering prediction obtained in the previous iteration.

5. The method according to claim 1, wherein: One or more of the following applies: Determining the next point in the series of points (60) includes: integrating the turn prediction; Determining the next point in the series of points (60) includes: integrating the turn prediction, wherein integrating the turn prediction includes: integrating curvature in a Bishop reference frame; The steering prediction is related to a second order derivative of the centerline (50) of the elongated object (41).

6. The method according to claim 1 or 5, wherein: The AI ​​model input input to the input layer in an iteration of the iterative process represents a sub-volume (91, 93; 172-174) of the tissue volume represented by the image data (40; 170), the sub-volume (91, 93; 172-174) depending on The position and / or orientation and / or turning prediction determined in one or several previous iterations of the iterative process.

7. The method according to claim 6, wherein: The orientation of the subvolumes (91, 93; 172-174) depends on the previous steering prediction.

8. The method according to claim 6 or 7, wherein: One of the following applies: The center of the subvolume (91, 93; 172-174) is offset relative to a previously determined point by an offset; The center of the subvolume (91, 93; 172-174) is offset relative to a previously determined point by an offset, wherein the offset depends on the steering prediction determined in the previous iteration.

9. The method according to any one of claims 6 to 8, further comprising: Multivariate interpolation is performed on pixel or voxel values ​​of the image data (40, 170) to determine the AI ​​model input.

10. The method according to claim 9, wherein: The multivariate interpolation includes: performing a weighted average of the pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data (40; 170), or Trilinear projection.

11. The method according to any one of the preceding claims, further comprising: An initialization of the iterative process is performed, wherein the initialization depends on whether an initial direction is available.

12. The method according to claim 11, wherein: When the initial direction is not available, the method includes determining the initial direction for an initial iteration of the iterative process.

13. The method according to claim 12, wherein: Determining the initial direction includes: Determine uncertainty estimates for the predictions in various directions, and From each direction, the direction with the smallest estimated uncertainty in the corresponding prediction is selected and the steering prediction is made.

14. The method according to claim 13, wherein: The uncertainty estimate was determined using a Monte Carlo dropout method.

15. A method according to any one of the preceding claims, wherein: One of the following applies: The AI ​​model (110; 130) includes a convolutional neural network (CNN) or a recurrent neural network (RNN). The AI ​​model (110; 130) includes the input layer, a plurality of convolutional layers, a dropout layer, and the output layer. Among them, the AI ​​model (110; 130) includes one or more of the following: pooling layer, attention module, self-attention module, void convolution, normalization layer, batch normalization layer, transformer, and recurrent layer.

16. A method according to any one of the preceding claims, in, One, more or all of the following apply: The output layer is a linear layer; The plurality of convolutional layers include one or more strided convolutional layers; The AI ​​model (110; 130) is or includes a fully convolutional network; The AI ​​model (110; 130) includes one or more nonlinearities; The AI ​​model (110; 130) includes an exponential linear unit ELU; The AI ​​model (110; 130) includes a rectified linear unit ReLU; The AI ​​model (110; 130) includes a scaled exponential linear unit SELU; The AI ​​model (110; 130) includes a Gaussian error linear unit GELU.

17. A method according to any one of the preceding claims, wherein: The elongated object (41) is a neurite.

18. The method according to any one of claims 1 to 16, wherein: The elongated object (41) is a blood vessel.

19. A method according to any one of the preceding claims, wherein: The elongated object (41) has a surface extending in a partially cylindrical manner around the center line (50).

20. The method according to any one of the preceding claims, further comprising: The AI ​​model (110; 130) is trained.

21. The method according to claim 20, wherein: One or both of the following apply: The AI ​​model (110; 130) is trained using supervised learning; Training the AI ​​model (110; 130) includes: gradient-based updating of parameters of the AI ​​model (110; 130).

22. The method according to claim 20 or 21, wherein: Training the AI ​​model (110; 130) includes: training the AI ​​model (110; 130) to comply with a flight strategy, wherein the iterative process converges to the elongated object (41) when starting from or reaching a position in the image data (40; 170) that leaves the elongated object (41).

23. The method according to claim 22, wherein: The convergence distance of the flight strategy determines the speed at which the iterative process converges back to the elongated object (41).

24. The method according to claim 23, wherein: One or both of the following apply: The convergence distance is a dynamic parameter that changes during training and / or inference of the AI ​​model (110; 130); During training of the AI ​​model (110; 130), the convergence distance is set as a function of the distance from the physiological boundary.

25. The method according to claim 24, wherein: During training of the AI ​​model (110; 130), the convergence distance is set as a function of the distance from the physiological boundary, wherein the convergence distance is a monotonically increasing function of the distance from the physiological boundary.

26. A medical image processing method, comprising: receiving medical image data (40; 170) comprising a 3D image or a set of 2D images of tissue, the tissue including neural tissue; performing a method according to any one of the preceding claims to track an elongated object (41) in the medical image data (40; 170); and Output one or more of the following: a visual representation of the series of points (60) or the centerline (50); a machine-readable representation of the series of points (60) or the centerline (50); a control signal dependent on the series of points (60) located along the centerline (50); A machine learning (ML) model (110; 130) having a structure that depends on the series of points (60) located along the center line (50).

27. A connectomics method comprising performing the elongated object (41) tracking method according to any one of claims 1 to 25.

28. The connectomics method of claim 27, further comprising: A synaptic interface classifier is executed, and a connectome is determined using at least an output of the elongated object tracking method and an output of the synaptic interface classifier.

29. A method of analyzing a biopsy sample, comprising: acquiring image data of the biopsy sample (40; 170); processing the image data (40; 170) using a method according to any one of claims 1 to 25; and The result of the processing of the image data (40; 170) is output via an interface.

30. A method of generating signal processing logic for controlling a physical asset, the method comprising: Performing the elongated object tracking method according to any one of claims 1 to 25; using, by at least one computing system, outputs of the elongated object tracking method to: Select or generate a machine learning ML model (110; 130) architecture; Training a ML model (110; 130) having the ML model (110; 130) architecture; deploying the trained ML model (110; 130) as the signal processing logic to a control device for execution; An image acquisition system (20) is controlled, the image acquisition system acquiring the image data (40; 170).

31. An image processing system (30) for tracking an elongated object (41) in image data (40; 170), the image processing system (30) comprising: An interface operable to receive the image data (40; 170), the image data (40; 170) is biological or medical image data (40; 170), and At least one processing device operable to: Determining positions of a series of points (60) along the elongated object (41) in a three-dimensional 3D space to 3D track the elongated object (41) in the image data (40; 170), wherein determining the positions of the series of points (60) comprises an iterative process, the iterative process comprising: inferring a steering prediction using an artificial intelligence (AI) model (110; 130), the steering prediction being dependent on the direction of the elongated object (41), optionally a centerline (50) of the elongated object (41), extending from a previously determined point along the elongated object (41), the AI ​​model (110; 130) comprising an input layer and an output layer, the input layer being operable to receive an AI model input based on pixel or voxel values ​​of the image data (40; 170), the output layer being operable to provide an AI model (110; 130) output as or indicative of the steering prediction at each iteration of the iterative process; using at least the turn prediction, calculating a next point in the series of points (60), the next point being subsequent to the previously determined point in the series of points (60); and The inferring step and the calculating step are iteratively repeated until a termination criterion is met, thereby determining the positions of the series of points (60).

32. The image processing system (30) according to claim 30, wherein: The image processing system is operable to: The AI ​​model includes: A convolutional neural network (CNN), comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer, or Recurrent Neural Network (RNN); the AI ​​model input input to the input layer in an iteration of the iterative process represents a sub-volume (91, 93; 172-174) of the tissue volume represented by the image data (40; 170), wherein the AI ​​model input comprises pixel or voxel values ​​representing the sub-volume of the image data (40; 170), or obtained by multivariate interpolation of pixel or voxel values ​​representing said sub-volume of said image data (40; 170); The orientation of the subvolume (91, 93; 172-174) depends on the steering prediction determined in a previous iteration of the iterative process.

33. The image processing system (30) according to claim 32, wherein: The image processing system is operable such that the steering prediction depends on a second order derivative of a centre line (50) of the elongate object (41).

34. The image processing system (30) according to claim 31, wherein: The image processing system is operable to: The AI ​​model is A convolutional neural network (CNN), comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer, or Recurrent Neural Network (RNN); The AI ​​model input input to the input layer in an iteration of the iterative process represents a sub-volume (91, 93; 172-174) of the tissue volume represented by the image data (40; 170); The steering prediction is a curvature value relative to a reference frame associated with the subvolume such that it produces a position offset for determining in an iterative manner a series of points located on a centerline (50) of the elongated object; as well as In each iteration following a previous iteration, the respective subvolume being processed is determined to have a rotational orientation determined by the extension direction of the elongated object determined by the steering prediction obtained in the previous iteration.

Images