Method and image processing system for tracing elongated objects in image data

Abstract

A method and system for tracing elongated objects in image data is disclosed. The method and system may operate, for example, for tracing neurites. An artificial intelligence model is used to infer a manipulation prediction. The manipulation prediction is processed to determine the next in a series of points (60) on a centerline (50) of the elongated object.

Description

[Technical Field]

[0001] IMAGE PROCESSING METHOD AND SYSTEM FIELD OF THE INVENTION Embodiments of the present invention relate to a method and system for tracing elongated objects, such as neurites, in image data representing a three-dimensional tissue volume. [Background technology]

[0002] Dense mapping of neuronal wiring and its synaptic connections to improve our understanding of computation in the brain is a major goal of high-resolution connectomics. 3 Although the acquisition of large three-dimensional electron microscopy (3D-EM) datasets is feasible, the reconstruction of axon and spine necks remains a significant challenge.

[0003] Human annotation of 3D image data can provide a low error rate, but is often prohibitively time-consuming. Therefore, there is a need for computer-assisted or computer-implemented techniques capable of tracing elongated objects in image data representing three-dimensional (3D) tissue volumes. Reconstruction of neurites by sparse tracing along their centerlines ("skeletonization") has resulted in an approximately 50-fold increase in annotation speed (see, e.g., M. Helmstaedter et al., 2011, "High-Accuracy Neurite Reconstruction for High-Throughput Neuroanatomy." Nature Neuroscience 14(8):594 1081-88). This approach builds on recent large-scale reconstructions (see, e.g., K. Eichler et al., 2017, "The Complete Connectome of a Learning and Memory Centre in an Insect Brain." Nature 548(7666):175-82). Skeletal reconstruction has been combined with automatically obtained segmentation to yield volume reconstructions (see, e.g., Helmstaedter et al., 2013, "Connectomic Reconstruction of the Inner Plexiform Layer in the Mouse Retina." Nature 500(7461):168-74; Dorkenwald et al., 2017, "Automated Synaptic Connectivity Inference for Volume Electron Microscopy." Nature Methods 14(4):435-42; and Staffler et al., 2017, "SynEM, Automated Synapse Detection for Connectomics." ELife 6(July):e26414).

[0004] Further speedups have been achieved by skeletonizing neurites using a user interface that presents the annotator with an egocentric view of the 3D-EM volume ("fly-around mode") (see, e.g., Boergens et al., 2017, "WebKnossos: Efficient Online 3D Data Annotation for Connectomics." Nature Methods 14(7): 691-94).Other approaches aim to reduce the number of locations requiring human annotation through automated reconstruction of neurites and subsequent detection of reconstruction errors (SMPlaza, 2016, "Focused Proofreading to Reconstruct Neural Connectomes from EM Images at Scale." In Deep Learning and Data Labeling for Medical Applications, pp. 249-58, Springer, Cham; Y. Meirovitch et al., 2016, "A Multi-Pass Approach to Large 620 Scale Connectomics." ArXiv E-Prints, December, http: / / arxiv.org / abs / 1612.02120; D. Rolnick et al., 2017, "Morphological Error Detection in 3D Segmentations." ArXiv E-Print, https: / / arxiv.org / pdf / 1705.10882.pdf; J. Zung et al., 2017, "An Error Detection and Correction Framework for "Connectomics.", NIPS, 6818-29; K. Dmitriev et al., 2018, "Efficient Correction for EM Connectomics with Skeletal Representation.", In British Machine Vision Conference (BMVC), http: / / bmvc2018.org / contents / papers / 0064.pdf; and A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the "Somatosensory Cortex.", Science, Vol 366, Issue 6469, p.eaay3134).

[0005] Further exemplary techniques for tracing objects are described in Jelmer M. Wolrterink et al., "Coronary artery centerline extraction in cardiac CT angiography using a CNN-based orientation classifier," MEDICAL IMAGE ANALYSIS, OXFORD UNIVERSITY PRESS, OXFORD, GB, vol. 51, October 22, 2018, (2018-10-22), pages 46-60, ISSN: 1361-8415, DOI: 10.1016 / J.MEDIA.2018.10.005; Rouchen Gao et al., "Joint Coronary Centerline Extraction And Lumen Segmentation From Ccta Using CNNTracker And Vascular Graph Convolutional Network," 2021 IEEE 18TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING (ISBI), IEEE, April 13, 2021 (2021-04-13), pages 1897-1901, DOI:10.1109 / ISBI4821120219433764; and Tianhong Dai et al., "Deep Reinforcement Learning for Subpixel Neural Tracking," Proceedings of Machine Learning Research, 2019, pages 130-150, http: / / proceedings.mlr.press / v102 / dai19a / dai19a.pdf. Summary of the Invention

[0006] There remains a need for improved techniques in image processing, and in particular for techniques that are capable of tracing elongated objects such as neurites in a computationally efficient manner and / or with a low error rate.

[0007] According to embodiments of the present invention, there are provided methods and systems as recited in the independent claims. The dependent claims define preferred embodiments.

[0008] A method for tracing an elongated object in image data, where the image data is biological or medical image data, may include tracing a three-dimensional (3D) elongated object in the image data by determining, with an image processing system, the positions of a series of points along the elongated object in 3D space, where the series of points may be determined such that the series of points lie along an approximation of a centerline of the elongated object.

[0009] Determining the locations of the series of points may involve an iterative process.

[0010] The iterative process may include using an artificial intelligence (AI) model to infer a manipulation prediction that depends on (may be indicative of) a direction along the elongated object (e.g., along the centerline or approximation of the centerline) that the elongated object extends from a previously determined point, the AI ​​model comprising an input layer operative to receive AI model input based on pixel or voxel values ​​of the image data, and an output layer operative to provide, at each iteration of the iterative process, an AI model output that is or is indicative of the manipulation prediction.

[0011] The iterative process may include using at least the operational prediction to calculate a next point in the series of points, the next point following a previously determined point in the series of points.

[0012] The iterative process may involve determining a series of points by iteratively repeating steps of estimation and calculation until a termination criterion is met.

[0013] According to a preferred embodiment, the AI ​​model may comprise a convolutional neural network (CNN) or a recurrent neural network (RNN) comprising an input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and an output layer, wherein the AI ​​model inputs input to the input layer in an iteration of the iterative process represent subvolumes of the tissue volume represented by the image data, the AI ​​model inputs comprising pixel or voxel values ​​representing the subvolumes of the image data or obtained by multivariate interpolation of pixel or voxel values ​​representing the subvolumes of the image data, and the orientation of the subvolumes depends on an operation prediction determined in a previous iteration of the iterative process.

[0014] According to a preferred embodiment, the manipulation prediction may depend on the second derivative of the elongated object, for example the centerline of the elongated object.

[0015] According to another preferred embodiment, the AI ​​model may be a convolutional neural network (CNN) or a recurrent neural network (RNN) comprising an input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and an output layer, wherein the AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes of the tissue volume represented by the image data, and the manipulation prediction is a curvature value relative to a reference system given in relation to the sub-volume, such that the manipulation prediction results in position offsets that iteratively determine a series of points located on the centerline of the elongated object, and wherein in each iteration following the previous iteration, each sub-volume processed has a rotational orientation determined by the extension direction of the elongated object as determined from the manipulation prediction obtained in the previous iteration.

[0016] The method may include outputting the results of the image processing via a graphical user interface or a data interface.

[0017] Determining the next point in the series of points may include integrating the maneuver prediction.

[0018] The manipulation prediction may depend on the second derivative of the centerline of the elongated object.

[0019] The AI ​​model inputs input to the input layer in an iteration of the iterative process may represent sub-volumes of the tissue volume represented by the image data.

[0020] The sub-volumes may depend on a previous operation prediction determined in a previous iteration of the iterative process and / or on a point determined in a previous iteration of the iterative process.

[0021] The orientation of the sub-volume may depend on prior manipulation predictions and / or on the uncertainty of the AI ​​model's predictions in multiple orientations.

[0022] The orientation of a sub-volume may depend on the direction in which an elongated object extends in or within the sub-volume.

[0023] The orientation of the sub-volume may be approximately aligned with the direction in which an elongated object extends in or within the sub-volume.

[0024] The centre of the sub-volume may be shifted relative to the previously determined point by an offset, further optionally the offset depending on a manipulation prediction determined in a previous iteration.

[0025] The method may involve multivariate interpolation (such as averaging) of pixel or voxel values ​​of the image data to determine the AI ​​model inputs.

[0026] Multivariate interpolation may comprise a weighted average of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0027] The multivariate interpolation may include a trilinear projection.

[0028] The method may include initializing the iterative procedure.

[0029] The initial setting may depend on whether an initial orientation is available a priori.

[0030] When an initial orientation is not available, the method may include determining an initial orientation to be used in an initial iteration of the iterative process.

[0031] Determining the initial orientation may include determining an uncertainty estimate for the AI ​​model predictions for various orientations, and selecting, from among the various orientations, the orientation for which the uncertainty estimate is smallest.

[0032] The uncertainty estimates may be determined using Monte Carlo dropout.

[0033] The AI ​​model may comprise a convolutional neural network (CNN).

[0034] The AI ​​model may comprise a recurrent neural network (RNN).

[0035] The AI ​​model may include an input layer, multiple convolutional layers, a dropout layer or several dropout layers, and an output layer.

[0036] The output layer may be a linear layer.

[0037] The multiple convolutional layers may comprise one or several strided convolutional layers.

[0038] The AI ​​model may be or may comprise a fully convolutional network.

[0039] The AI ​​model may include one or more of a pooling layer, an attention module, a self-attention module, an atlas convolution, a normalization layer, a batch normalization layer, a transformer, and a recurrent layer.

[0040] An AI model can have one or several nonlinearities.

[0041] The AI ​​model may comprise an exponential linear unit (ELU).

[0042] The AI ​​model may comprise a rectified linear unit (ReLU).

[0043] The AI ​​model may comprise a scaled exponential linear unit (SELU).

[0044] The AI ​​model may comprise a Gaussian Error Linear Unit (GELU).

[0045] The image data may represent a volumetric representation of the tissue.

[0046] The image data may comprise a set of three-dimensional (3D) images or two-dimensional (2D) images.

[0047] The method may further include acquiring image data.

[0048] The image data can be obtained using electron microscopy.

[0049] The image data may be obtained using optical microscopy, for example, light microscopy.

[0050] The image data may be acquired using positron emission tomography (PET), magnetic resonance (MR) imaging, x-ray tomography, or another medical imaging modality.

[0051] The elongated object may be a neurite.

[0052] The elongated object may be a blood vessel.

[0053] An elongated object may have surfaces that extend around a centerline in a locally cylindrical manner.

[0054] The method can be used to trace axons.

[0055] The elongated object may be an axon.

[0056] The elongated object may be a spine neck.

[0057] The method may be performed multiple times in parallel and / or serially to trace multiple axons and / or dendrites.

[0058] The method may include training an AI model.

[0059] The AI ​​model can be trained using supervised learning.

[0060] Training the AI ​​model may include gradient-based updating of the AI ​​model parameters.

[0061] Training the AI ​​model may include training the AI ​​model to follow a flight policy that causes the iterative process to converge to an elongated object (e.g., along the centerline or an approximation of the centerline) when starting from an initial location in the image data away from the centerline.

[0062] The convergence distance of the flight policy may determine how quickly the iterative process converges back to the centerline.

[0063] The convergence distance can be a dynamic parameter that is varied during training and / or inference of the AI ​​model.

[0064] The convergence distance can be set as a function of distance from physiological boundaries during training of the AI ​​model.

[0065] The convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0066] The method may include outputting one or more of: a visual representation of the series of points or centerline; a machine-readable representation of the series of points or centerline; a control signal dependent on the series of points located along the centerline; an uncertainty in the predicted control signal; and an indication of which stopping criterion / criteria terminated the iterative process.

[0067] The output may be via a user interface, such as a graphical user interface, or a data interface. The medical imaging method or medical image processing method includes receiving medical image data representing a tissue volume and comprising a set of 3D or 2D images of tissue comprising nervous tissue, performing an image processing method to trace elongated objects in the medical image data, and outputting one or more of: a visual representation of a series of points or a centerline, a machine-readable representation of the series of points or the centerline, a control signal dependent on the series of points located along the centerline, and a machine learning (ML) model having an architecture dependent on the series of points located along the centerline.

[0068] The connectomics method involves performing a method for tracing elongated objects.

[0069] A method for analyzing a biopsy sample may include obtaining image data of the biopsy sample, processing the image data using a method for processing image data, and outputting results of the processing of the image data via an interface.

[0070] Methods of analyzing biopsy samples can be performed to determine a portion of the connectome.

[0071] A biopsy sample is a sample that has not been implanted into the animal or human body.

[0072] A method for generating signal processing logic for controlling a physical asset may include performing a method for processing image data; and, by at least one computing system, using output of the elongated object tracing method to select or generate a machine learning (ML) model architecture; training an ML model having the ML model architecture; and deploying the trained ML model to a control device for execution as signal processing logic.

[0073] An image processing system for tracing elongated objects in image data may comprise an interface operative to receive the image data, the image data being biological or medical image data.

[0074] The image processing system may include at least one processing device that operates to determine the positions of a series of points along an elongated object in three-dimensional (3D) space (e.g., along a centerline or an approximation of a centerline) to trace the 3D elongated object in the image data, and determining the positions of the series of points may include an iterative process.

[0075] The iterative process may include using an artificial intelligence (AI) model to infer a manipulation prediction that depends on (or may be indicative of) a direction in which the elongated object extends from a previously determined point along the elongated object, the AI ​​model comprising an input layer operative to receive AI model input based on pixel or voxel values ​​of the image data, and an output layer operative to provide, at each iteration of the iterative process, an AI model output that is or is indicative of the manipulation prediction.

[0076] The iterative process may include using the operational prediction to calculate the next point in the series of points, the next point following a previously determined point in the series of points.

[0077] The iterative process may involve determining a series of points by iteratively repeating steps of estimation and calculation until a termination criterion is met.

[0078] According to a preferred embodiment, the image processing system may operate as follows: the AI ​​model may comprise a convolutional neural network (CNN) or a recurrent neural network (RNN) comprising an input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and an output layer; the AI ​​model inputs input to the input layer in an iteration of the iterative process represent subvolumes of a tissue volume represented by the image data; the AI ​​model inputs comprise pixel or voxel values ​​representing the subvolumes of the image data or are obtained by multivariate interpolation of pixel or voxel values ​​representing the subvolumes of the image data; and the orientation of the subvolumes depends on an operation prediction determined in a previous iteration of the iterative process.

[0079] According to a preferred embodiment, the manipulation prediction may depend on the second derivative of the elongated object, for example the centerline of the elongated object.

[0080] According to another preferred embodiment, the image processing system may operate as follows: the AI ​​model may be a convolutional neural network (CNN) or a recurrent neural network (RNN) comprising an input layer, a plurality of convolutional layers, a dropout layer or several dropout layers, and an output layer, wherein the AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes of the tissue volume represented by the image data, and the manipulation prediction is determined such that the manipulation prediction results in position offsets that iteratively determine a series of points located on the centerline of the elongated object, with respect to a reference system given in relation to the sub-volumes, and wherein in each iteration following the previous iteration, each sub-volume to be processed has a rotational orientation determined by the extension direction of the elongated object as determined from the manipulation prediction obtained in the previous iteration.

[0081] The image processing system may be operable to perform methods according to embodiments.

[0082] The image processing system may include an image acquisition system that operates to capture image data.

[0083] The image acquisition system may be operable to perform 3D electron or light microscopy.

[0084] The image processing system may be operable to output the results of the image processing via a graphical user interface or a data interface.

[0085] The image processing system may be operable such that determining the next point in the series of points may include integrating a steering prediction.

[0086] The image processing system may operate such that the AI ​​model inputs input to the input layer in an iteration of the iterative process may represent sub-volumes of the tissue volume represented by the image data.

[0087] The image processing system may operate such that the sub-volume may depend on a previous operation prediction determined in a previous iteration of the iterative process and / or on a point determined in a previous iteration of the iterative process.

[0088] The image processing system may operate such that the orientation of the sub-volume may depend on a prior manipulation prediction.

[0089] The image processing system may operate such that the orientation of a sub-volume may depend on the direction in which an elongated object extends in or within the sub-volume.

[0090] The image processing system may be operable such that the orientation of the sub-volume may be approximately aligned with the direction in which an elongated object extends in or within the sub-volume.

[0091] The image processing system may be operated such that the center of the sub-volume may be shifted by an offset relative to a previously determined point.

[0092] The image processing system may operate such that the offset may depend on a steering prediction determined in a previous iteration.

[0093] The image processing system may involve multivariate interpolation (such as averaging) of pixel or voxel values ​​of image data to determine AI model inputs.

[0094] The image processing system may be operable such that the multivariate interpolation may comprise a weighted average of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​in a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0095] The image processing system may be operable to perform a projection that determines pixel or voxel values ​​in a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0096] The image processing system may be operable to perform an initialization of the iterative procedure.

[0097] The image processing system may operate such that the initial setting may depend on whether an initial operation prediction is available a priori.

[0098] The image processing system may be operable such that when an initial orientation is not available, the image processing system determines an initial orientation to be used in an initial iteration of the iterative process.

[0099] The image processing system may be operable such that determining an initial orientation may include determining uncertainty estimates for various orientations and selecting, from among the various orientations, the orientation for which the uncertainty estimate is smallest.

[0100] The image processing system may be operated such that the uncertainty estimates may be determined using Monte Carlo dropout.

[0101] The AI ​​model may comprise a convolutional neural network (CNN).

[0102] The AI ​​model may comprise a recurrent neural network (RNN).

[0103] The AI ​​model may include an input layer, multiple convolutional layers, a dropout layer or several dropout layers, and an output layer.

[0104] The output layer may be a linear layer.

[0105] The multiple convolutional layers may comprise one or several strided convolutional layers.

[0106] The AI ​​model may be or may comprise a fully convolutional network.

[0107] The AI ​​model may include one or more of a pooling layer, an attention module, a self-attention module, an atlas convolution, a normalization layer, a batch normalization layer, a transformer, and a recurrent layer.

[0108] The AI ​​model may comprise an exponential linear unit (ELU).

[0109] The AI ​​model may comprise a rectified linear unit (ReLU).

[0110] The AI ​​model may comprise a scaled exponential linear unit (SELU).

[0111] The AI ​​model may comprise a Gaussian Error Linear Unit (GELU).

[0112] The image data may represent a volumetric representation of the tissue.

[0113] The image data may comprise a 3D image or a set of 2D images.

[0114] The image processing system may be operable to acquire image data.

[0115] The image data can be obtained using electron microscopy.

[0116] The image data may be obtained using optical microscopy, for example, light microscopy.

[0117] The image data may be acquired using positron emission tomography (PET), magnetic resonance (MR) imaging, or another medical imaging modality.

[0118] The elongated object may be a neurite.

[0119] The elongated object may be a blood vessel.

[0120] An elongated object may have surfaces that extend around a centerline in a locally cylindrical manner.

[0121] The image processing system may operate to train an AI model.

[0122] The AI ​​model can be trained using supervised learning.

[0123] The image processing system may operate such that training the AI ​​model may include gradient-based updating of the AI ​​model parameters.

[0124] The image processing system may be operable such that training the AI ​​model may include training the AI ​​model to follow a flight policy that causes an iterative process to converge to the centerline when starting from an initial location in the image data away from the centerline.

[0125] The image processing system may operate such that the convergence distance of the flight policy may determine how quickly the iterative process converges back to the centerline.

[0126] The image processing system may operate such that the convergence distance may be a dynamic parameter that is varied during training of the AI ​​model.

[0127] The image processing system may be operated such that the convergence distance may be set as a function of distance from the physiological boundary during training of the AI ​​model.

[0128] The image processing system may be operated such that the convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0129] The image processing system may be operable to output one or more of a visual representation of the series of points or centerline, a machine-readable representation of the series of points or centerline, and a control signal dependent on the series of points located along the centerline.

[0130] The image processing system may include a user interface, such as a graphical user interface, or a data interface for providing output.

[0131] A training method for training an AI model is also disclosed, the AI ​​model comprising an input layer operative to receive an AI model input based on pixel or voxel values ​​of image data representing a tissue volume, and an output layer operative to provide an AI model output that is or is indicative of a manipulation prediction, the manipulation prediction being dependent on (or may be indicative of) a direction along which an elongated object extends (e.g., along a centerline or an approximation of a centerline).

[0132] The training method may include training an AI model to predict the direction in which the centerline of an elongated object extends.

[0133] The training method may include training an AI model to output a derivative (e.g., first, second, or higher order derivative) of the direction along which the centerline of the elongated object extends.

[0134] The training method can be a gradient-based supervised learning method.

[0135] In the training method, the AI ​​model may be trained using supervised learning.

[0136] The training method may include gradient-based updating of the AI ​​model parameters.

[0137] The training method may include training the AI ​​model to follow a flight policy that causes the iterative process to converge to the centerline when starting from or reaching a location in the image data that is away from the centerline.

[0138] The convergence distance of the flight policy may determine how quickly the iterative process converges back to the centerline.

[0139] The convergence distance can be a dynamic parameter that is varied during training and / or inference of the AI ​​model.

[0140] The convergence distance may depend on the distance from the obstacle.

[0141] The obstacle may be a physiological boundary, such as a cellular boundary.

[0142] The convergence distance can be set as a function of distance from physiological boundaries during training of the AI ​​model.

[0143] The convergence distance may be a monotonically increasing function of the distance from the physiological boundary.

[0144] The AI ​​model inputs input to the input layer may represent sub-volumes of the tissue volume represented by the training data, and the training method may include multivariate interpolation of pixel or voxel values ​​of the image data to determine the AI ​​model inputs.

[0145] Multivariate interpolation may comprise a weighted average of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0146] In the training method, the AI ​​model may comprise a convolutional neural network (CNN).

[0147] In the training method, the AI ​​model may comprise a recurrent neural network (RNN).

[0148] In the training method, the AI ​​model may include an input layer, multiple convolutional layers, a dropout layer or several dropout layers, and an output layer.

[0149] In the training method, the output layer may be a linear layer.

[0150] In the training method, the multiple convolutional layers may comprise one or several strided convolutional layers.

[0151] In the training method, the AI ​​model may be or may comprise a fully convolutional network.

[0152] In the training method, the AI ​​model may include one or more of a pooling layer, an attention module, a self-attention module, an atlas convolution, a normalization layer, a batch normalization layer, a transformer, and a recurrent layer.

[0153] In the training method, the AI ​​model may be equipped with one or several nonlinearities.

[0154] In the training method, the AI ​​model may be equipped with exponential linear units (ELUs).

[0155] In the training method, the AI ​​model may be equipped with rectified linear units (ReLU).

[0156] In the training method, the AI ​​model may be equipped with a scaled exponential linear unit (SELU).

[0157] In the training method, the AI ​​model may be equipped with a Gaussian Error Linear Unit (GELU).

[0158] In the training method, the image data may represent a volumetric representation of the tissue.

[0159] In the training method, the image data may comprise a set of 3D images or 2D images.

[0160] In the training method, the image data can be electron microscopy (EM) images.

[0161] In the training method, the image data may be a 3D EM image.

[0162] In the training method, the image data may be an X-ray tomography image.

[0163] In the training method, the image data may be optical microscopy, for example light microscopy data.

[0164] In the training method, the image data may be positron emission tomography (PET) image data, magnetic resonance (MR) imaging image data, or other medical image data.

[0165] In the training method, the elongated object on which the AI ​​model is trained may be a neurite.

[0166] In the training method, the elongated object on which the AI ​​model is trained may be a blood vessel.

[0167] In the training method, the elongated object on which the AI ​​model is trained may have surfaces that extend around a centerline in a locally cylindrical manner.

[0168] According to another embodiment, there is provided a method of tracing an object in image data, the method comprising determining a series of points along the object to trace the object in the image data.

[0169] Determining the series of points may include an iterative process, inferring an operation prediction using an artificial intelligence (AI) model, the AI ​​model comprising: an input layer operative to receive an AI model input based on pixel or voxel values ​​of image data; and an output layer operative to provide, at each iteration of an iterative process, an AI model output that is or is indicative of an operation prediction; calculating a next point in the series of points using at least the operational prediction, the next point following a previously determined point in the series of points; iteratively repeating the steps of estimating and calculating until a termination criterion is met; may include:

[0170] The image data may be N-dimensional image data, where N is an integer. N may be at least 2, at least 3, or at least 4.

[0171] An object can be traced in ND dimensions, where D is an integer greater than or equal to 0 and less than N.

[0172] One of the dimensions of the image data may correspond to a time axis.

[0173] Determining the next point in the series of points may include integrating the maneuver prediction.

[0174] Integrating the steering prediction may include integrating the curvature in the Bishop frame of reference.

[0175] The steering prediction is related to the second derivative of the object's centerline.

[0176] The AI ​​model inputs input to the input layer in an iteration of the iterative process may represent sub-volumes of the image data.

[0177] The sub-volumes may depend on position and / or orientation and / or manipulation predictions determined in one or several previous iterations of the iterative process.

[0178] The orientation of the sub-volume may depend on a prior manipulation prediction.

[0179] The center of the sub-volume may be shifted by an offset relative to the previously determined point.

[0180] The offset may depend on a manipulation prediction determined in a previous iteration.

[0181] The method may further include multivariate interpolation of pixel or voxel values ​​of the image data to determine the AI ​​model inputs.

[0182] Multivariate interpolation may comprise a weighted average of pixel or voxel values ​​to generate the AI ​​model input as interpolated pixel or voxel values ​​on a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data.

[0183] Multivariate interpolation may include multi-linear projections.

[0184] The method may further include initializing the iterative procedure.

[0185] The initial setting may depend on whether an initial direction is available.

[0186] When an initial direction is not available, the method may include determining an initial direction to be used in an initial iteration of the iterative process.

[0187] Determining the initial direction may include determining prediction uncertainty estimates for various directions.

[0188] Determining the initial direction may include, from among various directions, a direction and maneuver prediction that has the smallest uncertainty estimate in the corresponding prediction.

[0189] The uncertainty estimates may be determined using Monte Carlo dropout.

[0190] The AI ​​model may comprise a convolutional neural network (CNN) or a recurrent neural network (RNN).

[0191] The AI ​​model may include an input layer, multiple convolutional layers, a dropout layer or several dropout layers, and an output layer.

[0192] The AI ​​model may include one or more of a pooling layer, an attention module, a self-attention module, an atlas convolution, a normalization layer, a batch normalization layer, a transformer, and a recurrent layer.

[0193] One, some, or all of the following may be applied: the output layer is a linear layer; the multiple convolutional layers include one or several strided convolutional layers; the AI ​​model is or includes a fully convolutional network; the AI ​​model includes one or several nonlinearities; the AI ​​model includes an exponential linear unit (ELU); the AI ​​model includes a rectified linear unit (ReLU); the AI ​​model includes a scaled exponential linear unit (SELU); and the AI ​​model includes a Gaussian error linear unit (GELU).

[0194] The object may be an object that extends along a time axis of the image data.

[0195] The object may be an object in a video.

[0196] The object may be an object that is traced through a series of image frames.

[0197] The object may be a biological, in particular a physiological, object.

[0198] The object may be a human or animal neurite.

[0199] The object may be a human or animal blood vessel.

[0200] The object may be an asset of an infrastructure, in particular a system of public utilities.

[0201] The object may be a pipe for a fresh water or sewer system.

[0202] The object may be piping in an industrial plant.

[0203] An object may have a surface that extends around the central axis of the object in a locally cylindrical manner.

[0204] The method may further include training the AI ​​model.

[0205] The AI ​​model can be trained using supervised learning.

[0206] Training the AI ​​model may include gradient-based updating of the AI ​​model parameters.

[0207] Training the AI ​​model may include training the AI ​​model to follow a flight policy that causes the iterative process to converge to the object when starting from or reaching a location in the image data that is away from the object.

[0208] The convergence distance of the flight policy may determine how quickly the iterative process converges back to the object.

[0209] The convergence distance can be a dynamic parameter that can be varied during the training of the AI ​​model.

[0210] The convergence distance can be set as a function of the distance from the boundary represented by the image data during training of the AI ​​model.

[0211] The convergence distance may be a monotonically increasing function of the distance from the boundary.

[0212] The image processing method may include receiving and / or acquiring image data and performing an object tracing method.

[0213] The image processing method may include outputting one or more of: a visual representation of the series of points or centerline; a machine-readable representation of the series of points or centerline; a control signal dependent on the series of points located along the centerline; and a machine learning (ML) model having an architecture dependent on the series of points located along the centerline.

[0214] The image data may comprise a temporal series of temporally consecutive image frames.

[0215] Tracing an object may include tracing a physical object along a time axis in a space-time coordinate system.

[0216] The control method includes an object tracing method or an image processing method, and further includes controlling at least one controllable asset based on a result of the object tracing method.

[0217] Also disclosed is a processing system comprising an interface for receiving image data and at least one circuit (eg, a set of processors or other integrated circuits) operative to perform an object tracing method or an image processing method.

[0218] Also disclosed is a system comprising at least one controllable asset (eg, a machine or actuator) and a processing system, where the controllable asset is responsive to control commands or control signals generated by the processing system.

[0219] Various effects and advantages are achieved by methods and systems according to embodiments of the present invention: the methods and systems enable tracing of elongated objects with a low error rate, and facilitate automated analysis of images, e.g., images of neural tissue, for purposes such as connectomics.

[0220] The method and system enable tracing of neurites with accuracy comparable to or superior to conventional automatic or semi-automatic techniques and approaching that of human annotators. The method and system are not only applicable to tracing dendrites, but can also trace, for example, axons and spine necks, which are typically more difficult to trace than dendrites.

[0221] According to a further aspect of the present invention, an image processing method and an image processing system operable to perform local realignment are also disclosed, which may include the following steps performed to perform local realignment of slices of image data along a slice direction: determining slice-to-slice shift vectors for a plurality of slices along the slice direction; identifying valid slice-to-slice shift vectors between the slices; and generating a locally realigned sub-volume using the valid slice-to-slice shift vectors.

[0222] The locally realigned subvolumes can be fed to an AI model input, e.g., the input layer of a CNN or RNN, or to a filter or other preprocessing that preprocesses the realigned subvolumes before they are processed by the CNN or RNN.

[0223] Determining the slice-to-slice shift vectors may include determining slice autocorrelations and cross-correlations.

[0224] Determining the slice-to-slice shift vector and its validity may include determining the mean and standard deviation of the correlation weighted shift vector based on a first threshold criterion.

[0225] Identifying a valid slice-to-slice shift vector may include verifying that the determined autocorrelation meets a first criterion (such as having a peak at a desired shift, e.g., zero shift).

[0226] Identifying a valid slice-to-slice shift vector may include verifying that the determined autocorrelation peak has a standard deviation that meets a second criterion (e.g., having a standard deviation that meets a second threshold criterion that is less than or equal to a certain number of pixels).

[0227] Identifying a valid slice-to-slice shift vector may include verifying that the combination of two consecutive autocorrelation and / or cross-correlation peaks meets a third criterion (e.g., optionally, meets a third threshold criterion that may depend on surrounding autocorrelations and / or cross-correlations).

[0228] Identifying a valid slice-to-slice shift vector may include verifying that the determined cross-correlation satisfies a first cross-correlation criterion that depends on the autocorrelation for the two slices involved (such as the referenced autocorrelation criterion being valid for both slices).

[0229] Identifying a valid slice-to-slice shift vector may include verifying that the determined cross-correlation normalized by the expected cross-correlation value for the strength of the perfect correlation satisfies a second cross-correlation criterion (such as the peak of the cross-correlation normalized by the expected cross-correlation value for the strength of the perfect correlation meeting a further threshold criterion, e.g., greater than or equal to a fourth threshold).

[0230] Identifying valid slice-to-slice shift vectors may include verifying that the standard deviation of the shift vectors meets a third cross-correlation criterion (such as the standard deviation being less than or equal to a fifth threshold).

[0231] Using a valid shift vector may include identifying one or more slices to be realigned even if the slice-to-slice shift vector is not considered valid, and in this case calculating the shift vector from valid shift vectors associated with the neighbors of the one or more slices.

[0232] Using the valid shift vectors may include generating locally realigned trilinearly interpolated sub-volumes.

[0233] Generating the locally realigned trilinearly interpolated sub-volumes may include performing a floor or ceiling operation.

[0234] Generating the locally realigned trilinearly interpolated sub-volumes may include performing the trilinear interpolation as a series of linear interpolations along the major axes.

[0235] The series of linear interpolations may include four linear interpolations along a first dimension, two linear interpolations along a second dimension, and one linear interpolation along a third dimension.

[0236] The locally realigned sub-volumes can be processed to trace elongated objects such as 3D blood vessels or neurites.

[0237] The local realignment technique may be used in connection with (e.g., as part of) a method for tracing an elongated object, but is not limited thereto, and provides for referencing of slices to a common frame of reference in a particularly efficient manner. [Brief explanation of the drawings]

[0238] The embodiments will be described in detail with reference to the drawings, in which the same or similar elements are designated by the same or similar reference numerals.

[0239] [Figure 1] FIG. 1 is a block diagram representation of a system according to an embodiment. [Figure 2] FIG. 2 shows an elongated object in 3D image data. [Figure 3] FIG. 3 shows an elongated object in 3D image data. [Figure 4] FIG. 4 is a series of points illustrating the operation of systems and methods according to embodiments. [Figure 5] FIG. 5 is a flowchart of a method according to an embodiment. [Figure 6] FIG. 6 is a diagram of a sub-volume of an imaged tissue volume illustrating the operation of an embodiment of a system and method. [Figure 7] FIG. 7 is a diagram of a voxel grid. [Figure 8] FIG. 8 is a diagram illustrating the determination of artificial intelligence (AI) model inputs. [Figure 9] FIG. 9 illustrates an AI model of systems and methods according to embodiments. [Figure 10] FIG. 10 illustrates an AI model of systems and methods according to embodiments. [Figure 11] FIG. 11 is a flowchart of a method according to an embodiment. [Figure 12] FIG. 12 is a flowchart of a method according to an embodiment. [Figure 13] FIG. 13 illustrates an AI model of an embodiment of a system and method, illustrating its operation. [Figure 14] FIG. 14 illustrates a further AI model of systems and methods according to embodiments, illustrating their operation. [Figure 15] FIG. 15 is a flowchart of a method according to an embodiment. [Figure 16] FIG. 16 illustrates flight policies that may be adopted during training of an AI model. [Figure 17] FIG. 17 is a flowchart of a method according to an embodiment. [Figure 18] FIG. 18 is a flowchart of a method according to an embodiment. [Figure 19] FIG. 19 illustrates the operation of the system and method according to an embodiment. [Figure 20] FIG. 20 illustrates a local coordinate system that may be employed by methods and systems according to embodiments. [Figure 21] FIG. 21 shows the error rate of an embodiment of a method and system as a function of training iterations. [Figure 22] FIG. 22 is a flowchart of a method according to an embodiment. [Figure 23] FIG. 23 illustrates an application for which an embodiment of the present invention is suitable due to its low reconstruction error rate and large distance between reconstruction errors. [Figure 24] FIG. 24 illustrates the improvement achieved by the embodiment compared to the prior art. [Figure 25] FIG. 25 illustrates the improvement achieved by the embodiment compared to the prior art. [Figure 26] FIG. 26 is a further diagram illustrating the improvement achieved by the embodiment compared to the prior art. [Figure 27] FIG. 27 is a flowchart of a method according to an embodiment. [Figure 28] FIG. 28 is a block diagram representation of a system according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0240] The embodiments will now be described in detail with reference to the drawings, in which the same or similar components are designated by the same or similar reference numerals.

[0241] Although embodiments are described in detail with reference to particular physiological objects such as neurites, axons, and spine necks, the techniques disclosed herein are not limited thereto. Methods and systems operating as disclosed herein can be used to process image data to trace various physiological objects, such as the walls of blood vessels or other elongated lumens. Methods and systems operating as disclosed herein can also be used to generate annotated image data from raw image data.

[0242] Although the embodiments are described in detail with reference to particular imaging techniques or particular image data, e.g., three-dimensional electron microscopy (3D EM) imaging and 3D EM images, the methods and systems may also be used to process images acquired using other imaging techniques, e.g., optical microscopy, magnetic resonance imaging, X-ray holographic nanotomography, etc.

[0243] Tracing long, thin objects The systems and methods disclosed herein operate to trace elongated objects (such as, but not limited to, neurites) in image data. The image data may be 3D image data or may be image data in a space-time coordinate system having a time axis as one of its axes. The image data may be biological or medical image data. The image data may include a series of frames that may be acquired sequentially in time. The techniques disclosed herein are applicable to other image data as well.

[0244] The systems and methods may determine a series of points that are predicted to lie on the centerline of an elongated object, thereby tracing the object. The systems and methods may operate to process voxel or pixel values ​​of image data without requiring prior object recognition in the image data.

[0245] The systems and methods disclosed herein may trace an object by (a) selecting a sub-volume of image data, (b) determining information about the direction in which the elongated object extends in the sub-volume, and (c) processing the direction to determine additional points located along the central axis. These steps may be repeated iteratively.

[0246] The sub-volumes processed in an iteration may have an orientation that depends on the object's elongation direction determined in the previous iteration. Thus, the direction drives the analysis process similar to the techniques of autonomous driving or flight. Thus, the direction along which the central axis of an elongated object extends is also referred to herein as the "flight direction." This direction determines the path along which the techniques disclosed herein operate.

[0247] The change in flight direction is determined at least by a maneuver prediction. The maneuver prediction may define or otherwise relate to a derivative of the flight direction (e.g., a first, second, or higher order derivative). As used herein, the term "maneuver prediction" refers to a quantity that depends on how far the elongated object extends along the elongated object from a previously determined point and / or that determines how the iterative techniques disclosed herein proceed (particularly with respect to the orientation and / or location of the sub-volumes).

[0248] The orientation of the sub-volume to be analyzed next may be set depending on the flight direction and / or maneuver prediction.

[0249] The determination of the manipulation prediction may use an AI model. The AI ​​model may have an input layer operative to receive voxel values ​​determined from the image data. The AI ​​model input may be obtained from the image data, or more generally, by interpolating neighboring voxels of the image data to correspond to any position and orientation of the subvolume. The AI ​​model may have an output layer operative to output the manipulation prediction. For improved accuracy and stability, the AI ​​model may be trained not only for positions located along the centerline of the elongated object in the training image data, but also for positions located away from the centerline of the elongated object in the training image data. Convergence back to the centerline may thereby be ensured during inference.

[0250] The determination of the new point along the centerline may be performed using an integral of the manipulation prediction, for example, when the manipulation prediction is with respect to higher order derivatives of the centerline direction of the elongated object, the integral may include several subsequent integrals.

[0251] The system and method are independent of individual voxel similarity predictions and do not require explicit volume segmentation of image data. Instead, they operate to directly solve the problem of following the centerlines of elongated objects, such as neuronal wiring. The system and method can facilitate large-scale connectomics reconstruction. The system and method are also computationally efficient, facilitating resource consumption for obtaining connectomes from animal and human brains, screening for circuit abnormalities in health and disease, and biopsy analysis of pathological tissue in clinical settings.

[0252] The system and method provide an approach for automated axonal reconstruction of 3D EM data by training an AI model to mimic axonal flight mode traces. The system and method may employ a convolutional neural network (CNN). Details of CNN are described, for example, in Y. LeCun et al., 1989, "Backpropagation Applied to Handwritten Zip Code Recognition." Neural Computation 1(4):541-51.

[0253] The system and method offer various differences and advantages compared to the CNN-based techniques of M. Januszewski et al., 2018, "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4.

[0254] The system and method determines the (approximate) centerline of an elongated object, such as a neurite, and through iterations, performs a skeletal reconstruction of the centerline. The subvolume selection (e.g., subvolume orientation) and problem description ensure that the analysis is aligned to the elongated axis. Only the deviation of the centerline from a straight line (along the subvolume orientation) needs to be predicted to keep it aligned.

[0255] That is, the systems and methods disclosed herein analyze, in each of several iterations of the iterative process, a sub-volume that is oriented in a manner that takes into account the local orientation of the elongated object.

[0256] If the centerline is a straight line, the steering estimate is 0. If the centerline deviates from a straight line, the steering estimate may capture this deviation, for example, by a parabolic approximation to the neurite at the current location.

[0257] In comparison, the subvolumes of the Flood-Filling Networks (FFNs) of M. Januszewski et al., 2018, "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks," Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4, are oriented along the major axes of the image dataset and therefore do not capture the orientation of the principal axes / centerlines of the neurites. This conventional technique does not provide a mechanism by which the principal axes or centerlines are captured or used for reconstruction. Instead, FFNs predict the similarity of each voxel, which adds to computational complexity and cost. FFNs are therefore more computationally expensive than the techniques disclosed herein. While FFNs are limited to integer coordinates, the techniques disclosed herein provide multivariate interpolation of image data and a continuous problem description in terms of a spatially varying coordinate system that is more easily applicable to continuous coordinates for input and / or output.

[0258] For the techniques disclosed herein, AI models are trained on the task of predicting local continuity of centerlines from neurite-center-aligned 3D-EM subvolumes, for example. This allows for straightforward optimization of error rates, metrics over which previous automated methods have made little progress, and metrics beyond voxel-based metrics that are appropriate for extracting connectomes from 3D-EM volumes that are much larger than the diameter of a typical neurite.

[0259] During inference, the system and method operates to incorporate manipulation predictions into new positions and orientations, so that repeated application results in a trace of visited positions similar to the skeleton generated by human annotation.

[0260] The directionality and continuity of elongated objects (such as neurites) can be directly incorporated into the problem statement.

[0261] Tests show that the techniques disclosed herein result in low error rates even with lower resolution data, at significantly reduced computational cost compared to previous methods. In addition, the methods and systems may eliminate any need for segmentation and may operate on 3D image data without prior segmentation.

[0262] Figure 1 is a block diagram of a system 10. The system 10 operates to trace elongated objects (such as neurites) in 3D image data (such as 3D-EM image data).

[0263] The system comprises an imaging system 20. The imaging system 20 may be a 3D EM imaging system. The imaging system 20 may comprise a source 21. The source 21 may comprise, for example, an electron source operable to output a beam of electrons. The imaging system 20 may comprise a detector 22. The detector 22 may be operable to detect electrons scattered, reflected, or refracted off physiological objects within a sample (such as a biopsy sample). The imaging system 20 may comprise an image alignment component 23 for image alignment prior to reconstruction.

[0264] Imaging system 20 may operate to perform local realignment. The local realignment may be performed by image alignment component 23, or imaging system 20 may operate to perform a second local image realignment component in addition to image alignment component 23. Imaging system 20 (particularly its local image realignment component) may operate to perform local realignment and dealignment for mapping to a global image coordinate system. By way of illustration, imaging system 20 may operate to perform the method of PHLi et al., "Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment." Preprint, https: / / doi.org / 10.1101 / 605634, particularly the technique described in connection with Figure 3 of this preprint.

[0265] Local realignment can be performed using techniques such as those specifically described herein below in more detail.

[0266] The imaging system 20 may include an image reconstruction circuit 24. The image reconstruction circuit 24 may operate to calculate voxels representing 3D image data of the sample. The 3D image data 40 may be provided to the image processing system 30. Instead of or in addition to the 3D image data, other forms of volumetric tissue representations, such as a series of two-dimensional (2D) images or projections, may be used. An example of such a representation is X-ray holographic nanotomography, which may be performed using the techniques of (e.g., A.T. Kuan et al., "Dense neuronal reconstruction through X-ray holographic nano-tomography," Nature Neuroscience volume 23, pages 1637-1643 (2020)).

[0267] The image processing system 30 comprises one or several processing circuits 31. The processing circuitry 31 may include one or several circuits, including, but not limited to, integrated circuits, such as field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), controllers, processors, superconducting circuits, and quantum bit (qubit) processing circuits.

[0268] The processing circuitry 31 may operate to perform tracing of an elongated object (e.g., but not limited to, a neurite) through the volumetric representation 40 of the tissue. Starting from the location of the elongated object's centerline, the AI ​​model 32 may be provided with AI model inputs that each depend on (and may represent) a subvolume of the image data 40. The AI ​​model 32 is trained to output a direction of the elongated object from an already known point in response to this image data. This direction determines the direction in which the tracing will continue and is therefore also referred to as a steering prediction. Based on the steering prediction, the next point along the centerline (adjacent to the already known point) is calculated. This may be done by integration, which may be performed by steering prediction processing 33. The determination of a steering prediction by the AI ​​model and the determination of a new point along the centerline by the steering prediction processing may be repeated, thereby generating a series of points located on the centerline.

[0269] Segmentation of image data 40 is not required for the systems or methods to operate in the manner disclosed herein.

[0270] Determining each point located along the 3D centerline includes determining the location of the point within the 3D volume represented by image data 40. Determining the points includes determining information that allows all three coordinates for the point to be specified in the volume represented by image data 40. Determining the points may include determining three Cartesian coordinates for the point, and it will be understood that many equivalent descriptions may be used to specify the location of a point in 3D.

[0271] Results of the image processing may be output by the image processing system 30. The image processing system 30 may comprise a graphical user interface (GUI) that outputs a graphical representation of a series of points and / or a centerline of the elongated object. The GUI may be operative to output a visualization of the centerline of the elongated object. Alternatively or additionally, the image processing system 30 may comprise an output interface 34. The output interface 34 may be operative to output machine-readable data dependent on the results of the tracing of the elongated object. The machine-readable data may comprise control signals dependent on the results of the tracing of the elongated object. The machine-readable data may comprise control logic dependent on the results of the tracing of the elongated object, for example, when using the results of the tracing of neurites for designing or training machine learning (ML) models derived from neuroscience.

[0272] Image processing system 30 may also be operative to control GUI 30 to allow a user to input a starting point for tracing of the object. Image processing system 30 may also be operative to control GUI 30 to allow a user to optionally input a starting direction for tracing of the object. As described in more detail herein, recognition of the initial direction is optional. The initial direction may be determined automatically using, for example, uncertainty estimates obtained from Monte Carlo dropout.

[0273] Although tracing of elongate objects is further illustrated below with reference to tracing one elongate object, the systems and methods operate to trace multiple elongate objects. The techniques are also applicable to multiple elongate objects and / or multiple elongate portions of an object and / or multiple portions of an elongate object. This may be done in a parallel or serial manner. Illustratively, multiple instances of the image processing techniques disclosed herein may be instantiated to simultaneously trace multiple elongate objects in parallel. Alternatively, or in addition, the systems and methods may loop through the techniques disclosed herein several times for different elongate objects. This may be done until a 3D trace is determined for each desired elongate object. Illustratively, 3D traces of multiple (e.g., more than 100 or more than 1000) neurites, axons, and / or neural spine necks may be traced. The methods disclosed herein are suitable for, but not limited to, use in connectomics.

[0274] 2 shows a volume 40 represented by image data and an elongated object 41 extending therethrough. The systems and methods disclosed herein operate to trace the centerline of the elongated object 41. The systems and methods may follow the centerline of the elongated object 41 using manipulation prediction-dependent subvolume selection, AI model processing of the image data in the subvolume to update the manipulation prediction, and integration of the manipulation prediction. The systems and methods may employ these techniques for, but are not limited to, tracing neurites.

[0275] 3 shows a centerline 50 of an elongated object. The techniques disclosed herein are applicable not only to neural processes but also more broadly to other objects, such as blood vessels, the lumen of the respiratory tract, etc. The techniques are also applicable to tracing in a space-time coordinate system (where the physical object tracing an elongated path along a time axis is not necessarily physically elongated). The techniques are also applicable to multiple elongated objects and / or multiple elongated portions of an object and / or multiple portions of an elongated object.

[0276] The methods and systems disclosed herein can be used to trace objects that have a locally cylindrical shape (e.g., the centerline 50 of an elongated object). The term "locally cylindrical" is used to refer to a shape in which the outer wall of the elongated object can be locally approximated by cylindrical segments 42, 43. The orientation of the various cylindrical segments 42, 43 can be tilted relative to one another in 3D space. The diameter of the cylindrical segments 42, 43 can also vary along the centerline 50 (as is also typically the case for neurites).

[0277] 4 illustrates the operation of the method and system disclosed herein. Processing begins with image data and an initial point 61 located on the elongated object to be traced. The initial point 61 may be set by a user to identify which object is to be traced. Alternatively, automatic object recognition may be employed to identify the starting point 61 on the object. An initial flight direction 70 specifying the orientation of the elongated object at the starting point 61 may be received by user input or may be automatically estimated by the system using a prediction uncertainty specifying the uncertainty of the prediction estimate for various directions (such as, but not limited to, obtainable from Monte Carlo dropout).

[0278] The system and method include: - identifying a first sub-volume located around the starting point 61 (but typically with some first central offset from the starting point 61); - determining a first operation prediction 71 by processing image information from the first sub-volume; - processing at least a first operation prediction 71 (e.g., using integration), a position offset 72 giving the first point 62 after the starting point 61, Operate to determine new subvolume orientations (not shown) for subsequent iterations.

[0279] These steps may utilize a starting point 61 and a starting direction 70 as input.

[0280] The process may then be repeated in an iterative manner. Illustratively, the system and method then: - identifying a second sub-volume located around the first point 62 (but typically having some second central offset from the first point 62); - determining a second operation prediction by processing image information from the second subvolume; - It may operate to determine a second point 63 after the first point 62 by processing (e.g., using integration) at least the second operation prediction to determine a position offset 73 and a new sub-volume orientation (not shown).

[0281] The process may be repeated in an iterative manner to determine, by iterative guessing, a series 60 of points located on the centerline 50. In this manner, further operational predictions may be determined that result in further position offsets 74, 75, 79 and further points 64, 65 in the series 60.

[0282] In each iteration of the iterative process, a subvolume 76 of the volume represented by the image data 40 is determined. The subvolume 76 includes a current point 65 determined in a previous iteration. The subvolume 76 has an orientation that depends on a previous maneuver prediction determined in a previous iteration. The image data for the subvolume 76 is processed by the AI ​​model to determine a maneuver prediction 78 for the current iteration of the iterative process. The next point and orientation following the current point 65 is determined using the maneuver prediction 78 for the current iteration, for example, by integration (e.g., by adding a vector proportional to the instantaneous flight direction 77 to the current point 65 to determine the next point, by adding a vector proportional to the maneuver prediction to the instantaneous flight direction / subvolume orientation 77 to determine the next flight direction / subvolume orientation, or alternatively, by using more complex extrapolation techniques for extrapolating from a series of previously determined maneuver predictions). As previously described, analysis of the image data contained in the subvolume 76 provides the maneuver prediction 78, which is integrated to determine a next position offset 79 to continue tracing the elongated object.

[0283] The systems and methods disclosed herein may thereby process subvolumes of volume 40 to infer the local direction of centerline 50 corresponding to the maneuver prediction. Each subvolume may be centered on a current point determined in a previous iteration in two directions transverse to the instantaneous flight direction / subvolume orientation. Each subvolume may have a center offset from a previous point determined in a previous iteration, the offset being in a direction parallel to the instantaneous flight direction. Illustratively, subvolume 76 may be positioned such that point 65 determined in a previous iteration is closer to the end face of subvolume 76 located in a direction opposite instantaneous flight direction 77 from point 65 than to the other end face of subvolume 76 located in a direction along instantaneous flight direction 77 from point 65.

[0284] Therefore, the methods and systems disclosed herein employ an approach where the field of view (i.e., subvolume) being analyzed at each step is oriented along an estimated tangent to the centerline, and the tracing process is automatically steered along the centerline. Continuity of elongated objects is automatically established in the process. The process is similar to tracing a desired path in autonomous driving (but in 3D).

[0285] 5 is a flow chart of a method 80. The method 80 may be performed automatically by an image processing system. The method 80 determines a series of points 60 along the centerline 50.

[0286] In step 81, AI model inputs are determined from the image data and a current point. In the first iteration, the current point is a starting point 61, which may be set by user input. In each subsequent iteration, the current point may be a point 62-65 on the centerline determined in the previous iteration.

[0287] Determining the AI ​​model input in step 81 may determine the AI ​​model input as image data from a subvolume 76 of volume 40. Subvolume 76 may include a current point, which may be the result of a previous iteration. Subvolume 76 may have an orientation that depends on a previous operation prediction and the orientation of the subvolume. Subvolume 76 may be tilted (usually tilted) relative to the edge of volume 40. In the first iteration, the orientation of the subvolume may be determined based on user input, based on calculated prediction uncertainty at various orientations, as a random orientation, or in other ways. In each subsequent iteration, the current position and orientation of the subvolume are those determined by the AI ​​model in the previous iteration of method 80.

[0288] Determining the AI ​​model inputs in step 81 may involve averaging (eg, weighted averaging) or other interpolation techniques performed on voxel or pixel values ​​of image data 40, as described in more detail herein.

[0289] In step 82, a sub-volume of the image is processed, which may include processing the AI ​​model input with an AI model. The output layer of the AI ​​model may output an updated operation prediction.

[0290] In step 83, at least the manipulation prediction obtained in step 82 is processed to determine the next orientation for the next point and subvolume along the centerline 50. This may involve vector addition or integration with respect to the previous (optionally, several previously determined) manipulation predictions, positions, and orientations.

[0291] At step 84, it is determined whether a termination criterion is met. The termination criterion may be any one or any combination of the following, but is not limited to: intersection of the analyzed subvolume and / or next position with a boundary of the image data 40 or a user-defined boundary; a threshold criterion for the number of iterations or accumulated path length; no discernible continuation of elongated objects in the subvolume; intersection of positions or accumulation of path lengths within objects defined by other means (e.g., intersection of a neurite being traced with a previously traced neurite). If the termination criterion is not met, the method may return to step 81. Otherwise, the results of the tracing of the elongated object may be output via a GUI or data interface or used in other ways at step 85.

[0292] The technique of Figure 5 may be used as part of a more complex process or method, or may be incorporated into some higher-level logic. Illustratively, a method is disclosed that automatically detects a missed sequence of an auto-reconfiguration, and in response to detecting the missed sequence at that point, begins the technique of Figure 5 and executes it until it finds another automatically reconfigured component. From that point, the technique disclosed herein (as described with reference to Figure 5) can then be executed backward to reconfirm that it ends at the same starting location.

[0293] Thus, the technique of FIG. 5 can be used as a building block for more complex reconstruction methods.

[0294] FIG. 6 illustrates location-dependent selection of subvolumes for analysis in the systems and methods disclosed herein. The centerline 50 and points 62 along the instantaneous flight direction 71 determined in each iteration of the iterative estimation process are used to determine which portion of the volume 40 will be next processed by the AI ​​model to continue tracing the elongated object. The subvolume 91 may have, but is not limited to, a rectangular parallelepiped shape. Four of the edges of the subvolume 91 may be parallel to the instantaneous flight direction 71. The four edges parallel to the maneuver prediction 71 may be chosen constant or, depending on the previous AI model prediction, may be shorter, longer, or the same length as the other edges of the subvolume 91. The subvolume 91 may have a size that depends on the object being traced.

[0295] Point 62 may be contained within sub-volume 91. Sub-volume 91 may be located such that its center 92 is offset from point 62 by an offset along instantaneous flight direction 73. That is, sub-volume 91 may be located such that it extends a greater distance along instantaneous flight direction 73 than opposite the instantaneous flight direction when viewed from point 62.

[0296] The orientation of the sub-volume varies depending on the previous flight direction and maneuver prediction determined in the previous iteration.

[0297] By way of illustration, the centerline 50 and points 65 along the instantaneous flight direction 77 determined in each iteration of the iterative estimation process are used to determine which portion of the volume 40 will be next processed by the AI ​​model to continue tracing the elongated object. The subvolume 93 may have, but is not limited to, a rectangular parallelepiped shape. Four of the edges of the subvolume 93 may be parallel to the instantaneous flight direction 77. The four edges parallel to the instantaneous flight direction 77 may be shorter, longer, or the same length as the other edges of the subvolume 93 and may be chosen as a constant or may depend on the AI ​​model predictions from the previous iteration.

[0298] Point 65 may be contained within sub-volume 93. Sub-volume 93 may be located such that its center 94 is offset from point 65 by an offset along instantaneous flight direction 77. That is, the sub-volume may be located such that it extends a greater distance along instantaneous flight direction 77 than the opposite of the maneuver projection when viewed from point 65.

[0299] The orientation of the edges of the subvolumes 91, 93 that are not parallel to the instantaneous flight direction 73, 77 is not germane. Random rotations 95, 96 about the instantaneous flight direction 73, 77 can be introduced during training and / or inference when selecting the subvolumes 91, 93. This further improves the robustness of the techniques disclosed herein.

[0300] The random rotations 95, 96 can be replaced or combined with a method that creates an AI model of a variant that is (approximately) equivalent to this rotation. This has the effect that providing a rotated input results in the operation prediction being rotated accordingly. This can further increase the robustness of the techniques disclosed herein.

[0301] The edges of the subvolumes 91, 93 parallel to the steering direction can be chosen constant or, depending on previous AI model predictions, can be shorter, longer, or the same length as the edges of the subvolumes 91, 93 not parallel to the steering direction, which further improves the robustness of the techniques disclosed herein.

[0302] The dependence of the sub-volumes 91, 93 on the preceding maneuver prediction, the preceding flight direction, and the preceding point has the effect that the voxels of the sub-volumes 91, 93 will not typically coincide with the voxels of the original image data 40. As shown in Figures 7 and 8, interpolation can be used to calculate the voxel values ​​within the sub-volumes 91, 93 for any orientation and position of the sub-volumes 91, 93.

[0303] 7 and 8 show voxels (e.g., voxel centers) of image data 40 as solid circles. Voxels of sub-volumes 91, 93 are shown by hollow circles and may generally be anisotropic in terms of physical length scales. The voxels of image data 40 define a grid 100. The voxels of the sub-volumes define a second grid 105, which may be anisotropic. A second coordinate axis 106 of second grid 105 may be rotated in 3D and / or shifted in 3D relative to the coordinate axis 105 of grid 100 on which the voxels of the original image data 40 are defined.

[0304] Interpolation techniques can be used to determine voxel values ​​at locations in the second grid 105 (e.g., as grayscale or color values ​​with various color channels), which are then one of the AI ​​model inputs.

[0305] The projection may be used to determine voxel values ​​in the second grid 105. By way of illustration, as shown in Figure 8, a projection (such as a trilinear projection) may be used to project the image data onto the face of the grid 105.

[0306] Other techniques may be used. Illustratively, a point in the second grid 105 (i.e., a voxel center of the subvolumes 91, 93) may be located within a cube defined by several (e.g., eight) voxel centers 103 of the original image data 40. A weighted average may be performed to calculate a voxel value for the point 106 in the subvolumes 91, 93 from the voxel values ​​of the original image data. Illustratively, the voxel value for the point 106 may be calculated as a weighted average of the voxel values ​​of eight voxel centers 103 located around the point in the second grid 105. Other techniques, such as trilinear projection (as described above), may be used to interpolate the image data to the grid 105 defined for the subvolumes, thereby determining the AI ​​model inputs.

[0307] As already mentioned and described in more detail herein below, local realignment may be applied. In Fig. 7, local realignment may be applied to (possibly non-integer) x and / or y offsets (which may be obtained from cross-correlation) of the 2D image planes (represented by the black dot planes in Fig. 7). Thus, trilinear interpolation (which, when subjected to local realignment of voxels at black dot locations, produces voxel values ​​at the white dot locations in Fig. 7) may be calculated using two bilinear interpolations for each white dot location determined separately in the z coordinates of the floor and ceiling, with possibly different x / y offsets applied to the black dot locations, and then combining those two intermediate values ​​from the bilinear interpolations with another linear interpolation along the z axis.

[0308] Regarding AI model configuration, an AI model including a convolutional layer has been determined to provide superior performance. Illustratively, the AI ​​model may be or may comprise a CNN.

[0309] 9 and 10 illustrate AI models that may be used in the systems and methods disclosed herein.

[0310] The AI ​​model 110 comprises an input layer 111 that operates to receive AI model inputs based on pixel or voxel values ​​of image data. The AI ​​model inputs may comprise or be image data within the sub-volumes 91, 93 as determined from voxel values ​​of the image data 40. The voxel values ​​may be converted from integer to floating-point representation and normalized before being processed by the AI ​​model. Normalization may be applied to restrict the AI ​​model input values ​​to a particular range and / or to modify the statistical properties of the AI ​​model input values, for example, to achieve a mean input value of 0 and a standard deviation of 1 in the input values.

[0311] The AI ​​model 110 comprises an output layer 112 that operates to provide an AI model output that is or is indicative of the operation predictions 71-75.

[0312] The AI ​​model 110 includes a hidden layer 113. The hidden layer 113 may include several convolutional layers 114, 115. The hidden layer 113 may include strided convolutional layers and / or fully connected layers. The strides may be different for different convolutional layers.

[0313] The hidden layer 113 may comprise several convolutional layers 114, dropout and / or reconstruction layers 116, and one or several fully connected layers 115.

[0314] The hidden layer 113 may comprise a linear layer as the final layer.

[0315] A variety of activation functions may be used. By way of illustration, both the exponential linear unit (ELU) and the rectified linear unit (ReLU) have been successfully used in the systems and methods disclosed herein. Additionally or alternatively, a wide variety of other activation functions may be used, including, but not limited to, any one or any combination of the following activation functions: scaled exponential linear unit (SELU) and Gaussian error linear unit (GELU). Additionally or alternatively, a wide variety of other artificial neural network layers may be used, including, but not limited to, any one or any combination of fully connected layers, pooling layers (e.g., max pooling), (self-)attention modules, atlas convolutions, (batch) normalization layers, transformers, recurrent layers such as long short-term memories (LSTMs), and gated recurrent units (GRUs).

[0316] For even greater efficiency and robustness, the AI ​​model can be trained to provide information about operational predictions in spatially varying reference frames along elongated curves. Illustratively, the AI ​​model can be trained (during a training phase) and operated during inference to provide information about operational predictions in (local) Bishop frames, e.g., by outputting two Bishop curvatures (see below for further details).

[0317] 11 is a flowchart of the method 120. The method 120 may be performed automatically by the image processing system 30. The method 120 may be performed to determine a series of points 60 along the centerline 50 by iterative estimation.

[0318] In step 121, a subvolume is selected for processing. The subvolume may have a position that depends on a previously determined point along the centerline (which may be specified by user input in the first iteration or as a result of a previous iteration in a later iteration). The subvolume may have an orientation that depends on the orientation of the previous subvolume and a previously determined manipulation prediction (which may be specified by user input or automatically determined in the first iteration and obtained from a previous iteration in a later iteration).

[0319] The AI ​​model operates on the image data within the sub-volumes at step 122. Interpolation of the image data voxels can be performed to determine voxel values ​​in the voxel grid of each sub-volume for any orientation and / or location of the sub-volume.

[0320] In step 123, the AI ​​model may provide an updated manipulation prediction as an AI model output. The updated manipulation prediction may be specified relative to a local frame of reference, for example, as a curvature value relative to a frame of reference given relative to the subvolume (such as the Bishop coordinate system). Further outputs, such as information regarding distance from obstacles, may be provided by the AI ​​model.

[0321] In step 124, a new point along the centerline may be determined based on the steering prediction from step 123. Illustratively, an integration may be performed to determine the new point along the centerline.

[0322] FIG. 12 is a flowchart of a method 125. The method 125 may be performed automatically by the image processing system 30. The method 125 may be performed to determine a series of points 60 along the centerline 50 by iterative estimation. The method 125 may include steps 121-124 described with reference to FIG. 11. The method 125 may further include a further step 126 of determining a rotation about the instantaneous flight direction, which is used to specify sub-volumes in subsequent iterations. Exemplary rotations 95, 96 are shown in FIG. 6.

[0323] 13 illustrates an AI model architecture and data processing that may be employed in the methods and systems disclosed herein. At each iteration, a subvolume 131 centered at the current position (in a direction perpendicular to the current flight direction) is analyzed.

[0324] The AI ​​model 130 may include an input layer operative to receive K×N×M voxel values ​​for image data in the subvolume 131. The number of voxels K and N in a direction transverse to the current manipulation prediction may exceed the number of voxels M parallel to the current manipulation prediction. Illustratively, K=N=96 and M=16 may be used. Furthermore, the physical size of a single voxel for the subvolume may be chosen anisotropically and independently from the physical size of a single voxel in the image data 40.

[0325] The values ​​K and N may be set depending on the object being traced. By way of illustration, it may be useful to set K and N to different values ​​for objects that are known or expected to have an elliptical cross-section in a plane transverse to the central axis.

[0326] In the implementation of Figure 13, the AI ​​model includes seven stride convolutional layers, conv1 through conv7. The layers need not all be the same. For example, three of the convolutional layers, conv1 through conv3, may have 5x5x3 convolutions with stride 2x2x1, and four of the convolutional layers, conv4 through conv7, may have 3x3x2 convolutions with stride 1x1x1.

[0327] In the implementation of FIG. 13, the AI ​​model includes a dropout layer that may have a dropout rate of 0.5.

[0328] In the implementation mode of FIG. 13, the AI ​​model includes three fully connected layers fc1 to fc3 and a linear layer.

[0329] The AI ​​model output 132 may be an operation prediction, optionally determined in a local frame of reference (eg, as Bishop curvature).

[0330] The process includes integration 133 to determine the next point along the centerline based on the AI ​​model output 132. The sub-volumes analyzed in subsequent iterations can be set without further random rotation (the normal guess path in FIG. 13) or with further random rotation 134.

[0331] FIG. 14 shows a further implementation of an AI model 138 that can be used in the methods and systems disclosed herein. Parameters 139 of convolutional layers conv1 to conv7 can be modified compared to the AI ​​model of FIG. 13. Alternatively or additionally, parameters of fully connected layers fc1 to fc3 can be modified compared to the AI ​​model of FIG. 13. By way of illustration, parameter values ​​before or after the vertical bar shown in FIG. 14 can be utilized. In one implementation, parameter values ​​139 and parameter values ​​139′ shown after the vertical bar in FIG. 14 can be used.

[0332] The system and method can be used for, but are not limited to, tracing neurites. The system and method enable neurite reconstruction to be performed from a 3D-EM volume as a centerline reconstruction task, where the neurite is represented by a series of visit points. The CNN is trained to predict local neurite continuity from neurite-center-aligned 3D-EM subvolumes, similar to the human annotation fly-by mode described, for example, in Boergens et al., 2017, "WebKnossos: Efficient Online 3D Data Annotation for Connectomics." Nature Methods 14(7):691-94.

[0333] Successive integration of the predicted neurites yields new positions and orientations that are used to generate subsequent CNN inputs. Iterative application of this procedure turns the starting locations and orientations into a neurite skeleton reconstruction using only 3D-EM data, without intermediate steps such as volume segmentation.

[0334] For the AI ​​model of Figure 13, the input to the CNN consists of a neurite-center-aligned 3D-EM subvolume 131 of 96 x 96 x 16 voxels (Vx). The subvolume may cover a field of view of approximately 1 x 1 x 0.7 μm for an axon, for example (see Figure 13).

[0335] The third (Z) dimension corresponds to the current flight direction of the process, while the current position may be centered in the fourth Z plane.

[0336] Thus, the field of view is asymmetric along the Z direction, with more contextual information available in the forward direction than in the reverse flight direction, allowing for better navigation towards the "exit" within the axon's varicosity.

[0337] The EM data can be projected onto the axon alignment plane by trilinear interpolation.

[0338] Note that by using only the 3D-EM subvolume as input, it is possible to extract relevant information about neurite continuity from a single input. Alternatively, one can also feed 2D EM images to a recurrent neural network, which can internally track the series of images and thereby estimate neurite continuity as well.

[0339] For the CNN architecture in Figure 13, seven 3D stride convolutional layers were used, followed by a dropout layer (dropout rate: 0.5), three fully connected layers, and a final linear layer that estimates two control commands and the distance to the membrane. The two control values ​​output by the output layer as control predictions may represent Bishop curvature, which will be described in more detail below. Exponential linear units (ELUs) were used as the nonlinearity (see, for example, DA-Clevert et al., 2015, "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)," Iclr. 566, https: / / doi.org / 10.3233 / 978-1-61499-672-9-1760).

[0340] For describing the inputs and outputs of the network, the Bishop system and associated Bishop curvatures can be used, as described in the technical section below. The Bishop system is a local Cartesian coordinate system in which a tangent vector and two normal vectors span, and obeys the rotation-minimum property of not allowing any twisting around the tangent vector. The alignment projection plane of the neurites with respect to the input of the AI ​​model can be determined by the Bishop normal vector, while the flight direction of the network can correspond to the tangent vector.

[0341] The evolutions of the Bishop system unit vectors, one for each normal vector direction, are connected via Bishop curvatures.

[0342] Bishop curvature can be the output of an AI model output layer used for steering. Each of the two Bishop curvatures is similar to a signed curvature for the corresponding normal vector direction. Signed curvatures are for only one steering direction, corresponding to a planar curve, but they have also been used as steering commands for image-based road following (see, for example, K. M. Bojarski et al., 2016, "End to End Learning for Self-Driving Cars," ArXiv E-Prints, https: / / arxiv.org / pdf / 1604.07316.pdf).

[0343] The task of predicting the continuity of a neurite in the form of Bishop curvature can be interpreted as fitting a parabola to the centerline of the neurite. The curvature vector, calculated as the sum of the Bishop curvatures multiplied by their corresponding Bishop normal vectors, determines the direction and magnitude of the curvature of this parabola.

[0344] Thus, the Bishop curvature output by the AI ​​model output layer can be integrated to determine the next in a series of points along the centerline (the series is unknown a priori and determined to trace the centerline).

[0345] training The operation of a trained AI model is described above. The systems and methods disclosed herein may operate to train an AI model for its intended use.

[0346] 15 is a flowchart of a method 140 that includes both AI model training 141 and operation of the trained AI model during inference 142. The use of the trained AI model in step 142 may be implemented using any one of the techniques described herein for inferring the direction of extension of an elongated object.

[0347] The training in step 141 may be supervised training. A set of annotated 3D-EM images may be used in the training. The training may include gradient-based updating of the AI ​​model parameters. Techniques known to those skilled in the art, such as gradient descent techniques, may be used to update the model parameters.

[0348] Training in step 141 may include training the AI ​​model to follow a flight policy that causes it to converge back to the centerline 50, even when starting from or reaching a position away from the centerline. To achieve this, training step 141 may include training the AI ​​model in scenarios where the AI ​​model starts from a position away from the centerline 50, and / or possibly, predictions of the AI ​​model during training may be integrated and fed back when made inferences that also result in positions away from the centerline. A flight policy may be implemented that enforces convergence back to the centerline 50, even when the initial subvolume (i) starts from a location offset from the centerline and / or (ii) is not aligned with the direction of the elongated object. Thus, training may also be performed using subvolumes where none of the voxel faces of the subvolumes 91, 93 have centers that coincide with locations along the centerline 50. For example, at least some of the subvolumes may be deliberately chosen so that the center of the fourth face (of the 16 faces of the subvolume for the model in FIG. 13) has an offset from a point on the centerline determined by the annotated data or in a previous iteration. The offset may be oriented transverse to the previously determined extension direction (i.e., transverse to the steering prediction in the Bishop system).

[0349] Training can be implemented in a manner that uses one or several training parameters that are varied during training, allowing different policies to be implemented during training.

[0350] The results of training models for elongated objects are much improved using such training techniques.

[0351] During training in step 141, the convergence distance of the flight policy may determine how quickly the iterative process converges back to the centerline. The convergence distance may be a dynamic parameter that is varied during training of the AI ​​model. The convergence distance may be set as a function of distance from a physiological boundary during training of the AI ​​model. The convergence distance may be a monotonically increasing function of distance from a physiological boundary, thereby implementing a stronger return to the centerline at locations closer to a physiological boundary, such as a cell boundary.

[0352] As an alternative to supervised learning with a handcrafted flight policy, a flight policy can also be learned directly using reinforcement learning by instead defining a reward function that promotes stable tracing of elongated objects.

[0353] As a specific example, the CNN architecture in Figure 13 can be trained for a neurite-tracking task using TensorFlow (M. Abadi et al., 2016, "TensorFlow: A System for Large-Scale Machine Learning." In 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI'16), 265-537 84). AI models minimize the mean squared error plus an L2 regularization loss term in neural network training parameters using RMSProp (e.g., T. Tieleman and G. Hinton, 2015, "Neural Networks for Machine Learning Lecture 6a: Overview of Mini-batch Gradient Descent," Lecture 2) with momentum (D.E. Rumelhart et al., 1986, "Learning Representations by Back-Propagating Errors," Nature 323(6088):533-36; I. Sutskever et al., 2013, "On the Importance of Initialization and Momentum in Deep Learning," In International Conference on Machine Learning, pp. 1139-1147). Notes, 2015, see http: / / www.cs.toronto.edu / ~tijmen / csc321 / slides / lecture_slides_lec6.pdf), or Adam (DP Kingma and JLBa, 2015, "Adam: A Method for Stochastic Optimization." In 3rd International Conference on Learning Representations, ICLR 2015-Conference 607 Track Proceedings, https: / / arxiv.org / abs / 1412.6980), with a mini-batch size of 128.

[0354] Training and subsequent testing show that training for positions away from the centerline can improve performance during inference. During inference, the current position and orientation depend on past AI model decisions. Therefore, even small errors in past network manipulation predictions can cause the network to move away from the neurite centerline.

[0355] To improve the stability of path following (i.e., tracing), the AI ​​model can also be trained for off-center positions and off-axis directions, and the steering prediction adjusted accordingly results in returning to the neurite centerline. To sample off-center positions within the neurite, the diameter and / or shape of the neurite must be known, at least approximately, during training. Volumetric reconstruction of the neurite can be used to determine off-center positions within the neurite. The mapping from a specific off-center position, off-axis direction state to a suitable steering is also referred to as a flight policy.

[0356] As described in more detail herein, a greedy flight policy can be used that is based only on local information available to the AI ​​model within a finite field of view. The distance to converge back to the centerline is a free parameter of this flight policy. Different convergence distances can be used for positions further from the center within the synaptic bouton. When closer to the membrane "bump," stronger steering with a shorter convergence distance is useful for staying within the neurite, while for larger distances to the membrane, gentler steering is available to better maintain alignment of the neurite with respect to the input field of view. Therefore, during training, a dynamic convergence distance can be used that can be set depending on (e.g., equal to) the distance to the membrane along the flight direction. This invokes the concept of obstacle, i.e., membrane, avoidance, and empirically works better than using a constant value for the convergence distance during training.

[0357] FIG. 16 shows a volume 40 of 3D training data. A neurite 151 is annotated in the 3D training data. The AI ​​model is also trained using subvolumes, such as exemplary subvolume 156, around a point, such as exemplary point 152, which is offset from the centerline of neurite 151 (located at the center of one voxel face of subvolume 156). Flight direction 153 may be significantly different from the extension direction of the neurite within subvolume 156 during training. One or several training parameters, such as the convergence distance, which determines how quickly the process converges back to the centerline of the neurite, may be dynamically varied. For example, the convergence distance may be set solely depending on the distance 155 from the cell membrane 154, which is shown diagrammatically. Distance 155 may be measured along the instantaneous flight direction.

[0358] Error rate performance To evaluate performance, the image processing system and method is evaluated on a validation set and two test sets of straight axon branches by performing iterative guesses starting from both sides. As disclosed in more detail herein, reaching a set of distance or angle thresholds relative to the ground truth trace triggers a reset to the ground truth trace. When the network reaches the other end of the axon branch, the tracing is stopped.

[0359] Each operational error resulting in a reset corresponds to both a merge error, which merges into the wrong process, and a split error, which does not continue the neurite of interest. Each reset is counted as two errors. The error rate is the number of errors divided by the path length of the ground truth skeleton. This metric allows for model selection in the case of multiple trained AI models based on a validation set, and for rough comparison with other automatic methods that report the distance between errors, i.e., the inverse of the error rate, as well as comparison with human error rates.

[0360] The image processing system and method were trained on a 3D-EM dataset from layer 4 of the primary somatosensory cortex of a mouse acquired using serial block-face scanning electron microscopy (SBEM (W. Denk and H. Horstmann, 2004, "Serial Block-Face Scanning Electron Microscopy to Reconstruct Three-Dimensional Tissue Nanostructure.", edited by Kristen M. Harris, PLoS Biology 2(11)). The dataset measured 61.8 x 94.8 x 92.6 μm 3 and 11.24x11.24x28nm 3 has a voxel size of

[0361] For model selection, the image processing system and method were evaluated on a validation set consisting of 13 axons with a path length of 1.4 mm. For the validation set, axonal branches with a path length less than 5 μm were excluded for more robust heuristic detection of errors. The axons in the training and validation sets were disjoint random subsets of axons seeded with presynaptic classification segments obtained using the technique disclosed in B. Staffler et al., 2017, "SynEM, Automated Synapse Detection for Connectomics," ELife 6 (July):e26414. Averaged across all models and training iterations, the random guessing mode outperformed its normal mode counterpart by 33% (range: 12%-52%).

[0362] An AI model identified as having particularly good performance on the validation set uses only EM data as input, has an ELU activation function, and was trained for 700,000 training iterations, yielding 16.4 errors / mm based on heuristic error detection. Manual inspection of the traces of the inventive technique yields 1 false positive and 0 false negative resets (96% precision, 100% recall). The true error rate is 15.7 errors / mm.

[0363] Application of the selection AI model to a set of 59 soma-seeded axons with a path length of 6.2 mm yields 29.8 errors / mm with 185 resets (95% precision, 95% recall) with 9 false negative resets and 5 false positive resets. The true error rate is 30.4 errors / mm.

[0364] Recognizing that soma-seeded axons are biased toward larger diameters, tests were also performed on a random set of 10 axons with a path length of 1.6 mm, previously used to quantify human error rates (see, e.g., Boergens et al., 2017, "WebKnossos: Efficient Online 3D Data Annotation for Connectomics." Nature Methods 14(7):691-94). On this test set, the image processing systems and methods disclosed herein yielded 58.1 errors / mm via heuristic error detection. Manual inspection revealed 4 false positive and 2 false negative resets (96% precision, 98% recall), resulting in a true error rate of 55.6 errors / mm.

[0365] Possible Further Training and Testing Implementations Exemplary training data and evaluation techniques have already been described above. Additional possible 3D-EM datasets of neural tissue, as well as validation and test sets within the training data, that can be used in conjunction with the methods and systems disclosed herein, are now provided.

[0366] The method and system described previously densely reconstructed 92.6 x 61.8 x 94.8 μm connectomic region from layer 4 of the primary somatosensory cortex in a 28-day-old rat by A. Motta et al. (2019), Dense connectomic reconstruction in layer 4 of the somatosensory cortex. Science, eaay3134, doi:10.1126 / science.aay3134. 3SBEM datasets (W. Denk and H. Horstmann (2004), Serial block-face scanning electron microscopy to reconstruct three-dimensional tissue nanostructure. PLoS Biology, 2, e329, doi:10.1371 / journal.pbio.0020329) can be trained on and / or used. Tissues are conventionally stained en bloc (K.L. Briggman et al. (2011), Wiring specificity in the direction-selectivity circuit of the retina. Nature, 471, 183-188, doi:10.1038 / nature09818) and measure 11.24x11.24 nm. 2The images were imaged at a nominal section thickness of 28 nm. Training and validation axons were sampled from the seeded axon set by presynaptic classification segments obtained using SynEM (B. Staffler et al., (2017), SynEM, automated synapse detection for connectomics. ELife, 6, e26414, doi:10.7554 / eLife.26414) and annotator-traced scaffolds. To obtain a volume reconstruction of the training axons, oversegmented segments obtained using SegEM (M. Berning et al., (2015), SegEM: Efficient Image Analysis for High-Resolution Connectomics. Neuron, 87, 1193–1206, doi:10.1016 / j.neuron.2015.09.003) with parameters similar to those set for whole-cell segmentation of the cortical dataset were also picked and combined. A volume mask may also be used to iteratively optimize the interpolated skeleton trace to yield a more robust centerline approximation. A set of 14 axons with a path length of 1.2 mm may be used for training, with up to 700,000 weight updates performed, corresponding to approximately 260 epochs. The validation set may consist of 13 axons with a path length of 1.4 mm, where branches less than 5 μm (2.5 μm) were excluded for more robust heuristic error detection. 3 A third set of 10 axons with a path length of 1.7 mm seeded from the bounding box of the vertex can be used as a test set. These are the same axons previously used to evaluate human and semi-automated segmentations (K.M. Boergens et al., (2017), webKnossos: efficient online 3D data annotation for connectomics. Nature Methods, 14, 691-694, doi:10.1038 / nmeth.4331; Motta et al., 2019, loc.cit.).

[0367] For automatic spine head addition on this dataset, an axon-trained AI model (e.g., as shown in Figure 14 and described in more detail herein) can be evaluated against a random subset of 50 spine heads previously added by human annotators, as well as against the set of spine heads previously used as a test set in Motta et al., 2019, loc.cit.

[0368] The AI ​​model of the methods and systems disclosed herein was then sectioned at 35 nm using ATUM (K.J. Hayworth et al., (2006), Automating the collection of ultrathin serial sections for large volume TEM reconstructions. Microscopy and Microanalysis: Cambridge University Press) and at 4x4 nm using a multibeam scanning electron microscope (A.L. Berle et al., (2015), High-resolution, high-throughput imaging with a multibeam scanning electron microscope. Journal of Microscopy, 259, 114-120, doi:10.1111 / jmi.12224). 2 A 1.3x1.3x0.25mm section from the barrel cortex of a 28-day-old rat can be stained according to the protocol by Y. Hua et al. (2015), Large-volume en-bloc staining for electron microscopy-based connectomics. Nature Communications, 6, 7923, doi:10.1038 / ncomms8923, with minor modifications. 3 It can be trained, tested, and applied on a subset of the dataset.

[0369] 8x8x35nm (downsampled by a factor of 2 in x and y) 3Segmentation and agglomeration, which may be applied to datasets at a resolution of 100 MHz, may be performed based thereon, for example, using conventional methods or techniques. Briefly, a 3D U-Net (O. Ronneberger et al., (2015), U-net: Convolutional networks for biomedical image segmentation. Lecture Notes in Computer Science (including the subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): Springer, Cham) may be used to predict voxel similarity along each major axis according to Lee, K., Zung, J., Li, P., Jain, V., and Seung, H.S. (2017), Superhuman Accuracy on the SNEMI3D Connectomics Challenge. arXiv e-prints, where watershed-based oversegmentation is generated. Segments from the over-segmentation can then be combined using hierarchical agglomeration, such as that proposed in J. Funke et al. (2019), Large Scale Image Segmentation with Structured Loss Based Deep Learning for Connectome Reconstruction. IEEE Trans Pattern Anal Mach Intell, 41(7), 1669-1680, doi:10.1109 / TPAMI.2018.2835450. Additionally, neurite type prediction, vessel and nucleus detection can be incorporated into the agglomeration to further reduce merging errors. Neurite type prediction is also used for spine head detection, which was the basis for spine head addition based on the methods and systems disclosed herein.

[0370] For training the AI ​​model of the methods and systems disclosed herein for axons, a training set can be obtained based on a set of 10 cell body-seeded axons from layer 4, where an annotator can pick segments from the over-segmentation to obtain a volume reconstruction. Kimimaro (W. Silversmith et al., (2021), Kimimaro: Skeletonize densely labeled 3D image segmentations. In GitHub repository (v 3.0.0 ed.)) can be used, followed by subsampling and B-spline interpolation, to extract a centerline skeleton from the volume reconstruction. This can result in a training set with a total axon path length of 21 mm. As a validation set of axons, a set of axons of size (2 μm) can be used. 3 A random selection of 10 out of 20 axons from the bounding box of layer 4 seeding yields a path length of 1 mm (50 μm). 3 For the test set of axons, a separate (1.5 μm) bounding box in layer 4 was used. 3 The bounding box of a random subset of 5 axons was densely annotated, yielding a path length of 1.7 mm (150 μm) (accessible at https: / / wklink.org / 7122). 3 can be traced within the subvolume Si150L4 of the

[0371] Separate AI models according to the methods and systems disclosed herein can be trained on spine attachment, for which training and validation sets consist of 20 (50 μm) spine heads sampled within layer 4 and annotated for spine heads. 3 A subset of about 1000 spine heads can be generated from a set of bounding boxes of 4x4x35 nm by the annotator from the spine head through the spine neck to the dendrite trunk. 3Here, we again used Kimimaro (W. Silversmith et al., 2021, loc.cit.) to extract centerline skeletons from the training volumetric mask, resulting in a 2 mm spine neck trace. Additionally, a random subset of 76 spine heads with 0.2 mm path lengths from 20 bounding boxes can be traced to serve as a validation set. Evaluation was performed on another randomly selected (5 μm) set containing 91 densely annotated spine heads against a skeleton trace of the dendrite trunk. 3 This can be done on the bounding box of (Si11L3 accessible at https: / / wklink.org / 2458).

[0372] To evaluate error correction for another state-of-the-art segmentation that can be performed using the methods and systems disclosed herein, the methods and systems disclosed herein perform a 4x4x33 nm voxel reconstruction and agglomeration using flood-filling networks (FFN; M. Januszewski et al., (2018), High-precision automated reconstruction of neurons with flood-filling networks. Nature Methods, doi:10.1038 / s41592-018-0049-4). 3 Recently published mm 3 This technique can be applied to multi-SEM datasets (A. Shapson-Coe et al., (2021), A connectomic study of a petascale fragment of human cerebral cortex. bioRxiv, 2021.2005.2029.446289. doi:10.1101 / 2021.05.29.446289). Specifically, training and evaluation of the technique involved a 6.5 mm axonal path length (150 μm) relative to the provided ground truth skeletal traces. 3The published ground truth skeletal traces for densely seeded axons can be used to evaluate the FFN for cell body-seeded axons. To fine-tune the processing by the methods and systems disclosed herein for this dataset, the ground truth skeletal traces for densely seeded axons are focused on the central (15 μm) 3 Size (2.5 μm) within the bounding box of 3 A bounding box of 1.25 mm is sampled and all processes are annotated in this bounding box, resulting in a path length of 1.25 mm (150 μm). 3 The model can be generated by sampling a random subset of five axons that can be traced throughout the bounding box of the vertex. Here, skeletal annotation has already been performed with high accuracy along the centerline, so that no post-processing is necessary. The volumetric neurite mask required for training can be generated by segment picking from the C3 FFN segmentation (see Shapson-Coe et al., 2021, loc.cit.). The best model checkpoint for the model used by the method and system according to embodiments was then used as an initial setting based on the validation set from training axons in the ATUM-multiSEM dataset of murine cortex. The model can be trained for another approximately 2 million gradient updates until convergence in terms of reset-based error rate on the training axons is achieved.

[0373] For evaluation of the FFN and the methods and systems disclosed herein (see, e.g., FIG. 26 and accompanying description), size (1.5 μm) 3 Another bounding box of 1.4 mm results in a path length of 150 μm. 3 A random subset of five axons traced throughout the bounding box of the vertex can be annotated.

[0374] Additionally, the methods and systems disclosed herein include: Artificially generated data that mimics the appearance of elongated objects that are traced during inference X-ray, in particular X-ray holographic nanotomography, optical microscopy, or magnetic resonance imaging of one or many elongated objects may be trained or used in any or a combination of:

[0375] The amount of path-length training data required for the methods and systems disclosed herein to function as described may generally depend on, among other things, the resolution and visual complexity of the data, the variability in the form and appearance of the elongated objects being traced, the algorithm and artificial neural network architecture employed for training, and the number, type, and order of image data used for training.

[0376] Performance compared to other image processing methods The performance of the image processing systems and methods disclosed herein was also compared with other techniques.

[0377] We investigated to what extent the techniques disclosed herein can replace human annotation and, therefore, what types of connectomics analysis can be fully automated. To this end, we performed a dense connectomics analysis of a piece of mammalian cortex (A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." Science, Vol. 366, Issue 6469, p. eaay3134). Based on the automated segmentation, human annotators were asked to resolve problematic locations in the automatically identified set, spending a total of 4,000 working hours. To resolve segmentation errors, human queries were performed on the ends of the object, asking to continue when possible and stopping the task when another automatically reconstructed object was reached. To resolve connection errors, chiasma configurations were detected and the exits were queried for the appropriate continuation to one of the other chiasma exits. The techniques disclosed herein were used to replace these human annotations by starting the techniques disclosed herein with similar queries from human annotators. Indeed, 76% of the terminal tasks and 78% of the chiasma tasks were correctly traced by the techniques disclosed herein (human: 74% and 94%, respectively). Furthermore, running the techniques disclosed herein back from the end point to the start provided an error signal, allowing most merging errors to be avoided, with the new techniques disclosed herein resulting in only 4% of queries that produced merging errors for terminal tasks and 1% for chiasma tasks, which is comparable to human annotations.

[0378] While this performance demonstrated that the techniques disclosed herein can faithfully replace human annotations, we wanted to quantify this using direct connectomics analysis as an indicator of reconstruction success. Some connectomics analyses require higher reconstruction accuracy than others. In particular, the following were considered: (1) paired same-axon-same-dendrite synapse analysis (Motta et al., loc.cit.), which aims to measure the small connectomes being learned; (2) spine ratio analysis for the identification of interneuron dendrites; and (3) axon type analysis based on the distribution of axon synaptic targets. Three types of connectomes were used for comparison. (I) A connectome obtained from a fully automated reconstruction in (Motta et al., loc.cit.) before any human annotation of note; (II) A connectome in (Motta et al., loc.cit.) including 4,000 hours of human annotation; and (III) A connectome obtained from combining the fully automated reconstruction in (Motta et al., loc.cit.) with the techniques disclosed herein, resulting in a fully automated, automatically calibrated connectome. These three types of connectomics analyses (I), (II), and (III) were performed. While paired synapse analysis already yields similar results with the automated state before correction achieved by the inventive techniques, correct spine proportions for apical dendrites and truly few excitatory axons, as determined by spine head selection, require either human correction or correction achieved by the inventive techniques. Furthermore, the correction achieved by the inventive techniques restores axon target specificity of inhibitory axons to apical and smooth dendrites, whereas this specificity was not detectable in the automated state prior to the techniques disclosed herein and human correction. Thus, when utilizing automated connectomics metrics that are limited by the types of connectomics problems that can be analyzed automatically, error correction using the techniques disclosed herein shifts automated analysis performance from simpler to more complex connectomics problems.

[0379] The techniques disclosed herein were also applied to datasets from automated tape-collecting ultramicrotome (ATUM) / multiSEM imaging techniques, which are substantially larger and offer different image resolution and contrast. The utility of the techniques disclosed herein was tested for seeded axon reconstructions, i.e., reconstructions of axons beginning at the cell body. After initial segmentation and agglomeration steps, terminals were interrogated using the techniques disclosed herein in the following manner: axon terminals were detected, and the techniques disclosed herein traced them until a new agglomerate of 2.5 μm in length was found and one of several stopping criteria was met. To avoid mismatches, the techniques disclosed herein validated the traces in a backward direction, and new agglomerates were added only if several type and angle criteria were met. A second inventive tracer trained on the spine neck was used when the axon training techniques disclosed herein failed. Using 20 iterations per axon, we found that the technique disclosed herein can increase axonal recall from 7% to 37% (median: 6% to 44%; precision decreased from >99% to 74%), enabling local connectomics analysis in volumes approximately 200-300 μm on a side, corresponding to connections within local layers of the murine cortex.

[0380] Similar to thin axons, thin spine necks can pose problems for current reconstruction pipelines. However, to assign synapses to the correct cells, spine necks need to be reconstructed with high accuracy. Therefore, we tested whether the techniques disclosed herein could improve the precision / recall of spine head addition from global agglomerations (100% / 43%) and local agglomerations (100% / 75%, similar to Motta et al., loc.cit.). To do this, the techniques disclosed herein begin tracing the detected spine heads for the two candidate directions with the lowest uncertainty in the prediction. The techniques disclosed herein trace for a maximum of 7 μm, or until a dendritic agglomerate of sufficient size is reached. The candidate order, type, and additional information are then used to add spine heads to the dendritic agglomerates. This increases the recall of spine head addition from 75% to 94%, while maintaining a high accuracy of 97%.

[0381] Our technique also offers improvements over techniques such as Flood-Filling Networks (M. Januszewski et al., 2018, "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4). Application of our technique to automatically detected terminals from Flood-Filling Networks (FFN) segmentations resolved 60% of the divisions in a random set of axons traced throughout a subvolume, thereby reducing the division rate from 65-70 divisions / mm to 28-31 divisions / mm, while introducing only a few new misconnections (the connection rate increased from 3-4 connections / mm to 5-6 connections / mm). In particular, mm 3Evaluation of random axons that are not necessarily connected to cell bodies within a volume of the scale yields images of sufficient reconstruction quality than restriction to axons close to cell bodies. Evaluation of FFN restricted to axons connected to cell bodies within a volume, as previously used to demonstrate reconstruction performance (Januszewski et al., 2018), underestimates the segmentation and connection errors of random axons by a factor of 3–4 (15–22 divisions / mm, 0.2–1.1 connections / mm).

[0382] As demonstrated by these results, the techniques disclosed herein provide sufficient orthogonal data to be used as a fully automated annotation technique for a wide variety of cases, improving all currently available automated segmentations. This was particularly evident for segmentations obtained using flood filling networks (FNNs), the current state-of-the-art technique. The computational complexity of the techniques disclosed herein is advantageously lower than other approaches, and their straightforward end-to-end approach may enable further optimization of both accuracy and computational efficiency, the next major challenge in connectomics for upcoming exabyte-scale datasets (A. Motta et al., 2019, "Big data in nanoscale connectomics, and the greed for training labels," CurrOpNeurobiol, Volume 55, April 2019, Pages 180-187).

[0383] Efficiency compared to other image processing methods To estimate the computational cost, we benchmarked the inventive techniques disclosed herein against approximately 128,000 terminal tasks from the first set of terminal detection tasks in (A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." Science, Vol. 366, Issue 6469, p. eaay3134). The step size factor was set to f = 5, and an analytically derived formula for the Bishop system along a parabola was used to integrate the operational prediction. The total run time on a single node using 32 cores (Intel® Xeon® Gold 6130 CPU, 2 sockets), a single Tesla V100 GPU (PCIE-16GB), and ~128GB RAM was 13.6 hours for the reconstruction of ~2.1 meters of axons (including the back validation trace, a total of 64 million CNN inferences), thus a reconstruction speed of ~160 mm / hr (~1300 steps / sec, 33 nm average step size). From this, and previous runs of our technique with f=1 and integration using the forward Euler method, we extrapolated to a total run time of 27.9 hours for the axon and spine neck trace (4.3 meters) and 7.6 hours for the initial direction prediction (for ~138,000 spine heads) on a single node for automatic error correction on the reconstruction state from before human intervention (Motta et al., 2019), resulting in a total of 35 single-GPU node hours. For Flood-Filling Networks, 6.964 GPU node hours were multiplied by 1000 (NVIDIA P100) GPUs (Januszewski et al., 2018).For the Local Shape Descriptor, the AcRLSD architecture (Sheridan et al., 2021, "Local Shape Descriptors for Neuron Segmentation." bioRxiv, 2021.01.18.427039v1, doi:10.1101 / 2021.01.18.427039v1) took a total of 10.5 hours on 24 NVIDIA V100 GPU nodes, while watershed and agglomeration took 7.7 CPU node hours on 100 cores. For (Motta et al., 2019), dense reconstruction, including segmentation, agglomeration, type, and synapse prediction, as well as processing of the human skeletal reconstruction, took 101 hours on 24 nodes with 16 CPU cores each. For all methods, CPU and GPU times were multiplied by the ratio of the respective dataset sizes (Motta et al., 2019), resulting in the computational cost estimates provided in the table below. [Table 1]

[0384] Operation of the system and method for image processing combined with other techniques (e.g., as an error corrector) The processing of 3D EM data or other volumetric image data using trained AI models as disclosed herein can be combined with other reconstruction techniques. By way of illustration, the techniques disclosed herein can be used to perform connectomics reconstruction (or tracing of other 3D elongated objects) to replace human annotation in conjunction with traditional annotation or tracing techniques. When used as an error corrector, the techniques disclosed herein enable fully automatic connectomics reconstruction of relevant volumes.

[0385] In embodiments, the image processing systems and methods disclosed herein may be used in conjunction with other reconstruction techniques. An automatic reconfiguration process may be performed, In response to detecting a missed sequence of automatic reconfigurations, the iterative techniques disclosed herein may be initiated at that point and run until it finds another component that has been automatically reconfigured. From that point on, Optionally, upon finding another automatically reconfigured component, the iterative techniques disclosed herein may be performed backwards to double-check that it ends up at the same starting location (for verification purposes); The skeletal reconstruction of the procedure defined above can be used to correct errors in the automatic reconstruction, optionally in an iterative manner.

[0386] Furthermore, as an example, An automatic reconfiguration process may be performed, In response to detecting an erroneous sequence or combination error of the automatic reconstruction, the iterative technique disclosed herein may be started at several locations around the error and may be performed until it leaves a bounding box around the location of the error; Optionally, upon leaving a bounding box around the location of the error, the iterative technique disclosed herein may be performed backwards to double-check that it ends up at the same starting location (for verification purposes); The skeletal reconstruction of the procedure defined above may be used to correct errors in the automatic reconstruction, optionally in an iterative manner, and further optionally interleaved with iterations of detection and correction of missed sequences.

[0387] A flow chart of the method is shown in Figure 17. The method can be performed automatically by an image processing system.

[0388] In step 161, an initial 3D image analysis is performed. The initial 3D image analysis can be performed to obtain a 3D connectomics reconstruction. Techniques such as, but not limited to, those described in A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." Science, Vol. 366, Issue 6469, p. eaay3134 or M. Januszewski et al., 2018, "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4 can be used to implement step 161.

[0389] In step 162, an image processing system or method disclosed herein may be used to act as an error corrector. An image processing system or method may be employed to trace neurites (e.g., axons) when other automatic techniques are unable to annotate the image data.

[0390] As an illustration, the techniques disclosed herein were applied to all detected spine heads determined using the technique of A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex," Science, Vol. 366, Issue 6469, p. eaay3134, which previously required human annotation (38% of all detections required approximately 900 annotator hours). A precision / recall rate of 89% / 84% was achieved for the added spine heads, approaching the accuracy of the human determinations. Starting positions from the spine head detections, also provided to the human annotators, were used. Direction prediction using the techniques disclosed herein was then performed. The iterative guessing mode of the techniques disclosed herein was applied to the candidate direction with the lowest prediction uncertainty, and tracing was performed until the dendrite trunk defined by the dendrite mask was reached or a path length threshold was exceeded. Notably, the precision / recall reported above was achieved without retraining the techniques disclosed herein on spine necks. Further improvements can be made by training on a dedicated training set from spine necks.

[0391] The starting positions and orientations from the technique of A. Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex," Science, Vol. 366, Issue 6469, p. eaay3134, which previously required thousands of annotator hours to correct segmentation and connection errors in axon reconstructions, were fed into the method and system disclosed herein. For segmentation error resolution, the technique disclosed herein was applied in small extensions, and each extension was verified by running the method backward. Only if the verification was successful did the trace continue to the next extension until it reached a known axon or the end of the dataset. The resulting skeletal trace could then be used similarly to human annotation. For connection error resolution, the stopping criterion was again similar to humans, based on a bounding box around the connection error, and the trace was accepted if backward verification resulted in the same skeletal reconstruction with some error tolerance. Resolution of segmentation and joining errors was performed iteratively until the remaining errors were reduced below a threshold. The final axon reconstruction for this fully automated method yielded 12 segmentation errors / mm and 16 joining errors / mm on a representative set of 10 axons (1.7 mm), which is comparable to or close to the error rates of human annotation (5 segmentation errors / mm and 12 joining errors / mm), and outperforms previous automated techniques (Motta et al., 2019, before human annotation, 32 segmentation errors / mm and 14 joining errors / mm).

[0392] Directional Prediction As described herein, image processing systems and methods operate to trace elongated objects by iteratively determining the local direction (referred to as the flight direction) in which the elongated object extends.

[0393] The initial direction (i.e., the direction used to select a subvolume for analysis by the AI ​​model in the first iteration) may be specified by user input. The image processing system may operate to control a GUI to enable the respective user input for subsequent use by the elongated object tracing technique disclosed herein. Alternatively, when the initial direction is not obtainable in this manner or is otherwise unknown a priori, the initial direction may be set as a random direction (the AI ​​model typically guides the process back to the centerline in terms of its training). Alternatively, a more targeted approach may be used, in which Monte Carlo dropout is used on multiple candidate orientations. The implementation of direction selection from the candidates may depend on the stability as estimated by the image processing system and method. More robust predictions (i.e., direction predictions with smaller uncertainty) are prioritized over predictions with higher uncertainty in the selection process.

[0394] 18 is a flow chart of method 165. Method 165 may be performed automatically by an image processing system or method.

[0395] Direction sampling is performed in step 166. A set of directions that may be equidistant or nearly equidistant on the surface of the unit sphere may be sampled.

[0396] In step 167, uncertainty estimates may be calculated for various directions. Calculating the uncertainty estimates may be done in various ways. By way of illustration, uncertainty estimates may be determined for multiple directions using a process that includes performing Monte Carlo dropout, calculating the covariance of the direction estimates (such as Bishop curvature) for each orientation, and determining the uncertainty estimates based on the covariance matrix of the direction estimates.

[0397] In step 168, one or several directions (eg, initial directions for the iterative guess) may be determined based on the uncertainty estimate.

[0398] Actions and Effects The image processing system and method addresses the need for a technique capable of performing automated axon tracing (while also being suitable for other applications).

[0399] The image processing system and method addresses axonal reconstruction in a manner similar to the flight reconstruction task, yielding a skeleton by a series of visit points rather than a volumetric (voxel) reconstruction. The image processing system and method provides an automated approach to predicting skeletal reconstructions in the field of connectomics directly from raw EM data.

[0400] The image processing system and method offers the possibility of end-to-end optimization. The image processing system and method focuses on the essence of neurite reconstruction, i.e., long-distance continuing neurites, throughout the dataset.

[0401] The image processing systems and methods take advantage of the fact that neurite reconstructions and error measures at the skeletal level are more robust than their voxel-based counterparts, which suffer from ambiguities in the similarity of individual voxels, e.g., the presence of staining or imaging artifacts.

[0402] The image processing system and method provides a new approach to neurite tracing that is skeleton-based rather than voxel-based and shows excellent results.

[0403] 19 and 20 illustrate the operation of the image processing system and method. The image processing system and method operate to trace a neurite (e.g., an axon) through a 3D volume represented by image data. Here, tracing refers to determining a 3D series of points or a 3D continuous curve that specifies the trace of the neurite. Subvolumes 172-274 of the series 171 are analyzed using the techniques disclosed herein. Multiple planes 175 (e.g., 16 planes) spaced along the flight direction each contain image data determined from the original image data using techniques such as interpolation or projection (e.g., trilinear projection). Information such as distance from the membrane can be determined and used, for example, in AI model training. Additionally, membrane likelihood 176, as predicted by other methods, can be provided as input to the AI ​​model; however, in experiments, this is only useful in the initial training phase, while ultimately, the raw EM data 175 provides sufficient information regarding the neural continuum for the method to be successful.

[0404] 20 illustrates a locally varying coordinate system 180, such as the Bishop system, that may be employed in the techniques disclosed herein. In effect, the image processing system and method operates by successively analyzing subvolumes of a larger tissue volume, where the subvolumes represent the field of view seen while flying along an already traced portion of the centerline of a 3D elongated object. The Bishop curvature vector 185 may be the output of the AI ​​model and may be integrated to determine new points along the centerline.

[0405] Techniques for training the AI ​​model are also described in detail. As explained, training the AI ​​model for positions and / or orientations away from the centerline provides improved stability. Error rates as low as approximately 14 errors / mm have been achieved even for complex tasks such as axon tracing or spine neck applications, as described in connection with the comparison to prior art techniques above. As shown by graph 190 in FIG. 21, the error rate saturates with increasing number of training iterations. In the example, approximately 700,000 training iterations were used herein for performance evaluation. Larger training sets may require more training iterations for saturation.

[0406] The image processing systems and methods disclosed herein can be used in a variety of applications, such as connectome reconstruction (Figure 22), analysis of biopsy samples (Figure 27), and / or artificial intelligence derived from neuroscience (Figure 28).

[0407] 22 is a flowchart of method 200. Method 200 may be performed by or using an image processing system such as those disclosed herein. In step 201, a 3D EM image is acquired. The 3D EM image may be acquired on a biopsy sample. A biopsy sample is a sample that is not implanted back into a living animal or human after imaging.

[0408] At least one neurite trace is determined in step 202. Several neurite traces may be determined simultaneously, for example, by running several instances of an image processing system in parallel. Furthermore, this step may include more complex logic for iterative reconstruction and may include module 80 from FIG. 5 as a submodule.

[0409] Termination criteria are checked in step 203. If the termination criteria are not met, the method returns to step 202. The method may continue until all neurites of interest have been traced.

[0410] In this manner, several neurites can be traced in the volume. Tracing neurites can include tracing axons, e.g., determining axonal connectomes and / or dendrites.

[0411] In step 204, synaptic information is calculated, which may be done using, but is not limited to, any synaptic interface classifier known to those skilled in the art.

[0412] Once the termination criteria are met (e.g., all neurites of interest have been traced), results may be obtained in step 205. The results may be output via, but are not limited to, a GUI or a data interface.

[0413] 23 is a graph 210 depicting the error rates required for various uses related to connectomics analysis. The low error rates and large distance between successive errors achieved by the methods and systems of the present invention, as indicated by region 211, provide for the application of these techniques to use cases not previously easily addressed by conventional computer processing techniques.

[0414] Figure 24 is a graph 220 depicting the evolution of the time required for 1 mm annotation as a function of currently available techniques. The techniques of Motta et al., 2019, "Dense Connectomic Reconstruction in Layer 4 of the Somatosensory Cortex." Science, Vol. 366, Issue 6469, p. eaay3134, provide an improvement 221. Furthermore, a very significant reduction 222 in annotation consumption rate is provided by the methods and systems of the present invention as disclosed herein.

[0415] Figure 25 is a graph 230 quantifying the axonal splitting and joining rates reported for randomly seeded axons. Results 231 were obtained for FFN-based reconstructions of human cortex (A. Shapson-Coe et al., (2021), "A connectomic study of a petascale fragment of human cerebral cortex." bioRxiv, 2021.2005.2029.446289, doi:10.1101 / 2021.05.29.446289 (2021)). The techniques disclosed herein provide significantly improved results 241 for both segmentation conditions analyzed, along with increased join rates. Note that the splitting rate requirement depends on the synapse rate along the axon; with at least five synapses per axon, splitting rates of less than 40 per mm and less than 20 per mm are required for typical murine and human axons, respectively. Therefore, moving toward automation in these areas is desirable for more challenging connectomics analyses. The methods and systems disclosed herein improved the reconstruction of densely seeded axons for large-scale data from murine cortex. Results 232 were obtained using conventional techniques (unpublished) compared to the present technology, which provided improved results 242. For SBEM data, the methods and systems disclosed herein allow for the complete replacement of human annotation for the employed analysis, providing an improvement over results 233 obtained using conventional techniques in Motta et al., loc.cit. The improved results obtained by the methods and systems disclosed herein are shown at 243. The potential additional benefit of a two-fold further improvement that may be achievable from human annotation (dashed black line) is not relevant to the analysis performed.

[0416] Figure 26 is a further graph 260 quantifying the rate of axonal division and merging reported for randomly seeded axons. For Figure 26, the technique was applied to 3D EM data acquired using state-of-the-art high-throughput 3D EM imaging techniques tracked by a multi-beam scanning electron microscope (multiSEM). Data was acquired for both murine and human cortex. Data 261, 262, and 263 correspond to conventional techniques. Data 264 corresponds to data acquired using further human annotation. Data 271, 272, and 273 correspond to data acquired using the methods and systems disclosed herein.

[0417] After automated agglomeration, the methods and systems disclosed herein were seeded at the ends of the automatically detected axon agglomerates and used to connect the split axon agglomerates. As a result, the axonal split rate was reduced by 7-fold (from 42.7 to 6.0 per mm of axon path length), while the join rate only increased slightly (from 3.3 to 4.5 per mm, as seen in Figure 26, comparing the upper data point 262 with the lower data point 272). Directly comparing the split rate at the same rate of join errors (3.9 join errors per mm of axon path length) with a higher agglomeration, agglomeration yields 31.9 splits per mm, resolving only 25% of the split errors, while the methods and systems disclosed herein yield 9.0 splits per mm, resolving 79%. Thus, in this region of splitting / combining error, the methods and systems disclosed herein improve splitting resolution over standard agglomeration by a factor of 3.1 (FIG. 26, data point 262 versus upper data point 272).

[0418] The methods and systems disclosed herein are based on a method for analyzing tissue samples of size (150 μm) from A. Shapson-Coe et al., (2021), "A connectomic study of a petascale fragment of human cerebral cortex.", bioRxiv, 2021.2005.2029.446289, doi:10.1101 / 2021.05.29.446289 (2021). 3 We evaluated this data for a subvolume of neurons in a 3D model and found that application of the methods and systems disclosed herein to axon terminals obtained from dense reconstructions of FFN segmentations (M. Januszewski et al., "High-precision automated reconstruction of neurons with flood-filling networks." Nat Methods, 2018 Aug;15(8):605-610, doi:10.1038 / s41592-018-0049-4, Epub, 2018 Jul 16, PMID:30013046) resolved 57% of the segmentations in a random set of axons traced throughout the subvolume. While this reduced the segmentation rate from 65 divisions / mm to 28 divisions / mm, reaching the segmentation length required for automated connectomics analysis, it also introduced several new mis-connections, an acceptable connection rate for the type of connectomics analysis studied, e.g., in the context of rat cortical SBEM reconstructions (see Figure 26, where data point 261 (bottom) is compared to data point 271 (bottom), the connection rate increased from 1.7 connections / mm to 3.1 connections / mm). Comparing the reduction in segmentation errors at the same connection error rate (c2: 1.7 connections / mm) for FFN and the methods and systems disclosed herein, we find that the FFN-based agglomeration of c3 to c2 resolves only 8% of the segmentation errors of c3. In contrast, agglomeration based on the methods and systems disclosed herein resolves 28% of the segmentation errors for c3 at the same joining error rate as c2, thereby being 3.5 times more effective than FFN in segmentation resolution (Figure 26, data points 261 (top) compared to 261 (bottom) and 271 (top)).

[0419] More specifically, Figure 26 shows quantification of the rate of axonal division and joining reported for randomly seeded axons. For flood-filling networks (FFN)-based reconstructions of the human cortex (A. Shapson-Coe et al., (2021), "A connectomic study of a petascale fragment of human cerebral cortex." bioRxiv, 2021.2005.2029.446289, doi:10.1101 / 2021.05.29.446289 (2021)), two agglomeration states (c2, c3) were analyzed. While changing the FFN agglomeration parameters from c3 to c2 resolves only 8% of the c3 segmentations in dense axons (FFN data point 261), applying the top 40% of validation traces obtained using the methods and systems disclosed herein with the highest confidence from the c3 terminus resolves 28% of the segmentation errors at the same joining error rate as c2, corresponding to a 3.5-fold improved segmentation resolution (data point 271). Applying all validation traces obtained using the methods and systems disclosed herein from the c2 terminus resolves 57% of the c2 segmentation errors, further reducing the segmentation rate from 65 divisions per mm to 28 divisions per mm, with only a small increase of 1.4 joining errors per mm.In the human cortex, a segmentation ratio of less than 30 segments per mm (axonal segments approximately 30 μm or larger) is required, for example, to accurately distinguish between excitatory and inhibitory axonal segments (see S. Loomba et al., (2022), “Connectomic comparison of mouse and human cortex.” Science, July 8, 2022, 377(6602):eabo0924, doi:10.1126 / science.abo0924, Epub, July 8, 2022, PMID:35737810). Meanwhile, in the rat cortex at the same segmentation ratio, associating 10 or more synapses per axonal segment allows, for example, analysis of inhibitory target specificity (A. Motta et al., (2019), “Dense connectomic reconstruction in layer 4 of the somatosensory cortex.” Science:eaay3134). Therefore, the shift to automation in this area is crucial for most connectomics analyses. For large-scale data from murine cortex (Si150L4, accessible at https: / / wklink.org / 7122), the methods and systems disclosed herein improve the reconstruction of densely seeded axons by 7x, while only slightly increasing connection errors from 3.3 to 4.5 per mm. Taking a 90% agglomeration threshold as a baseline, partial application of traces obtained using the methods and systems disclosed herein results in the same connection error rate, while effectively reducing segmentation by 3.1x over agglomeration. For SBEM data, the methods and systems disclosed herein enable the complete replacement of human annotation for the employed analysis (A. Motta et al., (2019), "Dense connectomic reconstruction in layer 4 of the somatosensory cortex." Science: eaay3134). The 2-fold additional benefit from manual annotation (dashed black line to data 264) is not relevant to the analysis performed in that study. For a fully automated analysis of local circuits, an error rate of less than 10 per mm is desirable.Such error rates can be achieved using the techniques disclosed herein (data 272, 273).

[0420] For the evaluation of agglomeration splitting and merging errors before and after error correction performed by the method and system disclosed herein (FIG. 26), merging errors extending farther than 2.2 μm from the ground truth were detected, and we manually verified that this heuristic accurately detects merging errors. For agglomerations before correction performed by the method and system disclosed herein, each merging error was counted as 1 / 2 and divided by the ground truth path length to yield a merging error rate. This is because each merging error typically connects two neurites, and counting them as one error per neurite rather than 1 / 2 overestimates the total amount of merging errors. For the sparse evaluation of error correction performed by the method and system disclosed herein, which is limited to agglomerates that overlap with the ground truth, when performed on both evaluated multiSEM datasets, the additional merging caused by the method and system disclosed herein was counted as 1 rather than 0.5, which accounts for merging from agglomerates that do not overlap with the ground truth and are therefore not observable in the sparse evaluation. For segmentation errors, the set of agglomerates was limited to those that overlapped with the ground truth by more than 2.5 μm. This overlap threshold was introduced to avoid the dominance of segmentation errors due to numerous small agglomerates or unagglomerated segments along thin axonal extensions. Regardless of this overlap length threshold, approximately 90% of the ground truth was still covered.

[0421] To test whether the corrections based on the methods and systems disclosed herein also reduce the segmentation error rate when compared with segmentation / agglomeration for two multiSEM datasets at the same combined error rate, the following strategy was used to limit the combined error rate by applying only a subset of the traces obtained using the methods and systems disclosed herein. The prediction uncertainty using Monte Carlo dropout, as previously used to identify the spine neck orientation, was quantified for every step performed by the methods and systems disclosed herein. The maximum uncertainty in the step was then obtained, and the forward validation trace was combined with the minimum of the two uncertainty scores. Thus, this approach is based on the concept of first scoring all directions of the validation traces relative to the position with the maximum uncertainty. Then, since the forward and backward traces lead to the same flight path, the minimum of these two uncertainty scores was obtained to quantify the uncertainty of the trace as a whole. Application of validation traces obtained using the methods and systems disclosed herein to the 0.2, 0.4, 0.6, and 0.8 percentiles of trace uncertainty then resulted in the partial corrections performed by the methods and systems disclosed herein. These were evaluated for segmentation and merging error rates, and the one with the smallest segmentation error rate at the same merging error rate as the segmentation / agglomeration was added as a midpoint within the segmentation-merging error plane in Figure 26. Specifically, for the rat cortex multiSEM dataset, application of the top 80% of validation traces obtained using the methods and systems disclosed herein to agglomerations at an 85% agglomeration threshold resulted in the same merging error rate as agglomerations at a 90% agglomeration threshold.Similarly, for a multiSEM dataset of human cortex, applying the top 40% of validation traces obtained using the methods and systems disclosed herein to an FNN c3 agglomeration resulted in the same connection error rate as an FNN c2 agglomeration (A. Shapson-Coe et al., (2021), "A connectomic study of a petascale fragment of human cerebral cortex.", bioRxiv, 2021.2005.2029.446289, doi:10.1101 / 2021.05.29.446289 (2021)).

[0422] 27 is a flowchart of method 240. Method 240 may be performed by or using an image processing system such as those disclosed herein. In step 241, a 3D EM image is acquired. The 3D EM image may be acquired on a biopsy sample. A biopsy sample is a sample that is not implanted back into a living animal or human after imaging.

[0423] Image acquisition is performed on the biopsy sample in step 241. The image acquisition may be the acquisition of a 3D EM image on the biopsy sample that is discarded after imaging.

[0424] At least one neurite trace is determined in step 242. Several neurite traces may be determined simultaneously, for example, by running several instances of an image processing system in parallel.

[0425] The results of the tracing are output in step 243. The results may be output via, but are not limited to, a GUI (e.g., by outputting a visual representation of a portion of the connectome via an optical output device) or a data interface.

[0426] The image processing systems and methods disclosed herein may be used in the process of generating, training, and / or operating processing logic (such as signal processing logic) that controls at least one physical asset (such as a machine).

[0427] 28 illustrates a system 250 comprising an image processing system operating as disclosed herein, which may operate to determine information about (part of) the connectome and / or other structural information of the nervous system of a human or other animal.

[0428] System 250 includes a physical asset 252 (such as a machine) controlled by a controller 251. Controller 251 executes machine learning (ML) models to process signal inputs (which may include sensor inputs from sensor measurements) and generate control signals that affect the operation of physical asset 252.

[0429] The ML model executed by the controller 251 may be generated by the control logic generation module 36 of the processing system 30. The control logic generation module 36 of the processing system 30 may use the results of the connectome analysis to at least one of: (i) select or generate an ML model having a connectome-dependent ML model architecture; (ii) train an ML model having a connectome-dependent ML model architecture; and / or (iii) deploy the trained connectome-dependent ML model to the controller 251 for execution as signal processing logic. The controller 251 may execute the connectome-dependent ML model determined by the processing system 30 to control the physical asset 252. The controller 251 may also feed information back to the processing system 30. Illustratively, the operation of the controller 251 and / or the asset 252 may be monitored during use in the field. The processing system 30 may be caused to repeat and / or modify the tracing of the object during use in the field of the controller 251 and / or the asset 252. In this manner, the control logic may be updated during use in the field.

[0430] Alternatively, or in addition, the controller 251 may be connected to the image acquisition system 20 to control its operation. This may be done in response to the results of image processing performed by the image processing system 30.

[0431] technology Aspects of the techniques are described that may be implemented in systems and methods for image processing. It will be understood that alternative implementations may be used in systems and methods for image processing.

[0432] EM image dataset The image processing system and method were trained and tested on an SBEM dataset from layer 4 of the primary somatosensory cortex of a 28-day-old rat. The tissue was routinely en-bloc stained (KL Briggman et al., 2011, "Wiring Specificity in the Direction-Selectivity Circuit of the Retina." Nature 471(7337):183-88) and measured in an 11.24x11.24 nm 2 and imaged with a nominal section thickness of 28 nm. Oversegmentation was used to generate volumetric reconstructions of axons in the training set. Oversegmentation was obtained using the technique of M. Berning et al., 2015, "SegEM: Efficient Image Analysis for High-Resolution Connectomics." Neuron 87(6):1193-1206, with parameters similar to those set for whole-cell segmentation of the cortical dataset.

[0433] The image processing system and method were also trained and tested on a dataset from the barrel cortex of a rat acquired using a multibeam scanning electron microscope. Tissue was stained according to the protocol by Y. Hua et al., 2015, "Large-Volume En-Bloc Staining for Electron Microscopy-Based Connectomics," Nature Communications 6 (August):7923, with minor modifications, using a 4x4 nm 2 and imaged at a nominal section thickness of 35 nm.

[0434] The image processing system and method uses flood-filling networks (M. Januszewski et al., 2018, "High-Precision Automated Reconstruction of Neurons with Flood-Filling Networks." Nature Methods, https: / / doi.org / 10.1038 / s41592-018-0049-4) to segment and agglomerate voxels of 4x4x33 nm. 3 It was also evaluated against a multiSEM dataset with (A. Shapson-Coe et al., (2021), "A connectomic study of a petascale fragment of human cerebral cortex.", bioRxiv, 2021.2005.2029.446289, doi:10.1101 / 2021.05.29.446289 (2021)).

[0435] Skeletal Interpolation Interpolation can be used to obtain a continuous representation of neurite branches (e.g., axonal branches) and / or centerline nodes at a desired step size from sparsely spaced nodes along the central axis.

[0436] The interpolation can be implemented as a quartic B-spline interpolation (L. Piegl and W. Tiller, 1996, The NURBS Book. 2nd ed. Springer Science & Business Media). As an example, the curve

number

number

[0437] The adaptive step size ensures that voxels up to a radius of half the projection plane size in the projection plane at the current position of the CNN are not skipped. Optionally, the same adaptive step size may be used during inference for the inventive techniques disclosed herein to determine the step size of integration steps that determine subsequent positions and orientations.

[0438] Bishop type The Bishop system (R.L. Bishop, 1975, "There Is More than One Way to Frame a Curve." The American Mathematical Monthly 82(3):246-51) is a system of three orthogonal vectors: a tangent vector (also called a flight vector) to a curve,

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0439] The Bishop system is often used as a local coordinate system for parameterized curves rather than other systems (such as the Frenet-Serret system) that are used as local coordinate systems in the differential geometry of parameterized curves.

number

number

[0440] Flight Policy (Applying the AI ​​model to a position away from the centerline will cause it to converge back to the centerline.) The flight policies available during AI model training are (a) the corresponding Bishop system

number

number

number

number

number

number

number

number

number

number

number

[0441] Away from the center line

number

number

number

number

number

number

number

[0442] s c Within a distance (of parameter s) of

number

number

number

[0443] Defining a Projection Operator

number

number

number

[0444] Parameter s c is a free parameter of the flight policy. As mentioned above, the parameter s c may be chosen depending on the distance of a location away from the current centerline from an obstacle, such as a membrane (e.g., a cell membrane). An obstacle (e.g., membrane) avoidance technique may thereby be implemented during training. Because the above derivation is based on a Taylor series expansion, convergence to the centerline is achieved through iterative application of the above operation.

[0445] Directional prediction using Monte Carlo dropout The techniques disclosed in this section can be used, for example, to perform directional prediction to obtain an initial steering prediction: a number N (e.g., N=256) of equidistant or nearly equidistant orientations on the surface of a unit sphere.

number

number

number

number

[0446] This quantity is similar to the coefficient of variation.

[0447] Using the cosine similarity for nearby angles within 30 degrees, the weighted average of the uncertainty is calculated, e.g., weight

number

number

number

[0448] The weighted average provides increased stability.

[0449] The orientation with the smallest average uncertainty may be used as the first orientation candidate.

[0450] A second candidate for the spine head may be chosen that is at least 110° from the first candidate and has the smallest average uncertainty among the remaining orientations.

[0451] Local Realignment As previously mentioned, local realignment may be performed in the method and / or by imaging system 20. Local realignment may be performed by image alignment component 23, or imaging system 20 may be operative to perform a second local image realignment component in addition to image alignment component 23.

[0452] The local realignment technique will be described at least in part in connection with 3D EM data. It should be understood that the technique may also be used in connection with other 3D image data or 2D image data sets.

[0453] The local realignment may include the following steps performed to locally realign slices of image data along the slice direction: determining slice-to-slice shift vectors for multiple slices along the slice direction; identifying valid slice-to-slice shift vectors between the slices; and generating a locally realigned subvolume using the valid slice-to-slice shift vectors.

[0454] The locally realigned subvolumes can be fed to an AI model input, e.g., the input layer of a CNN or RNN, or to a filter or other preprocessing that preprocesses the realigned subvolumes before they are processed by the CNN or RNN.

[0455] Determining the slice-to-slice shift vectors may include determining slice autocorrelations and cross-correlations.

[0456] Determining the slice-to-slice shift vector and its validity may include determining the mean and standard deviation of the correlation weighted shift vector based on a first threshold criterion.

[0457] Identifying a valid slice-to-slice shift vector may include verifying that the determined autocorrelation meets a first criterion (such as having a peak at a desired shift, e.g., zero shift).

[0458] Identifying a valid slice-to-slice shift vector may include verifying that the determined autocorrelation peak has a standard deviation that meets a second criterion (e.g., having a standard deviation that meets a second threshold criterion that is less than or equal to a certain number of pixels).

[0459] Identifying a valid slice-to-slice shift vector may include verifying that the combination of two consecutive autocorrelation and / or cross-correlation peaks meets a third criterion (e.g., optionally, meets a third threshold criterion that may depend on surrounding autocorrelations and / or cross-correlations).

[0460] Identifying a valid slice-to-slice shift vector may include verifying that the determined cross-correlation satisfies a first cross-correlation criterion that depends on the autocorrelation for the two slices involved (such as the referenced autocorrelation criterion being valid for both slices).

[0461] Identifying a valid slice-to-slice shift vector may include verifying that the determined cross-correlation normalized by the expected cross-correlation value for the strength of the perfect correlation satisfies a second cross-correlation criterion (such as the peak of the cross-correlation normalized by the expected cross-correlation value for the strength of the perfect correlation meeting a further threshold criterion, e.g., greater than or equal to a fourth threshold).

[0462] Identifying valid slice-to-slice shift vectors may include verifying that the standard deviation of the shift vectors meets a third cross-correlation criterion (such as the standard deviation being less than or equal to a fifth threshold).

[0463] Using a valid shift vector may include identifying one or more slices to be realigned even if the slice-to-slice shift vector is not considered valid, and in this case calculating the shift vector from valid shift vectors associated with the neighbors of the one or more slices.

[0464] Using the valid shift vectors may include generating locally realigned trilinearly interpolated sub-volumes.

[0465] Generating the locally realigned trilinearly interpolated sub-volumes may include performing a floor or ceiling operation.

[0466] Generating the locally realigned trilinearly interpolated sub-volumes may include performing the trilinear interpolation as a series of linear interpolations along the major axes.

[0467] The series of linear interpolations may include four linear interpolations along a first dimension, two linear interpolations along a second dimension, and one linear interpolation along a third dimension.

[0468] The locally realigned sub-volumes can be processed to trace elongated objects such as 3D blood vessels or neurites.

[0469] To efficiently reconstruct neurites across discontinuities in globally aligned 3D image data, the methods and systems disclosed herein may operate by receiving a locally realigned 3D subvolume, followed by a neurite reconstruction method, similar to the technique of PHLi et al., "Automated Reconstruction of a Serial-Section EM Drosophila Brain with Flood-Filling Networks and Local Realignment," preprint, https: / / doi.org / 10.1101 / 605634, and applying a local alignment method to the possibly imperfect global alignment. The reconstruction within the subvolume then needs to be dealigned, i.e., projected back to the global reference frame (as described in the cited reference by PHLi et al.).

[0470] The following description focuses on local realignment of 2D slices across slice axes, ignoring problems arising from in-plane stitching of 2D mosaics with overlapping fields of view. In a tradeoff between accuracy and efficiency, slice-to-slice shift vectors for local realignment are pre-computed on a grid for the entire data set and bilinearly interpolated to yield approximate voxel-by-voxel local alignment, where the size of the grid for the pre-computed shift vectors is a hyperparameter of the local realignment method.

[0471] To pre-compute slice-to-slice shift vectors for local realignment, e.g., an 8x8 nm pixel set with N=512 and pixel values ​​in the range [0, 255] is used. 2 Slice of N x N grayscale EM data at a resolution of z i The 2D image data from the slice (z i , z i ), (z i , z i+1 ), and (z i , z i+2 ) is the location of the global maximum of the auto / cross correlation of the pixel-level shift vector

number

number

number

number

number

[0472] A number of criteria can be used to filter irregularly shifted vectors. For example, realignment is effective for slice z. i It can be implemented such that for the autocorrelation of , the following conditions must apply: (1) The autocorrelation must peak at zero shift, i.e.,

number

[0473] The realignment is performed by the slice cross-correlation (z i , z i+1), the following must apply: (1) the autocorrelation condition must be true for both slices, (2) the cross-correlation of the peaks normalized to their respective maxima must be greater than or equal to 0.18, and (3) the shift vector standard deviation must be less than or equal to 10 pixels.

[0474]

number

number

[0475] otherwise invalid shift vector

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0476] the shift vector as determined above

number

number

number

number

number

number

number

number

number

number

[0477] To perform trilinear interpolation taking the shift vector into account, the method and system may utilize a description of trilinear interpolation as a series of linear interpolations along the following major axes: (1) four linear interpolations along the first dimension, (2) two linear interpolations of the result from (1) along the second dimension, and (3) one linear interpolation of the result from (2) along the third dimension. Specifically, the interpolation of a single voxel may be calculated by first linearly interpolating along the x and y axes independently for floor and ceiling z values ​​with corresponding shifts along x and y applied, respectively, and then linearly interpolating along the z axis.

[0478] Since predictions from a CNN in a locally realigned subvolume are only valid within this subvolume, the predictions must be de-aligned to be relative to the global coordinate system of the 3D EM dataset. The methods and systems disclosed herein first calculate the predicted Bishop curvature as before.

number

number

number

number

[0479] Alignment techniques can be used in conjunction with reassembling 2D images into a coherent 3D image volume with a common global reference frame. This is referred to as alignment. In the event of an error in any of the preparation, slicing, imaging, or reassembly steps, the resulting 3D image dataset generally will not be coherent in all places. The realignment techniques disclosed herein are a technical advantage over many of the imaging techniques disclosed herein. By way of illustration, to image nervous tissue using 3D EM, the nervous tissue is physically decomposed (sliced) along one axis, imaged in 2D using an electron microscope, possibly as a mosaic with overlapping fields of view, and virtually reassembled to ideally yield a globally coherent 3D image dataset. Upon this reassembly and / or after a coarse global reassembly, the realignment techniques disclosed herein can be applied. The alignment techniques disclosed herein provide excellent coherence while avoiding excessive computational runtime. The alignment techniques disclosed herein are advantageous for many of the applications contemplated herein.

[0480] Although embodiments have been described in detail, various modifications and changes may be implemented in other embodiments.

[0481] By way of illustration, the techniques disclosed herein may be applied to tracing a wide variety of elongated objects.

[0482] The techniques disclosed herein are not limited to tracing elongated objects and / or tracing in 3D, but may be applied to a wide variety of other use cases, such as tracing an object through a series of image frames (e.g., in a video), tracing an object in a space-time coordinate system, and / or performing image processing for a wide variety of control actions.

Claims

1. A method for tracing an elongated object (41) in image data (40, 170), said image data (40, 170) being biological or medical image data (40, 170), said method comprising the following steps performed by an image processing system (30): determining positions of a series of points (60) along the elongated object (41) in three-dimensional (3D) space to trace the elongated object (41) in 3D in the image data (40, 170), wherein the step of determining the positions of the series of points (60) comprises an iterative process, the iterative process comprising: using an artificial intelligence (AI) model (110, 130) to infer a manipulation prediction that depends on a direction in which the elongated object (41), optionally a centerline (50) of the elongated object, extends from a previously determined point along the elongated object (41), the AI ​​model (110, 130) comprising an input layer operative to receive AI model input based on pixel or voxel values ​​of the image data (40, 170), and an output layer operative to provide, at each iteration of the iterative process, an AI model (110, 130) output that is or is indicative of the manipulation prediction; calculating a next point (62-65) in the series of points (60) using at least the operational prediction, the next point following the previously determined point (61-64) in the series of points (60); determining the positions of the series of points (60) by iteratively repeating the steps of estimating and calculating until a termination criterion is met; A method comprising:

2. The AI ​​model is a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer; or It has a recurrent neural network (RNN), The AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes (91, 93, 172-174) of a tissue volume represented by the image data (40, 170), and the AI ​​model inputs include: comprising pixel or voxel values ​​representing said sub-volume of said image data (40, 170); or obtained by multivariate interpolation of pixel or voxel values ​​representing said sub-volume of said image data (40, 170); The method of claim 1, wherein the orientation of the sub-volumes (91, 93, 172-174) depends on the manipulation prediction determined in a previous iteration of the iterative process.

3. The method of claim 2 , wherein the operation prediction depends on a second derivative of a centerline (50) of the elongated object (41).

4. The AI ​​model is a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer; or A recurrent neural network (RNN), the AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes (91, 93, 172-174) of a tissue volume represented by the image data (40, 170); the manipulation prediction is a curvature value relative to a given reference system associated with the sub-volume, such that the manipulation prediction results in position offsets that iteratively determine a series of points located on a centerline (50) of the elongated object, 2. The method of claim 1, wherein in each iteration following a previous iteration, each sub-volume to be processed is determined to have a rotational orientation determined by the extension direction of the elongated object as determined from the manipulation prediction obtained in the previous iteration.

5. The following, i.e., determining the next point in the series of points (60) includes integrating the operation prediction; determining the next point in the series of points (60) includes integrating the maneuver prediction, and integrating the maneuver prediction includes integrating curvature in a Bishop frame of reference; the operation prediction is related to a second derivative of the centerline (50) of the elongated object (41).

6. The AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes (91, 93, 172-174) of a tissue volume represented by the image data (40, 170), the sub-volumes (91, 93, 172-174) comprising: The method of claim 1 or claim 5, which relies on the position and / or orientation and / or manipulation predictions determined in one or several previous iterations of the iterative process.

7. The method of claim 6, wherein the orientation of the sub-volumes (91, 93, 172-174) depends on the previous manipulation prediction.

8. The following, i.e., the centers of said sub-volumes (91, 93, 172-174) are shifted relative to said previously determined points by an offset; the centers of the sub-volumes (91, 93, 172-174) are shifted relative to the previously determined points by an offset, the offset depending on the manipulation prediction determined in the previous iteration.

9. 9. The method of any one of claims 6 to 8, further comprising multivariate interpolation of pixel or voxel values ​​of the image data (40, 170) to determine the AI ​​model inputs.

10. The multivariate interpolation a weighted average of the pixel or voxel values ​​that generate the AI ​​model input as interpolated pixel or voxel values ​​in a grid that is tilted and / or offset relative to the pixel or voxel grid of the image data (40, 170); or The method of claim 9 comprising a trilinear projection.

11. The method of claim 1 , further comprising initializing the iteration procedure, said initializing depending on whether an initial direction is available.

12. The method of claim 11 , wherein when the initial direction is not available, the method includes determining the initial direction to be used in an initial iteration of the iterative process.

13. Determining the initial direction comprises: determining uncertainty estimates for the predictions in various directions; selecting, from among the various directions, the direction and maneuver prediction for which the uncertainty estimate in the corresponding prediction is smallest; 13. The method of claim 12, comprising:

14. The method of claim 13 , wherein the uncertainty estimate is determined using Monte Carlo dropout.

15. The following, i.e., The AI ​​model (110, 130) comprises a convolutional neural network (CNN) or a recurrent neural network (RNN); The AI ​​model (110, 130) comprises the input layer, a plurality of convolutional layers, a dropout layer, and the output layer; The method of any one of claims 1 to 14, wherein the AI ​​model (110, 130) comprises one or more of a pooling layer, an attention module, a self-attention module, an atlas convolution, a normalization layer, a batch normalization layer, a transformer, and a recurrent layer.

16. The following, i.e., the output layer is a linear layer; The plurality of convolutional layers comprises one or several strided convolutional layers; The AI ​​model (110, 130) is or comprises a fully convolutional network; The AI ​​model (110, 130) comprises one or several nonlinearities; The AI ​​model (110, 130) comprises an exponential linear unit (ELU); The AI ​​model (110, 130) comprises a rectified linear unit (ReLU); The AI ​​model (110, 130) comprises a scaled exponential linear unit (SELU); The method of any one of claims 1 to 15, wherein one, some, or all of the following are applied: the AI ​​model (110, 130) comprises a Gaussian Error Linear Unit (GELU).

17. 17. The method according to any one of claims 1 to 16, wherein the elongated object (41) is a neurite.

18. 17. The method according to any one of claims 1 to 16, wherein the elongated object (41) is a blood vessel.

19. 19. The method according to any one of claims 1 to 18, wherein the elongated object (41) has a surface that extends around the centerline (50) in a locally cylindrical manner.

20. The method of any one of claims 1 to 19, further comprising training the AI ​​model (110, 130).

21. The following, i.e., The AI ​​model (110, 130) is trained using supervised learning; and training the AI ​​model (110, 130) includes gradient-based updating of AI model (110, 130) parameters.

22. 22. The method of claim 20 or 21, wherein training the AI ​​model (110, 130) comprises training the AI ​​model (110, 130) to follow a flight policy that causes the iterative process to converge to the elongated object (41) when starting from or reaching a location in the image data (40, 170) away from the elongated object (41).

23. 23. The method of claim 22, wherein a convergence distance of the flight policy determines how quickly the iterative process converges back to the elongated object (41).

24. The following, i.e., the convergence distance is a dynamic parameter that is varied during training and / or inference of the AI ​​model (110, 130); and the convergence distance is set as a function of distance from physiological boundaries during training of the AI ​​model (110, 130).

25. 25. The method of claim 24, wherein the convergence distance is set as a function of distance from a physiological boundary during training of an AI model (110, 130), and the convergence distance is a monotonically increasing function of the distance from the physiological boundary.

26. receiving medical image data (40, 170) comprising a set of 3D or 2D images of tissue comprising neural tissue; 26. Performing the method of any one of claims 1 to 25, tracing an elongated object (41) in the medical image data (40, 170); The following, i.e., a visual representation of said series of points (60) or said centerline (50); a machine-readable representation of said series of points (60) or said centerline (50); a control signal dependent on the series of points (60) located along the centerline (50); a machine learning (ML) model (110, 130) having an architecture that depends on the set of points (60) located along the centerline (50); A medical image processing method, comprising:

27. A connectomics method comprising performing the method for tracing the elongated object (41) according to any one of claims 1 to 25.

28. 28. The connectomics method of claim 27, further comprising executing a synaptic interface classifier and determining a connectome using at least the elongated object tracing method and an output of the synaptic interface classifier.

29. 1. A method of analyzing a biopsy sample, comprising: acquiring image data (40, 170) of said biopsy sample; Processing the image data (40, 170) using the method of any one of claims 1 to 25; outputting the results of processing the image data (40, 170) via an interface; A method comprising:

30. 1. A method for generating signal processing logic for controlling a physical asset, the method comprising: performing a method for tracing an elongate object according to any one of claims 1 to 25; using, by at least one computing system, output of the elongated object tracing method; Selecting or generating a machine learning (ML) model (110, 130) architecture; training an ML model (110, 130) having the ML model (110, 130) architecture; deploying the trained ML model (110, 130) to a control device for execution as the signal processing logic; controlling an image acquisition system (20) to acquire said image data (40, 170); A method comprising:

31. An image processing system (30) for tracing an elongated object (41) in image data (40, 170), said image processing system (30) comprising: an interface operative to receive the image data (40, 170), the image data (40, 170) being biological or medical image data (40, 170); at least one processing device; wherein the at least one processing device Determining positions of a series of points (60) along the elongated object (41) of the elongated object (41) in three-dimensional (3D) space and operating to trace the elongated object (41) in 3D in the image data (40, 170), wherein determining the positions of the series of points (60) comprises an iterative process, the iterative process comprising: using an artificial intelligence (AI) model (110, 130) to infer a manipulation prediction that depends on a direction in which the elongated object (41), optionally a centerline (50) of the elongated object (41), extends from a previously determined point along the elongated object (41), the AI ​​model (110, 130) comprising an input layer operative to receive AI model input based on pixel or voxel values ​​of the image data (40, 170), and an output layer operative to provide, at each iteration of the iterative process, an AI model (110, 130) output that is or is indicative of the manipulation prediction; calculating a next point in the series of points (60) using at least the operation prediction, the next point following the previously determined point in the series of points (60); determining the positions of the series of points (60) by iteratively repeating the steps of estimating and calculating until a termination criterion is met; An image processing system (30) comprising:

32. The image processing system operates as follows: The AI ​​model is a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer; or It has a recurrent neural network (RNN), the AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes (91, 93, 172-174) of a tissue volume represented by the image data (40, 170), and the AI ​​model inputs comprise: comprising pixel or voxel values ​​representing said sub-volume of said image data (40, 170); or obtained by multivariate interpolation of pixel or voxel values ​​representing said sub-volume of said image data (40, 170); 31. The image processing system (30) of claim 30, wherein the image processing system operates such that the orientation of the sub-volumes (91, 93, 172-174) depends on the manipulation prediction determined in a previous iteration of the iterative process.

33. 33. The image processing system (30) of claim 32, wherein the image processing system is operative such that the manipulation prediction depends on a second derivative of a centerline (50) of the elongate object (41).

34. The image processing system operates as follows: The AI ​​model is a convolutional neural network (CNN) comprising the input layer, a plurality of convolutional layers, a dropout layer, and the output layer; or A recurrent neural network (RNN), the AI ​​model inputs input to the input layer in each iteration of the iterative process represent sub-volumes (91, 93, 172-174) of a tissue volume represented by the image data (40, 170); curvature values ​​relative to a given reference system associated with the sub-volume, such that the manipulation prediction results in position offsets that iteratively determine a series of points located on a centerline (50) of the elongated object, 32. The image processing system (30) of claim 31, wherein the image processing system operates such that in each iteration following a previous iteration, each sub-volume to be processed is determined to have a rotational orientation determined by the direction of extension of the elongated object as determined from the manipulation prediction obtained in the previous iteration.

Images