Method and device for reconstructing biological tubular structure in electron microscope microscopic image and medium
By using the segmentation network of self-attention mechanism and the tracking network of skeleton point prediction in bioelectromics microscopy, the problem of insufficient segmentation and tracking in three-dimensional reconstruction of biotubular structures is solved, and higher reconstruction accuracy is achieved and stability.
Patent Information
- Application Number
- CN202411102017.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The prior art has problems in the three-dimensional reconstruction of biological tubular structures in bioelectron microscopy images, and the tracking results are prone to fracture and incorrect connection, especially in areas with weak signals and poor image quality.
Image segmentation network with self-attention mechanism is used for image segmentation, combined with a biotubular structure tracking network with skeleton point prediction, the long-distance dependency and connection direction of the skeleton point set are captured through the self-attention mechanism to construct a tree structure Reconstruct.
The segmentation accuracy and tracking accuracy of the biological tubular structure are improved, missing points and wrong connections are reduced, and more accurate three-dimensional reconstruction results are obtained.
Smart Images

Figure CN119048673A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to the technical field of electron microscope microscopic image processing, and in particular to a method for reconstructing biological tubular structures in large-scale electron microscope microscopic images. Background Art
[0002] The trachea and blood vessels play a vital role in the respiratory system, and the study of their structure and function is crucial for a deeper understanding of respiratory diseases and the development of new treatments. The development of three-dimensional reconstruction technology has provided a deeper understanding of typical branched tubular structures such as the trachea and blood vessels. This technology aims to reconstruct the tubular circuits of the trachea or blood vessels in biological electron microscope microscopic images, providing an important tool for biomedical analysis. It not only helps scientists to have a deeper understanding of the natural changes of these structures, but also provides powerful assistance to doctors in pathological diagnosis. By reconstructing the trachea and blood vessels in three dimensions, researchers can observe structural abnormalities more clearly, which is crucial for the study of the pathogenesis of vascular-related diseases such as asthma, tracheitis, and atherosclerosis. The study of the trachea and blood vessels not only helps people to have a deeper understanding of the complexity of the respiratory system, but also provides an important scientific basis for the prevention and treatment of related diseases.
[0003] With the development of high-speed and high-resolution microscopic imaging technology, ultra-high-resolution scanning of the whole brain has become possible. In 2018, the Howard Hughes Medical Institute published the first complete nanoscale electron microscope scan image of the fruit fly brain (Full Adult Fly Brain, referred to as FAFB dataset) in the journal Cell. Its physical resolution reached 4nm×4nm×40nm. The whole brain image includes 7062 slice images and the total data volume reaches 106TB (ZHENG Z, et al., "A complete electron microscopy volume of the brain of adult Drosophila melanogaster", in Cell, 2018). At nanoscale resolution, the details of microscopic structures such as trachea, neuronal bundles, and cell bodies can be observed more clearly. This high-resolution imaging capability provides rich data resources for fine tracheal three-dimensional reconstruction on large-scale datasets and in-depth biological research.
[0004] In recent years, researchers have invested a lot of research work on the reconstruction of biological tubular tree structures (such as trachea, blood vessels, neurons, etc.), which can be divided into segmentation and tracking steps. In segmentation, most existing methods use 3D CNN series methods to segment three-dimensional medical image data. This type of method uses a three-dimensional convolution kernel to process three-dimensional images, effectively utilizes the z-axis spatial information of biomedical images, and deeply mines the characteristics of three-dimensional space. Limited by the size of the three-dimensional perception field, this type of method has weak processing capabilities for global context information. In recent years, some methods using the Transformer model have been applied to medical organ segmentation tasks. With its powerful self-attention mechanism, the Transformer model not only focuses on local details when processing sequence data, but also takes into account global context information, and can more effectively capture long-distance dependencies, which is crucial for continuity and accuracy in tubular structure segmentation tasks.
[0005] Since image segmentation only generates pixel-by-pixel or voxel-by-voxel category labels to form segmentation masks, it cannot directly express the three-dimensional structural characteristics of the target, such as connectivity, tubular bifurcation, axial length, etc. Therefore, it is necessary to track biological tubular structures, connect tissue fragments in the segmentation results, and reconstruct tubular structure pathways for subsequent biological analysis. Existing traditional tracking methods often rely on manually designed features, selected seed points, track voxels belonging to specific biological tubular structures, and reconstruct the morphology of biological tubular structures. Such methods are highly dependent on the quality of image segmentation, and often have poor effects in areas with poor image quality, which easily leads to a large number of breaks and incorrect connections in the tracking results, and subsequent artificial introduction of priors is required for correction. In recent years, some methods have used the power of neural networks to improve tracking accuracy. Most of these methods divide tracking into two steps: point set prediction and point set connection. These methods use neural networks to extract features from biological electron microscope microscopic images, learn and identify multi-scale and multi-level feature representations in images, improve tracking performance in areas with poor image quality and complex structures, and show better robustness. However, for areas with weak signals, the performance of many methods is affected. For example, when PointNeuron performs point set prediction, it samples point clouds from the image and loses image information in areas with weaker signals. In contrast, NRTR uses the DETR method based on target detection, see the literature "Carion N, et al., End-to-end object detection with Transformers, in ECCV, 2020", to predict point sets from optical microscope images, thereby retaining more image details. However, the basic version of DETR used by NRTR is unstable during training and converges slowly. In addition, when processing some dense electron microscope data, NRTR frequently generates connection errors in the point set connection stage, affecting the accuracy of the final tubular structure reconstruction.
[0006] In view of this, the present invention is proposed. Summary of the invention
[0007] The purpose of the present invention is to provide a method, device and medium for reconstructing biological tubular structures in electron microscope microscopic images, which can obtain more accurate three-dimensional tubular structure reconstruction results from biological electron microscope microscopic images, thereby solving the above-mentioned technical problems existing in the prior art.
[0008] The objective of the present invention is achieved through the following technical solutions:
[0009] A method for reconstructing biological tubular structures in electron microscope microscopic images, characterized by comprising:
[0010] Step 1, the original electron microscope microscopic image is divided into blocks to obtain block images, and a biological tubular structure segmentation network with a self-attention mechanism is used to capture the long-range dependency relationship of the biological tubular structure in the original electron microscope microscopic image through the self-attention mechanism, and each block image is segmented into biological tubular structures block by block to obtain a biological tubular structure probability segmentation image based on the long-range dependency relationship;
[0011] Step 2, pre-processing the biological tubular structure probability segmentation image obtained in step 1 to obtain a tracking image block, predicting the obtained tracking image block through a biological tubular structure tracking network with skeleton point prediction to obtain a skeleton point set of the biological tubular structure, connecting each skeleton point of the skeleton point set of the biological tubular structure according to the connection direction to construct a tree-like biological tubular structure, and obtaining a reconstruction result of the biological tubular structure.
[0012] A processing device, comprising:
[0013] at least one memory for storing one or more programs;
[0014] At least one processor can execute one or more programs stored in the memory. When the one or more programs are executed by the processor, the processor can implement the method described in the present invention.
[0015] A readable storage medium stores a computer program, which can implement the method described in the present invention when the computer program is executed by a processor.
[0016] Compared with the prior art, the method, device and medium for reconstructing biological tubular structures in electron microscope microscopic images provided by the present invention have the following beneficial effects:
[0017] By using a biological tubular structure segmentation network with a self-attention mechanism, the original electron microscope microscopic image is accurately segmented into biological tubular structures to obtain a biological tubular structure probabilistic segmentation image. The biological tubular structure probabilistic segmentation image is pre-processed to obtain a tracking image block, which is predicted by a biological tubular structure tracking network with skeleton point prediction to obtain a biological tubular skeleton point set. The skeleton points in the biological tubular structure skeleton point set are connected according to the connection direction to construct a tree-like biological tubular structure, and the biological tubular structure reconstruction result is obtained. Due to the biological tubular structure segmentation network with a self-attention mechanism, the long-distance dependency information of the biological tubular structure in the original electron microscope microscopic image is effectively processed by the self-attention mechanism, which improves the accuracy of segmentation and reduces missed points; due to the use of a biological tubular structure tracking network with skeleton point prediction, the features of interest can be selected from the output features of the Transformer encoder to generate skeleton proposal points as the initial query of the Transformer decoder, optimize the network structure, stabilize the network training, and improve the network performance; and the connection direction of each skeleton point is additionally predicted in the skeleton point set prediction, and the connection direction is used to guide the process of connecting each skeleton point in the skeleton point set, so as to obtain a more accurate reconstruction result. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0019] Figure 1 A flow chart of a method for reconstructing biological tubular structures in electron microscope images provided in an embodiment of the present invention.
[0020] Figure 2 A flowchart of biological tubular structure tracking in the reconstruction method provided in an embodiment of the present invention.
[0021] Figure 3 A comparative schematic diagram of the reconstruction results of biological tubular structures of the reconstruction method provided in an embodiment of the present invention; wherein, (a) is a schematic diagram of the biological tubular structure of the segmented image block; (b) is a schematic diagram of the reconstruction results of the biological tubular structure by the manual labeling method; (c) is a schematic diagram of the reconstruction results of the biological tubular structure by the NGPST method; (d) is a schematic diagram of the reconstruction results of the biological tubular structure by the NRTR method; and (e) is a schematic diagram of the reconstruction results of the biological tubular structure by the method of the present invention.
[0022] Figure 4 A schematic diagram of the whole brain reconstruction result of the reconstruction method provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0023] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the specific content of the present invention; it is obvious that the described embodiments are only part of the embodiments of the present invention, not all of the embodiments, which does not constitute a limitation of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of the present invention.
[0024] First, the terms that may be used in this article are explained as follows:
[0025] The term “and / or” means that either or both of them can be realized at the same time. For example, X and / or Y means both “X” or “Y” and “X and Y”.
[0026] The terms "include", "comprises", "contains", "has" or other descriptions with similar semantics should be interpreted as non-exclusive inclusion. For example, including certain technical feature elements (such as raw materials, components, ingredients, carriers, dosage forms, materials, dimensions, parts, components, mechanisms, devices, steps, procedures, methods, reaction conditions, processing conditions, parameters, algorithms, signals, data, products or products, etc.) should be interpreted as including not only certain technical feature elements explicitly listed, but also other technical feature elements known in the art that are not explicitly listed.
[0027] The term "consisting of..." means excluding any technical feature elements not explicitly listed. If this term is used in a claim, it will make the claim closed, so that it does not contain technical feature elements other than the technical feature elements explicitly listed, except for the conventional impurities related to them. If this term only appears in a clause of a claim, it only limits the elements explicitly listed in the clause, and the elements recorded in other clauses are not excluded from the overall claim.
[0028] Unless otherwise specified or limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example: it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this article can be understood according to specific circumstances.
[0029] The orientation or position relationship indicated by terms such as "center", "longitudinal", "lateral", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. are based on the orientation or position relationship shown in the drawings and are only for the convenience and simplification of description, and do not explicitly or implicitly indicate that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation of this document.
[0030] The scheme provided by the present invention is described in detail below. The contents not described in detail in the embodiments of the present invention belong to the prior art known to professionals in the field. If the specific conditions are not specified in the embodiments of the present invention, they are carried out according to the conventional conditions in the field or the conditions recommended by the manufacturer. If the manufacturers are not specified in the reagents or instruments used in the embodiments of the present invention, they are all conventional products that can be purchased commercially.
[0031] like Figure 1 As shown, an embodiment of the present invention provides a method for reconstructing a biological tubular structure in an electron microscope microscopic image, comprising:
[0032] Step 1, the original electron microscope microscopic image is divided into blocks to obtain block images, and a biological tubular structure segmentation network with a self-attention mechanism is used to capture the long-range dependency relationship of the biological tubular structure in the original electron microscope microscopic image through the self-attention mechanism, and each block image is segmented into biological tubular structures block by block to obtain a biological tubular structure probability segmentation image based on the long-range dependency relationship;
[0033] Step 2, pre-processing the biological tubular structure probability segmentation image obtained in step 1 to obtain a tracking image block, predicting the obtained tracking image block through a biological tubular structure tracking network with skeleton point prediction to obtain a skeleton point set of the biological tubular structure, connecting each skeleton point of the skeleton point set of the biological tubular structure according to the connection direction to construct a tree-like biological tubular structure, and obtaining a reconstruction result of the biological tubular structure.
[0034] It can be known that the reconstruction result of the above-mentioned biological tubular structure is a kind of reconstruction data of a biological tubular structure, which can be saved in a standard text SWC file. In the field of modern biology, this biological tubular structure is often defined as a tree-like skeleton. The tree-like skeleton is composed of many skeleton points. Each skeleton point contains its three-dimensional spatial position, radius size and connection information with other nodes. The tubular structure pathway is constructed through the connection information between skeleton points.
[0035] Preferably, in step 2 of the above reconstruction method, the biological tubular structure probability segmentation image obtained in step 1 is pre-processed in the following manner to obtain a tracking image block, the obtained tracking image block is predicted by a biological tubular structure tracking network with skeleton point prediction to obtain a skeleton point set of the biological tubular structure, and the skeleton points of the skeleton point set of the biological tubular structure are connected according to the connection direction to construct a tree-like biological tubular structure, and the reconstruction result of the biological tubular structure is obtained, including:
[0036] Step 21, using the segmentation probability of the biological tubular structure region in the biological tubular structure probability segmentation image obtained in step 1, obtain a block-by-block three-dimensional segmentation probability image, combine multiple blocks of three-dimensional segmentation probability images into an overall three-dimensional segmentation probability image, downsample the overall three-dimensional segmentation probability image obtained and cut it into at least one fixed-size tracking image block,
[0037] Step 22, inputting each tracking image block obtained in step 21 into a biological tubular structure tracking network with skeleton point prediction to perform skeleton point prediction, and obtaining the three-dimensional spatial position and connection direction of the skeleton point of the biological tubular structure in the tracking image block, wherein the three-dimensional spatial position is composed of coordinates and radius;
[0038] Step 23, using the obtained three-dimensional spatial position and connection direction of the skeleton points of the biological tubular structure to connect each skeleton point, complete the three-dimensional construction of the tree structure of the biological tubular structure of each tracking image block, and finally splice the tree structure reconstruction results of the biological tubular structure of multiple tracking image blocks to obtain the biological tubular structure reconstruction result corresponding to the overall three-dimensional segmentation probability image.
[0039] Preferably, in the above reconstruction method, the biological tubular structure segmentation network with a self-attention mechanism adopts a Swin UNETR network with a self-attention mechanism;
[0040] The physical resolution of the input electron microscope image of the Swin UNETR network with self-attention mechanism is 16nm×16nm×40nm, and the voxel size is 128×128×16;
[0041] The output of the Swin UNETR network with self-attention mechanism is a probability image of whether each voxel belongs to a biological tubular structure, that is, a biological tubular structure probability segmentation image;
[0042] The biological tubular structure tracking network with skeleton point prediction includes: an image feature extraction network, a Transformer-based skeleton point set prediction network and a skeleton point set connection module; wherein,
[0043] The image feature extraction network can take the tracking image block obtained by pre-processing the biological tubular structure probability segmentation image obtained in step 1 as input, and extract low-resolution image features as output;
[0044] The Transformer-based skeleton point set prediction network is communicatively connected to the output end of the image feature extraction network, and can use the low-resolution image features output by the image feature extraction network as an input tag sequence, calculate the association between each image block tag in the input tag sequence, generate a skeleton proposal point feature sequence of a biological tubular structure, and predict the biological tubular proposal point feature sequence to obtain the classification probability, three-dimensional spatial position and connection direction of each skeleton point in the skeleton point sequence of the biological tubular structure, and the classification probability, three-dimensional spatial position and connection direction of each skeleton point constitute a skeleton point set, and the three-dimensional spatial position is composed of a three-dimensional spatial coordinate and a radius;
[0045] The skeleton point set connection module is communicatively connected to the output end of the Transformer-based skeleton point set prediction network, and can take the skeleton point set predicted by the Transformer-based skeleton point set prediction network as input, and based on the predicted connection direction of each skeleton point, connect each skeleton point in the skeleton point set to generate a final tree structure of the biological tubular structure.
[0046] Preferably, in the above reconstruction method, the image feature extraction network is composed of a three-dimensional residual network and a feature pyramid network connected in sequence, wherein the three-dimensional residual network takes the tracking image block obtained by pre-processing the biological tubular structure probability segmentation image obtained in step 1 as input, and can perform three-dimensional residual processing and then output to the feature pyramid network to extract low-resolution image features to obtain low-resolution image features;
[0047] The Transformer-based skeleton point set prediction network consists of a Transformer encoder, a skeleton proposal point set prediction submodule, a Transformer decoder and a skeleton point set prediction submodule; wherein,
[0048] The Transformer encoder is connected to the image feature extraction network for communication, and can take the position code of each pixel position of the low-resolution image feature output by the image feature extraction network and the pixel feature sequence composed of all pixel features of the low-resolution image feature as input, and map the pixel feature sequence and the position code of each pixel position to a feature sequence containing position information and content information of all image features through a self-attention mechanism, and output the feature sequence;
[0049] The skeleton proposal point prediction submodule is communicatively connected to the output end of the Transformer encoder, and can use the feature sequence output by the Transformer encoder as input to respectively predict the classification probability, three-dimensional spatial position and connection direction of the biological tubular structure skeleton proposal point, wherein the three-dimensional spatial position is composed of a three-dimensional spatial coordinate and a radius;
[0050] The Transformer decoder is respectively connected to the Transformer encoder and the skeleton proposal point set prediction submodule for communication, and can use the first N biological tubular structure skeleton points with the largest probability in the classification probability output by the skeleton proposal point set prediction submodule as skeleton proposal points, select N features corresponding to the N skeleton proposal points from the feature sequence to reassemble into a biological tubular structure point prediction feature proposal sequence as an initial object query matrix, and use the position encoding of the N skeleton proposal points and the feature sequence output by the Transformer encoder to input into each layer of the Transformer decoder as a key and value, use a cross attention mechanism to capture the dependency between the target biological tubular structure point and all positions in the image, and output the target biological tubular structure point features with the same shape as the biological tubular structure point prediction feature proposal sequence;
[0051] The skeleton point set prediction submodule is communicatively connected to the Transformer decoder, and can use the target biological tubular structure point features output by the Transformer decoder as input to obtain the classification probability, three-dimensional spatial position and angle of each skeleton point, wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius, and the obtained skeleton points constitute a skeleton point set.
[0052] Preferably, in the above reconstruction method, in the Transformer-based skeleton point set prediction network, the Transformer encoder is composed of a plurality of encoder layers connected in sequence, and each encoder layer is composed of a multi-head self-attention module and a multi-layer perception module;
[0053] The skeleton proposal point set prediction submodule is composed of six multi-layer perceptrons, wherein the first two multi-layer perceptrons are used to predict the classification probability and three-dimensional spatial position of the skeleton points of the biological tubular structure, and the last four multi-layer perceptrons are used to predict the horizontal angle and vertical angle corresponding to the connection direction;
[0054] The Transformer decoder consists of multiple decoder layers connected in sequence, and each decoder layer consists of a multi-head self-attention submodule, a multi-head cross-attention submodule and a multi-layer perception submodule.
[0055] Preferably, in the above-mentioned reconstruction method, the skeleton point set prediction submodule is composed of six multi-layer perceptrons, wherein the first two multi-layer perceptrons are used to predict the classification probability and three-dimensional spatial position of the biological tubular structure points, respectively, and the last four multi-layer perceptrons are used to predict the horizontal angle and vertical angle corresponding to the connection direction.
[0056] Preferably, in the above reconstruction method, the total loss function of the biological tubular structure tracking network with skeleton point prediction is:
[0057]
[0058] The meaning of each parameter is: Represents all predicted skeleton proposal points The matching point with its true value The sum of the matching losses; Represents all predicted skeleton points The matching point with its true value The sum of the matching losses; The skeleton proposal point set prediction submodule of the skeleton point set prediction network based on Transformer of the biological tubular structure tracking network with skeleton point prediction uses the Top-K strategy to select the skeleton proposal point set consisting of N biological tubular structure points with the highest probability; Represents the skeleton proposal point set Optimal match with the true value skeleton point set Y; Represents a skeleton point set consisting of N skeleton points output by the Transformer-based skeleton point set prediction network; Represents the skeleton point set The optimal match with the true value skeleton point set Y; ω1 and ω2 are hyper parameters respectively;
[0059] Of the above total losses, The calculations are as follows:
[0060]
[0061] Among them, c i , L i , Ai represent the true value Y i The classification probability, three-dimensional spatial position and angle of , wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius; Respectively represent the prediction results The classification probability, three-dimensional spatial position and angle of ω, wherein the three-dimensional spatial position is composed of coordinates and radius; cls ,ω box ,ω iou ,ω dir are hyperparameters respectively; represents the classification loss, which is set to the cross entropy loss function; and Both represent position loss, is the L1 loss function, is the sphere intersection-union ratio loss function; is the direction loss, The calculation is as follows:
[0062]
[0063] in, Represents the true value direction of each skeleton point d i Corresponding horizontal angle classification, horizontal angle internal offset value, vertical angle classification and vertical angle internal offset; Respectively represent the prediction direction Horizontal angle classification, horizontal angle internal offset value, vertical angle classification and vertical angle internal offset; represents the angle classification loss, which is set to the cross entropy loss function; It represents the intra-class offset regression loss of angle, adopts L1 loss function, and only calculates the difference between the predicted value and the true value in the offset value of the angle target interval.
[0064] Preferably, in step 2 of the above reconstruction method, the skeleton point set of the biological tubular structure is connected according to the connection direction to construct a tree-like biological tubular structure to obtain a reconstructed image of the biological tubular structure, including:
[0065] The skeleton points whose classification probability is less than the predetermined threshold are deleted from the skeleton point set of the biological tubular structure, and the remaining skeleton points are used as the screened skeleton points. The predicted connection direction of each skeleton point is calculated based on the predicted angle of the screened skeleton points. A point-edge graph is established for all the remaining skeleton points. An edge relationship is established between two skeleton points whose distance between the two points is less than the distance threshold and whose angle difference between the two points and the predicted direction is less than the angle threshold. Then, the angle difference and the distance between the points are used as the edge weight e. i,j , e i,j for:
[0066] e i,j =ω3dist(P i , P j )+
[0067] ω4min{[1-cos(d i ,dir i,j )],[1-cos(d j ,dir j,i )]} (9);
[0068] Among them, dist(P i , Pj ) represents the Euclidean space distance between skeleton point i and skeleton point j; (P i , P j ) represents the position of skeleton point i and skeleton point ,; cos(d i ,dir i,j ) represents the predicted direction of skeleton point i and the cosine distance from skeleton point i to skeleton point, direction; (d i ,dir i,j ) represents the predicted direction of skeleton point i and the direction from skeleton point i to skeleton point ,; cos(d j ,dir j,i ) represents the predicted direction of skeleton point j and the cosine distance from skeleton point j to skeleton point i; (d j ,dir j,i ) represents the predicted direction of skeleton point j and the direction from skeleton point j to skeleton point i; ω3 and ω4 are both hyperparameters; min{} represents the minimum value;
[0069] The minimum spanning tree is used to remove the loops from the constructed point-edge graph, and the final tree structure of the biological tubular structure is obtained.
[0070] The embodiment of the present invention further provides a processing device, including:
[0071] at least one memory for storing one or more programs;
[0072] At least one processor can execute one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor can implement the above method.
[0073] The embodiment of the present invention further provides a readable storage medium storing a computer program, which can implement the above method when the computer program is executed by a processor.
[0074] In summary, the method of the embodiment of the present invention, by using a biological tubular structure segmentation network with a self-attention mechanism, accurately performs biological tubular structure segmentation on the original electron microscope image to obtain a biological tubular structure probabilistic segmentation image, and uses a biological tubular structure tracking network with skeleton point prediction to pre-process the biological tubular structure probabilistic segmentation image to obtain a tracking image block to obtain a skeleton point set of the biological tubular structure, and connects the skeleton point set of the biological tubular structure according to the connection direction to construct a tree-like biological tubular structure to obtain a reconstructed image of the biological tubular structure. The Swin UNETR network is used as a biological tubular structure segmentation network with a self-attention mechanism. The self-attention mechanism is used to effectively process the long-distance dependency information of the biological tubular structure in the original electron microscope microscopic image, thereby improving the accuracy of segmentation and reducing missed points. Due to the use of a biological tubular structure tracking network with skeleton point prediction, the features of interest can be selected from the output feature sequence output by the Transformer encoder to generate skeleton proposal points as the initial query of the Transformer decoder, thereby optimizing the network structure, stabilizing network training, and improving network performance. In addition, the connection direction of the skeleton point set is additionally predicted in the skeleton point set prediction, and the connection direction is used to guide the process of connecting each skeleton point in the skeleton point set, thereby obtaining a more accurate reconstruction structure. The method of the present invention is experimented on FAFB electron microscope data, and a small amount of data is used to train the biological tubular structure segmentation network with a self-attention mechanism and the biological tubular structure tracking network with skeleton point prediction. On the test data set, the method of the present invention achieves the best in all indicators of the reconstructed image compared with the mainstream traditional NGPST method and the deep learning NRTR method, and has a significant improvement in all indicators.
[0075] In order to more clearly demonstrate the technical solution and technical effects provided by the present invention, the method for reconstructing biological tubular structures in electron microscope microscopic images provided by the embodiment of the present invention is described in detail below with reference to a specific embodiment.
[0076] Example 1
[0077] like Figure 1As shown, an embodiment of the present invention provides a method for reconstructing biological tubular structures in electron microscope microscopic images, and takes the reconstruction of Drosophila trachea based on large-scale electron microscope microscopic images of Transformer as an example for explanation, specifically including: trachea segmentation and trachea tracking. The complex diversity of tracheal morphology and the uneven quality of electron microscope microscopic images have posed major challenges to tracheal reconstruction. In response to these problems, the present invention adopts a trachea segmentation method based on the Swin UNETR network (see document: Hatamizadeh A, et al., "Swin UNETR: Swin Transformers for semantic segmentation of brain tumors in MRI images", in BrainLes, 2021), and a trachea tracking method based on DETR and connection direction. The overall method steps of the present invention are as follows:
[0078] First, considering the large variation of tracheal radius and the long span of trachea, the commonly used 3D CNN method is limited by the size of the three-dimensional perception field and has a weak ability to process global context information. It is easy to miss points in the segmentation of long and thin trachea, resulting in tracheal breakage in the subsequent reconstruction process, affecting the accuracy of the reconstruction results. The Swin UNETR network is used as a biological tubular structure segmentation network with a self-attention mechanism. The self-attention mechanism is used to effectively process the long-distance dependent information of the trachea, improve the accuracy of tracheal segmentation, and reduce missed points. In addition, the existing traditional biological tubular structure tracking method is greatly affected by image quality, and it is difficult to reconstruct an accurate whole-brain trachea, and it is often necessary to manually introduce biological prior knowledge for correction. The latest deep learning tracking method NRTR directly predicts the skeleton point set and then uses a simple connection method to construct a tree structure, but its network training is unstable, and directly applied to electron microscope tracheal data will produce incorrect connections in dense trachea areas. Based on the NRTR method (see the literature: Wang Y, et al., “NRTR: Neuron reconstruction with Transformer from 3D optical microscopy images”, in IEEE TMI, 2024), the present invention adds a newly designed skeleton proposal point prediction submodule to the biological tubular structure tracking network with skeleton point prediction, selects features of interest from the output feature sequence output by the Transformer encoder to generate skeleton proposal points as the initial query of the Transformer decoder, optimizes the network structure, stabilizes network training, and improves network performance; and in the skeleton point set prediction, additionally predicts the connection direction of each skeleton point in the skeleton point set, and uses the connection direction to guide the process of connecting each skeleton point in the skeleton point set to obtain more accurate reconstruction results.
[0079] Specifically, the embodiment of the present invention uses a Swin UNETR network with a self-attention mechanism to perform trachea segmentation on the original electron microscope microscopic image to obtain a probabilistic trachea segmentation image; after pre-processing the probabilistic trachea segmentation image to obtain an image block for tracking, a biological tubular structure tracking network with skeleton point prediction is used to predict the image block to obtain a skeleton point set of the trachea structure in the image block, wherein the skeleton point set specifically includes the three-dimensional spatial position (composed of three-dimensional spatial coordinates and a radius) and a connection direction of the skeleton point, and the skeleton points in the skeleton point set are connected according to the connection direction to construct a tree structure reconstruction result of the trachea structure.
[0080] Each part will be described in detail next.
[0081] (1) Swin UNETR network with self-attention mechanism (i.e. biological tubular structure segmentation network with self-attention mechanism):
[0082] Tracheal segmentation is the first step in the tracheal reconstruction process, and the accuracy of tracheal segmentation often determines the accuracy of subsequent reconstruction results. Considering the large differences in tracheal morphology in the whole brain of Drosophila, such as large differences in diameter range and long tracheal span, the tracheal segmentation network needs to maintain spatial continuity in the slender trachea. To this end, the present invention adopts the Swin UNETR network with a self-attention mechanism, which not only retains the U-shaped structure and jump connection design of U-Net, effectively integrates multi-scale image features, but also uses a powerful self-attention mechanism to effectively capture long-distance dependencies in image sequences, thereby mastering global information and generating fine and spatially continuous segmentation results. Considering that the complete image is too large, it needs to be divided into blocks and input into the Swin UNETR network with a self-attention mechanism to obtain the segmentation results and piece them together into the whole brain segmentation results. Specifically, the physical resolution of the input image of the Swin UNETR network with a self-attention mechanism in this embodiment is 16nm×16nm×40nm, the voxel size is 128×128×16, and the output is the probability of whether each voxel belongs to the trachea.
[0083] (2) Biological tubular structure tracking network with skeleton point prediction:
[0084] Due to the dense distribution of some areas of the fruit fly trachea and the presence of noise in the images, the trachea tracking task in high-resolution electron microscope images faces the challenge of tracking errors. The traditional tracking methods of modern biological tissues are either based on sequential tracking or global graph optimization. NGPST, the mainstream tracking method for biological tissues, automatically samples seed points based on the segmented image, tracks to neighboring positions based on the image intensity, and determines termination and bifurcation based on manually set rules to reconstruct the morphology of biological tissues. These methods are highly dependent on image quality and often produce multiple errors in areas with poor image quality. Subsequent work often requires the introduction of biological prior knowledge to correct the tracking results, which requires manual adjustment of the correction rules based on the correction results, which is often time-consuming and labor-intensive.
[0085] In view of the above problems, the present invention proposes a biological tubular structure reconstruction method based on DETR (Carion N, et al., End-to-end object DEtection with Transformers, in ECCV, 2020) detection network and connection direction, which can learn and identify multi-scale and multi-level feature representations in the segmented image, and improve the tracking performance of areas with poor image quality and complex structures. Given an electron microscope slice image, the trachea segmentation probability is first predicted using the Swin UNETR network to obtain a three-dimensional segmentation probability image, and multiple segmentation results are stitched into an overall three-dimensional segmentation probability image; secondly, the overall three-dimensional segmentation probability is downsampled and cut into multiple image blocks of fixed size, and each image block is input into a biological tubular structure tracking network with skeleton point prediction to detect the three-dimensional spatial position and connection direction of the skeleton points in the image block; then, the three-dimensional spatial position and connection direction information of the skeleton point set are used to complete the construction of the tubular tree structure, and finally the reconstruction results of multiple image blocks are stitched together to obtain the trachea reconstruction result corresponding to the overall three-dimensional segmentation probability image.
[0086] The present invention proposes a connection direction-guided electron microscope microscopic image biological tubular structure tracking method, which specifically includes: an image feature extraction network, a Transformer-based skeleton point set prediction network and a skeleton point set connection module; the method flow is as follows Figure 2 As shown,
[0087] The Transformer-based skeleton point set prediction network includes: a Transformer encoder, a Transformer decoder, a skeleton proposal point set prediction submodule and a skeleton point set prediction submodule. The image feature extraction network uses the tracking image block obtained by dividing the overall three-dimensional segmentation probability image of the trachea into blocks as input, extracts image features, and then inputs the image features of each image block into the Transformer encoder as a word (token). The Transformer encoder calculates the dependency between each token in the input image features and outputs a series of skeleton proposal points. The Transformer decoder uses the output of the skeleton proposal point set prediction submodule as the query target, and generates a skeleton point feature sequence through the attention mechanism with the output features of the Transformer encoder, thereby generating a set of skeleton points containing probability, three-dimensional spatial position and connection direction, and then based on the predicted connection direction of each skeleton point, connects each skeleton point in the skeleton point set to generate the final tree structure reconstruction result of the tracheal structure.
[0088] 1) Network structures:
[0089] Compared with the high-resolution tracheal segmentation results of each voxel, the tree-like structure of the tracheal structure realizes a more simplified point set representation by abstracting multiple voxels in the segmented image into a spherical point. In order to input the whole brain segmentation image into the biological tubular structure tracking network with skeleton point prediction, the segmented image needs to be downsampled and cropped into smaller image blocks. The biological tubular structure tracking network with skeleton point prediction of the present invention reads the cropped probabilistic segmentation image blocks And predict the skeleton point set of the tracheal structure from it. The point set can be expressed as Where W, H, and D are the width, height, and depth of the image block, and N is a hyperparameter that needs to be greater than the number of skeleton points of the real tracheal structure corresponding to any image block. j ,y j , z j ), r j With d j are the normalized three-dimensional spatial position (consisting of three-dimensional spatial coordinates and radius) and connection direction of the skeleton point, c j ∈[0, 1] indicates the classification probability of the predicted point belonging to the skeleton point, d j is the direction vector of the three-dimensional prediction direction. Specifically, after many experiments, this embodiment selects the physical resolution of the input image of the biological tubular structure tracking network with skeleton point prediction to be 128nm×128nm×128nm (the image can be obtained by interpolation sampling of the three-dimensional probability segmentation image), and the voxel size is 64×64×64.
[0090] For the three-dimensional connection direction prediction, in actual training, the three-dimensional direction d j Split into horizontal angle ψ j Angle φ j , the two angles are predicted separately, and the calculation method is as follows:
[0091] d j =(d j,x , d j,y , ,d j,z ) (1);
[0092]
[0093] The present invention follows the method of directional target detection and adopts the method of "segmented prediction" (binning) to discretize the continuous angle into λ intervals, each of which represents a size of The angle range is expressed as ψ j For example, first predict the probability that the angle belongs to each interval Then predict the specific offset value of the angle in each interval (Required according to Normalized), φ j Similarly, in this way, the azimuth angle (ψ j ,φ j ) is converted to And to A j Make predictions. In the inference phase, we get After that, the specific angle is calculated according to the interval where the maximum probability of the two angles is located and the offset within the interval, and then the predicted direction d is obtained according to the inverse process of the above formulas (1) to (3). j .
[0094] The image feature extraction network includes: a three-dimensional residual network and a feature pyramid network. The input of the image feature extraction network is a single-channel segmentation probability image After the image feature extraction network, low-resolution image features are obtained The low-resolution image features I f Input to Transformer-based skeleton point set
[0095] The prediction network uses each pixel of the low-resolution image feature as a token, and all pixels are input into the Transformer encoder as a sequence. The sequence is Among them, t represents the length of the sequence, that is, the low-resolution image feature I f The image size is t = D0×H0×W0. At the same time, considering the spatial position in the image, the position encoding of each feature position is calculated, and the position encoding is The position encoding is implemented using cosine position encoding. The sequence S and the position encoding P are used together as the input of the Transformer encoder.
[0096] As a feature extractor of the Transformer-based skeleton point set prediction network, the Transformer encoder consists of K encoder layers, each of which consists of a multi-head self-attention module and a multi-layer perceptron (MLP). Since the Transformer architecture is permutation invariant, the output of the upper encoder layer and the position encoding are input into the next encoder layer each time. The Transformer encoder maps the sequences S and P into the feature sequence Z through the self-attention mechanism, and inputs the feature sequence Z into the skeleton proposal point set prediction submodule and the Transformer decoder.
[0097] The Transformer decoder relies on target queries to model the relationship between the skeleton point set and the segmented image. In order to obtain better initial query points, the present invention proposes a dynamic query point generation process. In the NRTR method, query points are generated by a learnable linear layer. However, within a single iteration round, for different images, these query point inputs are usually static, that is, they are the same. In the initial learning stage of the network, this linear layer is often generated in an all-zero or randomly initialized manner, resulting in unclear meaning of the query points, which in turn affects the stability and convergence speed of the training. In order to further utilize the Transformer encoder, the present invention selects some tokens features from the feature sequence Z output by the last encoder layer of the Transformer encoder as a priori to enhance the Transformer decoder query. The present invention connects a skeleton proposal point set prediction submodule to a Transformer encoder, calculates the scores of all features, and selects the first N (the same number as the query points) encoder features therefrom. Specifically, the skeleton proposal point set prediction submodule predicts the classification probability that each feature position point belongs to the tracheal structure skeleton point, and uses the features of the N tokens with the highest probability as proposal points. The features of these proposal points and the corresponding position codes are used as the target query of the Transformer decoder, so that the target query is associated with the target area that may exist in the segmented image at the beginning, so that the Transformer decoder can focus on these specific areas more effectively. The present invention constrains the skeleton proposal point set prediction submodule and the Transformer encoder by calculating the loss between the prediction result of the skeleton proposal point set prediction submodule and the true value. Through such a method, the training of the Transformer-based skeleton point set prediction network can be made more stable, converge faster, and obtain better prediction results.
[0098] The skeleton proposal point set prediction submodule includes six multi-layer perceptrons. The first two multi-layer perceptrons are used to predict the classification probability and three-dimensional spatial position of the skeleton proposal points, and the last four multi-layer perceptrons are used to predict the two angles corresponding to the connection direction of each skeleton proposal point, including the classification probability of different angle intervals of the horizontal angle ψ and the vertical angle φ and the offset value within the interval. In the skeleton proposal point set prediction submodule, the feature sequence Z output by the Transformer encoder passes through the skeleton proposal point set prediction submodule to output the probability of t skeleton proposal points. Three-dimensional space position angle The dynamic query of the present invention will use this information to propose points according to the probability C of t skeletons. E Select partial information as the target initial query of the Transformer decoder. Specifically, the present invention adopts the Top-K strategy to select the probability C of t skeleton proposal points E The top N predicted points with the highest probability are used as query proposal points, and the corresponding N features are selected from the feature sequence Z output by the Transformer encoder and recombined into a sequence Q 0 , input to the Transformer decoder as the initial object query matrix of the Transformer decoder, and also encode the position of these N proposal points P 0 Input to each layer of the Transformer decoder.
[0099] The Transformer decoder models the relationship between the skeleton point target and the image through the target object query (Object Query). It consists of K decoder layers, each of which consists of a multi-head self-attention module, a multi-head cross-attention module (Multi-head Cross-attention) and a multi-layer perceptron. The predicted feature Q of the skeleton proposal point set prediction submodule is 0 With P 0 , according to the feature sequence Z output by the encoder, it is input into the Transformer decoder. Since the Transformer decoder is also permutation-invariant, each layer of the decoder also needs to input the position code. The Transformer decoder uses the cross-attention mechanism to capture the dependency between the target skeleton point and all positions in the image. The output is consistent with the sequence Q 0 Target skeleton point features Q of the same dimension.
[0100] The target skeleton point feature Q is input into the skeleton point set prediction submodule to obtain detailed information of the final tracheal structure skeleton points. The network architecture of the skeleton point set prediction submodule is consistent with that of the skeleton proposal point set prediction submodule, but the specific network parameters are different and need to be learned during network training. The output feature sequence of the Transformer decoder is output through the skeleton point set prediction submodule to output the classification probability of N skeleton points Three-dimensional space position angle The final skeleton point set prediction result is used to subsequently construct the tree structure of the trachea structure.
[0101] 2) Loss function:
[0102] The loss function for training a biological tubular structure tracking network with skeleton point prediction is as follows:
[0103] The skeleton point set prediction submodule outputs a fixed-size predicted skeleton point set, which contains N predicted skeleton points. In addition, in order to make full use of the Transformer encoder information, the Transformer encoder outputs the features of t voxel points, and selects N skeleton proposal points in the skeleton proposal point set prediction submodule to input into the Transformer decoder.
[0104] The loss function of the present invention is calculated using the N skeleton proposal points of the skeleton proposal point set prediction submodule and the N predicted skeleton points of the skeleton point set prediction submodule. First, these skeleton point sets must be matched with the real point set, and then the loss is calculated. For the convenience of introduction, it is first assumed that the predicted skeleton point set has been matched with the real point set, and the matching loss is introduced. Then the matching method is given to calculate the final total loss.
[0105] (21) Various loss functions:
[0106] The skeleton proposal point set prediction submodule will output N skeleton proposal points. The skeleton point set prediction submodule will output N skeleton points. Assuming that there are M skeleton points in the true value, N>M is required. First, NM categories are added to the true value as empty. points, so that there are finally N points in the true value, which are matched one-to-one with the predicted N skeleton points. For a predicted skeleton point and its matching point Y in the true value i , the loss is calculated as follows:
[0107]
[0108] Among them, c i , L i , A i Represent the true value Y iThe classification probability, three-dimensional spatial position and angle of , wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius; Respectively represent the prediction results The classification probability, three-dimensional spatial position and angle of ω, wherein the three-dimensional spatial position is composed of coordinates and radius; cls ,ω box ,ω iou ,ω dir are hyperparameters respectively; represents the classification loss, which is set to the cross entropy loss function; and Both represent position loss, is the L1 loss function, is the sphere intersection-union ratio loss function; is the direction loss, The calculation is as follows:
[0109]
[0110] in, Represents the true value of the direction of each skeleton point d i Corresponding horizontal angle classification, horizontal angle internal offset value, vertical angle classification, vertical angle internal offset; Respectively represent the prediction direction Horizontal angle classification, horizontal angle internal offset value, vertical angle classification, vertical angle internal offset; represents the angle classification loss, which is set to the cross entropy loss function; It is the intra-class offset regression loss of the angle, which is set to L1 loss, and only calculates the difference between the predicted value and the true value in the offset value of the angle target interval.
[0111] The following is an explanation of the loss of the predicted skeleton point set. Assume that their best matches have been obtained The loss is calculated as follows:
[0112]
[0113] When the true point is a skeleton point, The classification, position and orientation losses are calculated. When the ground truth point is empty, only the classification loss is calculated. The loss of two point sets is the sum of the losses of each point. The next section will introduce how to determine the optimal match between point sets.
[0114] 3) Point set matching:
[0115] The above explains how to calculate the loss when determining the point set matching. The following will explain how to find the best match between point sets. Here, the point set matching is defined as: for a true value skeleton point set Y with M points, and a predicted skeleton point set with N points For each skeleton point in the predicted skeleton point set, how to find the most accurate unique matching point from the real point set. The present invention adopts the Hungarian algorithm for matching. First, NM empty set points are added to the real point set. After the number of points in the two sets is consistent, the point set matching scheme of all one-to-one corresponding points is considered to minimize the optimal matching loss. The calculation formula is as follows:
[0116]
[0117] Using the above method, the optimal matching solution between the predicted skeleton point set and the true value skeleton point set is found. On the one hand, in order to make the final output as close to the true value skeleton point set as possible, the output of the skeleton point set prediction submodule needs to be constrained; on the other hand, in order to make the initialization target query of the Transformer decoder as close to the true value skeleton point set as possible, the output of the skeleton proposal point set prediction submodule also needs to be constrained. Therefore, the final loss of the skeleton proposal point set prediction submodule is:
[0118]
[0119] in, Represents the skeleton proposal point set prediction submodule using the Top-K strategy to select the N most likely skeleton proposal point sets; Represents the skeleton proposal point set Optimal match with the true value skeleton point set Y; Represents the N skeleton point sets output by the skeleton point set prediction submodule, Represents the predicted skeleton point set The best match with the true value skeleton point set Y. ω1 and ω2 are hyper parameters.
[0120] 4) Point set connection:
[0121] The skeleton point set connection module connects the predicted skeleton point set into a tree structure based on the predicted direction of each skeleton point. After the skeleton point set is obtained through the Transformer-based skeleton point set prediction network, the skeleton point set needs to be processed and connected to form the final tree structure reconstruction result. First, a probability threshold of 0.5 is used to filter the skeleton points, and the skeleton points with a classification probability less than 0.5 are deleted. The predicted connection direction of each skeleton point is calculated based on the angle information predicted by the skeleton point. A point edge graph is established for all the remaining skeleton points. An edge relationship is established between two skeleton points whose distance between the two points is less than the distance threshold and whose angle difference between the two points and the predicted direction is less than the angle threshold. Then, the angle difference and the distance between the points are used as the edge weight, that is:
[0122] e i,j =ω3dist(P i , P j )+
[0123] ω4min{[1-cos(d i ,dir i,j )],[1-cos(d j ,dir j,i )]} (9);
[0124] Among them, (P i , P j ) represents the position of skeleton point i and skeleton point ,; (d i ,dir i,j ) represents the predicted direction of skeleton point i and the direction from skeleton point i to skeleton point j; dist(P i , P j ) represents the Euclidean space distance between skeleton point i and skeleton point ,; cos(d i ,dir i,j ) represents the predicted direction of skeleton point i and the cosine distance from skeleton point i to skeleton point i; ω3 and ω4 are both hyperparameters; since there cannot be a loop in the tracheal structure skeleton SWC file (that is, there is only one path between two skeleton points), the minimum spanning tree is used to remove the loop from the constructed point-edge graph to obtain the final tree structure of the tracheal structure.
[0125] 5) Overall framework:
[0126] The method of the present invention is mainly divided into two steps. The first step is to use the Swin UNETR network with a self-attention mechanism to extract the tracheal structure area from the original electron microscope microscopic image. Through the self-attention mechanism of the Transformer network, the long-distance dependencies in the image are captured to obtain a more accurate tracheal structure segmentation result with better spatial continuity. The second step is to use a biological tubular structure tracking method guided by a connection direction using a biological tubular structure tracking network with skeleton point prediction to track the trachea. This method can extract multi-level and multi-scale image features, and can adaptively process areas with poor image quality and areas with complex biological structures to generate better quality and more accurate tracking results. By using the connection direction of the skeleton point set predicted by the biological tubular structure tracking network with skeleton point prediction to guide the connection process, a more accurate tracheal structure reconstruction result is obtained.
[0127] Table 1 shows the accuracy comparison of each method for Drosophila electron microscope image trachea reconstruction results
[0128]
[0129] Table 2 shows the ablation experimental results of the method of the present invention on the Drosophila trachea tracking task
[0130]
[0131] It can be seen from Table 1 that, compared with the traditional tracking method NGPST and the deep learning method NRTR, the method of the present invention achieves the best in all indicators and has a significant improvement. It can be seen from Table 2 that the method of the present invention achieves more accurate reconstruction results by using the skeleton proposal point prediction submodule to increase the Transformer decoder query and use the direction prediction information to guide the tracheal structure connection.
[0132] The embodiment of the present invention uses a biological tubular structure segmentation network with a self-attention mechanism to perform fruit fly trachea segmentation in fruit fly whole brain electron microscope microscopic images, thereby obtaining a trachea tracking result with a more accurate three-dimensional spatial position of skeleton points and better spatial continuity, thereby solving the problem of missing segmentation of slender trachea. The present invention achieves accurate three-dimensional tracking of tracheal structures in electron microscope microscopic images by using a biological tubular structure tracking network with skeleton point prediction based on a detection network and connection direction prediction. The present invention proposes a query point dynamic generation method in the skeleton point prediction submodule, which makes network training more stable and obtains better prediction results. A tree-like connection structure of the tracheal structure is constructed based on the predicted connection direction of the skeleton points to obtain accurate three-dimensional reconstruction results. According to the method of the present invention, accurate three-dimensional reconstruction results of the trachea can be obtained with a small amount of tracheal data annotation in fruit fly whole brain electron microscope microscopic images, see. Figure 4. Under experimental verification, the method of the present invention has achieved the most advanced three-dimensional reconstruction performance of biological tubular structures.
[0133] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0134] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims. The information disclosed in the background technology section of this article is only intended to deepen the understanding of the overall background technology of the present invention, and should not be regarded as an admission or in any form that the information constitutes prior art known to those skilled in the art.
Claims
1. A method for reconstructing biological tubular structures in electron microscope images, characterized in that: include: Step 1, the original electron microscope microscopic image is divided into blocks to obtain block images, and a biological tubular structure segmentation network with a self-attention mechanism is used to capture the long-range dependency relationship of the biological tubular structure in the original electron microscope microscopic image through the self-attention mechanism, and each block image is segmented into biological tubular structures block by block to obtain a biological tubular structure probability segmentation image based on the long-range dependency relationship; Step 2, pre-processing the biological tubular structure probability segmentation image obtained in step 1 to obtain a tracking image block, predicting the obtained tracking image block through a biological tubular structure tracking network with skeleton point prediction to obtain a skeleton point set of the biological tubular structure, connecting each skeleton point of the skeleton point set of the biological tubular structure according to the connection direction to construct a tree-like biological tubular structure, and obtaining a reconstruction result of the biological tubular structure.
2. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 1, characterized in that: In the step 2, the biological tubular structure probability segmentation image obtained in step 1 is pre-processed in the following manner to obtain a tracking image block, the obtained tracking image block is predicted by a biological tubular structure tracking network with skeleton point prediction to obtain a skeleton point set of the biological tubular structure, and the skeleton points of the skeleton point set of the biological tubular structure are connected according to the connection direction to construct a tree-like biological tubular structure, and the reconstruction result of the biological tubular structure is obtained, including: Step 21, using the segmentation probability of the biological tubular structure region in the biological tubular structure probability segmentation image obtained in step 1, obtain a block-by-block three-dimensional segmentation probability image, combine multiple blocks of three-dimensional segmentation probability images into an overall three-dimensional segmentation probability image, downsample the overall three-dimensional segmentation probability image obtained and cut it into at least one fixed-size tracking image block, Step 22, inputting each tracking image block obtained in step 21 into a biological tubular structure tracking network with skeleton point prediction to perform skeleton point prediction, and obtaining the three-dimensional spatial position and connection direction of the skeleton point of the biological tubular structure in the tracking image block, wherein the three-dimensional spatial position is composed of a three-dimensional spatial coordinate and a radius; Step 23, using the obtained three-dimensional spatial position and connection direction of the skeleton points of the biological tubular structure to connect each skeleton point, complete the three-dimensional construction of the tree structure of the biological tubular structure of each tracking image block, and finally splice the tree structure reconstruction results of the biological tubular structure of multiple tracking image blocks to obtain the biological tubular structure reconstruction result corresponding to the overall three-dimensional segmentation probability image.
3. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 1 or 2, characterized in that: The biological tubular structure segmentation network with a self-attention mechanism adopts a Swin UNETR network with a self-attention mechanism; The physical resolution of the input electron microscope image of the Swin UNETR network with self-attention mechanism is 16nm×16nm×40nm, and the voxel size is 128×128×16; The output of the Swin UNETR network with self-attention mechanism is a probability image of whether each voxel belongs to a biological tubular structure, that is, a biological tubular structure probability segmentation image; The biological tubular structure tracking network with skeleton point prediction includes: an image feature extraction network, a Transformer-based skeleton point set prediction network and a skeleton point set connection module; wherein, The image feature extraction network can take the tracking image block obtained by pre-processing the biological tubular structure probability segmentation image obtained in step 1 as input, and extract low-resolution image features as output; The Transformer-based skeleton point set prediction network is communicatively connected to the output end of the image feature extraction network, and can use the low-resolution image features output by the image feature extraction network as an input tag sequence, calculate the association between each image block tag in the input tag sequence, generate a skeleton proposal point feature sequence of a biological tubular structure, and predict the biological tubular proposal point feature sequence to obtain the classification probability, three-dimensional spatial position and connection direction of each skeleton point in the skeleton point sequence of the biological tubular structure, and the classification probability, three-dimensional spatial position and connection direction of each skeleton point constitute a skeleton point set, and the position consists of coordinates and radius; The skeleton point set connection module is communicatively connected to the output end of the Transformer-based skeleton point set prediction network, and can take the skeleton point set predicted by the Transformer-based skeleton point set prediction network as input, and based on the predicted connection direction of each skeleton point, connect each skeleton point in the skeleton point set to generate a final tree structure of the biological tubular structure.
4. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 3, characterized in that: The image feature extraction network is composed of a three-dimensional residual network and a feature pyramid network connected in sequence, wherein the three-dimensional residual network takes the tracking image block obtained by pre-processing the biological tubular structure probability segmentation image obtained in step 1 as input, and can perform three-dimensional residual processing and then output it to the feature pyramid network to extract low-resolution image features to obtain low-resolution image features; The Transformer-based skeleton point set prediction network consists of a Transformer encoder, a skeleton proposal point set prediction submodule, a Transformer decoder and a skeleton point set prediction submodule; wherein, The Transformer encoder is connected to the image feature extraction network for communication, and can take the position code of each pixel position of the low-resolution image feature output by the image feature extraction network and the pixel feature sequence composed of all pixel features of the low-resolution image feature as input, and map the pixel feature sequence and the position code of each pixel position to a feature sequence containing position information and content information of all image features through a self-attention mechanism, and output the feature sequence; The skeleton proposal point prediction submodule is communicatively connected to the output end of the Transformer encoder, and can use the feature sequence output by the Transformer encoder as input to respectively predict the classification probability, three-dimensional spatial position and connection direction of the biological tubular structure skeleton proposal point, wherein the three-dimensional spatial position is composed of a three-dimensional spatial coordinate and a radius; The Transformer decoder is respectively connected to the Transformer encoder and the skeleton proposal point set prediction submodule in communication, and can use the first N biological tubular structure skeleton points with the largest probability in the classification probability output by the skeleton proposal point set prediction submodule as skeleton proposal points, select N features corresponding to the N skeleton proposal points from the feature sequence to reassemble into a biological tubular structure point prediction feature proposal sequence as an initial object query matrix, and use the position encoding of the N skeleton proposal points and the feature sequence output by the Transformer encoder to input each layer of the Transformer decoder as a key and value, use a cross attention mechanism to capture the dependency between the target biological tubular structure point and all positions in the image, and output the target biological tubular structure point feature with the same shape as the biological tubular structure point prediction feature proposal sequence; The skeleton point set prediction submodule is communicatively connected to the Transformer decoder, and can use the target biological tubular structure point features output by the Transformer decoder as input to obtain the classification probability, three-dimensional spatial position and angle of each skeleton point, wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius, and the obtained skeleton points constitute a skeleton point set.
5. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 4, characterized in that: In the Transformer-based skeleton point set prediction network, the Transformer encoder is composed of a plurality of encoder layers connected in sequence, and each encoder layer is composed of a multi-head self-attention module and a multi-layer perception module; The skeleton proposal point set prediction submodule is composed of six multi-layer perceptrons, wherein the first two multi-layer perceptrons are used to predict the classification probability and three-dimensional spatial position of the skeleton points of the biological tubular structure, and the last four multi-layer perceptrons are used to predict the horizontal angle and vertical angle corresponding to the connection direction; The Transformer decoder consists of multiple decoder layers connected in sequence, and each decoder layer consists of a multi-head self-attention submodule, a multi-head cross-attention submodule and a multi-layer perception submodule.
6. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 4, characterized in that: The skeleton point set prediction submodule is composed of six multi-layer perceptrons, wherein the first two multi-layer perceptrons are used to predict the classification probability and three-dimensional spatial position of biological tubular structure points, respectively, and the last four multi-layer perceptrons are used to predict the horizontal angle and vertical angle corresponding to the connection direction.
7. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 3, characterized in that: The total loss function of the biological tubular structure tracking network with skeleton point prediction is: The meaning of each parameter is: Represents all predicted skeleton proposal points The matching point with its true value The sum of the matching losses; Represents all predicted skeleton points The matching point with its true value The sum of the matching losses; The skeleton proposal point set prediction submodule of the skeleton point set prediction network based on Transformer of the biological tubular structure tracking network with skeleton point prediction uses the Top-K strategy to select the skeleton proposal point set consisting of N biological tubular structure points with the highest probability; Represents the skeleton proposal point set Optimal match with the true value skeleton point set Y; Represents a skeleton point set consisting of N skeleton points output by the Transformer-based skeleton point set prediction network; Represents the skeleton point set The optimal match with the true value skeleton point set Y; ω1 and ω2 are hyper parameters respectively; Of the total losses above, The calculations are as follows: Among them, c i , L i , A i Represent the true value Y i The classification probability, three-dimensional spatial position and angle of , wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius; Respectively represent the prediction results The classification probability, three-dimensional spatial position and angle of ω, wherein the three-dimensional spatial position is composed of three-dimensional spatial coordinates and a radius; cls ,ω box ,ω iou ,ω dir are hyperparameters respectively; represents the classification loss, which is set to the cross entropy loss function; and Both represent position loss, is the L1 loss function, is the sphere intersection-union ratio loss function; is the direction loss, The calculation is as follows: in, Represents the true value direction of each skeleton point d i Corresponding horizontal angle classification, horizontal angle internal offset value, vertical angle classification and vertical angle internal offset; Respectively represent the prediction direction Horizontal angle classification, horizontal angle internal offset value, vertical angle classification and vertical angle internal offset; represents the angle classification loss, which is set to the cross entropy loss function; It represents the intra-class offset regression loss of angle, adopts L1 loss function, and only calculates the difference between the predicted value and the true value in the offset value of the angle target interval.
8. The method for reconstructing biological tubular structures in electron microscope microscopic images according to claim 1 or 2, characterized in that: In the step 2, the skeleton point set of the biological tubular structure is connected according to the connection direction to construct a tree-like biological tubular structure to obtain a reconstructed image of the biological tubular structure, including: The skeleton points whose classification probability is less than the predetermined threshold are deleted from the skeleton point set of the biological tubular structure, and the remaining skeleton points are used as the screened skeleton points. The predicted connection direction of each skeleton point is calculated based on the predicted angle of the screened skeleton points. A point-edge graph is established for all the remaining skeleton points. An edge relationship is established between two skeleton points whose distance between the two points is less than the distance threshold and whose angle difference between the two points and the predicted direction is less than the angle threshold. Then, the angle difference and the distance between the points are used as the edge weight e. i,j , e i,j for: And i,j =ω3dist(P i ,P j )+ ω4min{[1-cos(d i ,dir i,j )],[1-cos(d j ,d i r j,i )]} (9); Among them, dist(P i , P j ) represents the Euclidean space distance between skeleton point i and skeleton point j; (P i , P j ) represents the position of skeleton point i and skeleton point j; cos(d i ,dir i,j ) represents the predicted direction of skeleton point i and the cosine distance from skeleton point i to skeleton point j; (d i ,dir i,j ) represents the predicted direction of skeleton point i and the direction from skeleton point i to skeleton point j; cos(d j ,dir j,i ) represents the predicted direction of skeleton point j and the cosine distance from skeleton point j to skeleton point i; (d j ,dir j,i ) represents the predicted direction of skeleton point j and the direction from skeleton point j to skeleton point i; ω3 and ω4 are both hyperparameters; min{} represents the minimum value; The minimum spanning tree is used to remove the loops from the constructed point-edge graph, and the final tree structure of the biological tubular structure is obtained.
9. A processing device, characterized in that: include: at least one memory for storing one or more programs; At least one processor is capable of executing one or more programs stored in the memory, and when the one or more programs are executed by the processor, the processor is capable of implementing the method described in any one of claims 1 to 8.
10. A readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 can be implemented.
Citation Information
Patent Citations
Tubular structure image segmentation method
CN108510506A
Cross-modal double-branch complementary fusion image segmentation method and device
CN115482241A
Method and device for segmenting lung blood vessels in CT image, terminal equipment and medium
CN116740081A
