Volume electron microscope data visualization method, device and storage medium for generating instantiated projections

Abstract

The present invention discloses a volume electron microscope data visualization method, device and storage medium for generating instantiated projections, relating to the field of image processing technology, comprising: performing dot multiplication on a texture data block of volume electron microscope data and a segmentation mask block corresponding to the volume electron microscope data to obtain instantiated texture data filtered by the mask, and superimposing a marked subcellular structure on the instantiated texture data filtered by the mask to obtain instantiated texture data with superimposed marks; converting the instantiated texture data with superimposed marks on a plane at any angle into an orthogonal plane by data rotation, and using a projection function to obtain an instantiated projection image on the current orthogonal plane and normalize it; synthesizing the normalized projection image after spatial scale correction, thereby outputting a visualization result corresponding to the current ID; the volume electron microscope data visualization method can quickly generate a visualization format containing morphology and texture of multiple objects in the volume electron microscope data, thereby realizing rapid retrieval, verification and analysis of the volume electron microscope data.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a volume electron microscope data visualization method, device and storage medium for generating instantiated projections. Background Art

[0002] The neural tissue image sequences acquired by electron microscopy technology have high resolution but huge data volumes. How to effectively extract and visualize neuronal structural information from segmentation data is a key technical challenge.

[0003] Browsing or visualizing volume electron microscopy data requires traversing all layers or displaying them using 3D rendering. For example, in sequential images, each image represents a layer, while 3D structures in volume electron microscopy data often span multiple layers. Therefore, browsing the structure requires traversing all slices where the structure exists. For example, in electron microscopy imaging of biological tissues such as neurons, these cellular structures span too large a range in volume electron microscopy data, requiring only small data blocks for browsing. This often requires accessing all layers of these small data blocks, resulting in inefficiencies.

[0004] Another approach, using 3D rendering, involves algorithmically generating a tabular mesh from the surface of each object in the data block and rendering it. However, this mesh generation algorithm typically requires specialized design to handle different surface topography, and results vary depending on the rendering tool and settings, making it difficult to implement for widespread application. Furthermore, rendering based on the surface mesh completely loses the image texture, which is crucial for understanding and subsequent processing of the data object.

[0005] In general, the existing technology lacks an efficient and intuitive visualization method for multiple objects in volume electron microscopy data. Summary of the Invention

[0006] Based on the technical problems existing in the background technology, the present invention proposes a volume electron microscopy data visualization method, device and storage medium for generating instantiated projections, which can quickly generate a visualization format containing the morphology and texture of multiple objects in volume electron microscopy data, and realize the rapid retrieval, verification and analysis of volume electron microscopy data.

[0007] The present invention proposes a method for visualizing volume electron microscopy data by generating instantiated projections, comprising:

[0008] Multiplying the texture data block of the volume electron microscopy data by a segmentation mask block corresponding to the volume electron microscopy data to obtain mask-filtered instantiated texture data, and superimposing the marked subcellular structure on the mask-filtered instantiated texture data to obtain superimposed marked instantiated texture data;

[0009] The instanced texture data of the superimposed marker on the plane of any angle is converted to the orthogonal plane by data rotation, and the instanced projection image on the current orthogonal plane is obtained and normalized using the projection function;

[0010] The normalized projection image is synthesized after spatial scale correction to output the visualization result corresponding to the current ID.

[0011] Furthermore, the texture data block and the segmentation mask block have the same size and are aligned;

[0012] Extract all non-zero ID values ​​from the segmentation mask block by removing duplicate values ​​to form a set , where the set Each ID represents an instance object;

[0013] Set the The voxel with the first ID value is 1, and the others are 0, thus creating the The segmentation mask block of IDs, .

[0014] Furthermore, the subcellular structure markers are superimposed on the mask-filtered instantiated texture data, specifically:

[0015] The intersection area between the mask-filtered instantiated texture data and the set subcellular structure segmentation mark mask is obtained and marked;

[0016] If there is no subcellular structure to be marked, all values ​​of the subcellular structure segmentation mark mask are set to 0;

[0017] If there are multiple subcellular structures to be marked, the instantiated texture data after superimposing the current subcellular structure segmentation mark mask will be superimposed on the next subcellular structure segmentation mark mask in sequence.

[0018] Furthermore, objects are marked by marking functions. When marking multiple objects, each marking function is independent. The marking functions are as follows:

[0019] ;

[0020] ;

[0021] in, For the The mask-filtered instanced texture data for each object. Mask-filtered instanced texture data for overlay marking. is the 3D coordinate of the voxel in the texture data block, is the marking function, is a mask for subcellular structures, For example, It's a red mark. To highlight, The brightness value is usually 255. is the highlight coefficient.

[0022] Furthermore, the instantiated texture data with the superimposed mark on the arbitrary angle plane is converted into an orthogonal plane by data rotation, and the orthogonal plane includes 、 and flat;

[0023] Using the projection function, we can get the projection image on the current orthogonal plane, specifically:

[0024] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale;

[0025] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale;

[0026] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale.

[0027] Furthermore, the normalized projection image is corrected by spatial scale, specifically:

[0028] right Plane projection, Plane projection and The projection images on the plane projection are scaled to match the proportions of the actual physical space.

[0029] Furthermore, GPU is used to accelerate the calculation of the volume electron microscopy data visualization method.

[0030] A volume electron microscope data visualization device for generating instantiated projections, comprising a data preprocessing module, a projection map calculation module, and a visualization module;

[0031] The data preprocessing module is used to perform a dot multiplication on the texture data block of the volume electron microscopy data and the segmentation mask block corresponding to the volume electron microscopy data to obtain instantiated texture data filtered by the mask, and to superimpose the marked subcellular structure on the instantiated texture data filtered by the mask to obtain instantiated texture data superimposed with the marks;

[0032] The projection image calculation module is used to convert the instanced texture data of the superimposed mark on the plane of any angle into an orthogonal plane by data rotation, and use the projection function to obtain the projection image on the current orthogonal plane and normalize it;

[0033] The visualization module is used to synthesize the normalized projection image after spatial scale correction, thereby outputting the visualization result corresponding to the current ID.

[0034] Furthermore, the data preprocessing module includes a mask superposition module, a first marking module and a second marking module;

[0035] The mask superposition module is used to obtain and mark the intersection area between the instantiated texture data filtered by the mask and the set subcellular structure segmentation mark mask;

[0036] If there is no subcellular structure to be marked, all values ​​of the subcellular structure segmentation mark mask are set to 0 through the first marking module;

[0037] If there are multiple subcellular structures to be marked, the instantiated texture data after superimposing the current subcellular structure segmentation mark mask is sequentially superimposed with the next subcellular structure segmentation mark mask through the second marking module.

[0038] A computer-readable storage medium stores a plurality of classification programs, wherein the plurality of classification programs are used to be called by a processor and execute the above-mentioned volume electron microscope data visualization method.

[0039] The advantages of the volume electron microscopy data visualization method, device and storage medium for generating instantiated projections provided by the present invention are that they can quickly generate a visualization format containing the morphology and texture of multiple objects in volume electron microscopy data, thereby realizing rapid retrieval, verification and analysis of volume electron microscopy data. Moreover, the instantiated projection map of this embodiment contains structural and texture projection information, which is close to the visual effects of X-ray and CT imaging, provides an intuitive neuronal morphological representation, and helps researchers to quickly understand complex three-dimensional structures. The instantiated projection map can calculate the projections of multiple objects at the same time, and can also be combined with subcellular structures (synapses and mitochondria, etc.) to achieve the distinction of the position, morphology and spatial relationship of multiple three-dimensional objects in the instantiated projection map. It can be applied to volume electron microscopy data and segmentation results from different sources, and supports the visualization requirements of various neuronal morphologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a structural schematic diagram of the present invention;

[0041] Figure 2 Schematic diagram showing input data: (a) volume electron microscopy data of biological neural tissue, (b) segmentation mask, (c) rendering of a small number of objects, and (d) a single object in a slice.

[0042] Figure 3 The instantiated projection image and the three-view image are shown in Figure 1. A is the instantiated projection image obtained by combining the segmented object and the electron microscope image in the volume electron microscope data and projecting them along multiple directions. B is the three-view image. The instantiated projection image is generally presented in the three-view format. Since the electron microscope image is used in the projection, the mitochondrial texture can be observed in the instantiated projection image (no mitochondrial mask is set).

[0043] Figure 4 Schematic diagrams of three practical variants of segmentation projection maps; (a) is a segmentation projection map using only segmentation for projection, (b) is a labeled projection map using neuron segmentation projection and superimposed synaptic segmentation (red), and (c) is a texture projection map using segmentation and texture dot product projection;

[0044] Figure 5 These are instantiated projections of multiple objects in the same area; (a) is the projection of dendritic bifurcation, (b) is the projection of dendritic trunk, (c) is the projection of axonal endings, and (d) is the projection of thin axonal endings. DETAILED DESCRIPTION

[0045] The technical solutions of the present invention are described in detail below through specific embodiments. Numerous specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0046] In this embodiment, some specialized terms are used, such as: volume electron microscopy data → sequential images, data blocks, where each sequential image is a layer and when combined, they form a volume block; 3D structures in volume electron microscopy data → objects, segmented objects. The volume electron microscopy data mentioned in this embodiment often contains multiple 3D structural objects, and each object may span a data block at any angle.

[0047] like Figures 1 to 5 As shown, the volume electron microscope data visualization method for generating instantiated projections proposed by the present invention includes the following steps:

[0048] Step 1: dot-multiplying the texture data block of the volume electron microscopy data and the segmentation mask block corresponding to the volume electron microscopy data to obtain mask-filtered instantiated texture data, and superimposing the marked subcellular structure on the mask-filtered instantiated texture data to obtain superimposed marked instantiated texture data;

[0049] Step 2: Convert the instanced texture data with superimposed marks on any angle plane to an orthogonal plane by data rotation, and use the projection function to obtain the instanced projection image on the current orthogonal plane and normalize it;

[0050] Step 3: The normalized projection image is synthesized after spatial scale correction to output the visualization result corresponding to the current ID.

[0051] According to steps one to three, a visualization format containing the morphology and texture of multiple objects in the volume electron microscope data can be quickly generated, thereby realizing rapid retrieval, verification and analysis of the volume electron microscope data. Moreover, the instantiated projection map of this embodiment contains structural and texture projection information, which is close to the visual effects of X-ray and CT imaging, and provides an intuitive representation of neuronal morphology, which helps researchers to quickly understand complex three-dimensional structures. The instantiated projection map can calculate the projections of multiple objects at the same time, and can also be combined with subcellular structures (synapses and mitochondria, etc.) to achieve the distinction of the position, morphology and spatial relationship of multiple 3D objects in the instantiated projection map. It can be applied to volume electron microscope data and segmentation results from different sources, and supports the visualization requirements of various neuronal morphologies.

[0052] It should be noted that the volume electron microscope data visualization method of this embodiment can be used for neurons or other structural objects such as glial cells and muscle cells that densely occupy the three-dimensional space of the tissue, so as to realize the visualization of the neuronal morphology or type cell morphology of the nervous tissue. The following is explained with neurons, and other types of cells can be obtained directly in a similar way.

[0053] In one embodiment, step one is specifically:

[0054] Texture data blocks ( Figure 2 (a) is a three-dimensional data block composed of an electron microscope image sequence. The grayscale value of each pixel represents the brightness of the neural tissue electron microscope image. Due to the heavy metal staining process in the sample preparation process, the brightness of lipid-containing areas such as cell membranes and organelle membranes is relatively high. is the three-dimensional data coordinate.

[0055] Neuron segmentation mask ( Figure 2 (b)): A 3D data block of the same size as the texture data block, where each neuron object is represented by a unique ID and the background is usually set to 0. The texture data block and the segmentation mask block have the same size and are aligned. The mask can be generated by manual annotation or existing automatic segmentation algorithms.

[0056] Subcellular structure segmentation labeling mask : A three-dimensional data block with the same size as the texture data block. The background is usually set to 0. The subcellular structures that need to be marked, such as mitochondria and synapses, have non-zero values. A type of subcellular structure is often represented by an ID (for example, the labeling of a single subcellular structure is used later, and the ID is 1). The mask can be generated by manual annotation or existing automatic segmentation algorithms.

[0057] The specific electron microscope data loading process is as follows: read the 3D electron microscope image sequence and invert it as a texture data block (pixel value range is 0 to 255, inverted: pixel value = 255 - original pixel value); read the corresponding neuron segmentation mask block and subcellular structure segmentation marker mask data;

[0058] In this embodiment, the neuron ID is first extracted from the neuron segmentation mask block: all non-zero ID values ​​are extracted from the neuron segmentation mask block by removing duplicate values ​​to form a set , where each ID represents a neuron object. For each neuron ID (denoted as ).

[0059] Then retain the neuron mask area in the 3D texture data, specifically:

[0060] Set the The voxel with the first ID value is 1, and the others are 0, thus creating the Neuron segmentation mask block with ID , the neuron segmentation mask block With texture data blocks Perform a dot multiplication operation to obtain the instanced texture data filtered by the mask , where each element is obtained by multiplying the mask by the value of the corresponding position of the three-dimensional data in the texture data block:

[0061] ;

[0062] In the instanced texture data of mask filtering, the pixel values ​​of the target neuron are retained with the cell membrane of the neuron as the boundary, and the pixel values ​​outside the target neuron are set to 0. This operation can separate the instanced texture data of the target neuron.

[0063] Finally, we overlay the marker (3D) on top of the mask-filtered instanced texture data:

[0064] Instanced texture data after filtering the mask Subcellular structures in the segmentation (specific structure segmentation inside the neuron segmentation object, such as synapses, mitochondria, etc.) are segmented according to the subcellular structure marker mask , take the intersection area of ​​the two for marking. If there is no subcellular structure to be marked, segment the subcellular structure and mark the mask All values ​​of are set to 0; if there are multiple subcellular structures to be labeled, the instantiated texture data after superimposing the current subcellular structure segmentation labeling mask will be superimposed with the next subcellular structure segmentation labeling mask in sequence until all objects are labeled. The labeling function can be different in each superposition labeling, that is, multiple different types of subcellular structures can be labeled on the same neuron.

[0065] Among them, the mark processing function is ,function This can be done by setting the corresponding pixel to red ( Figure 4 (b)) or other highlight operations (for example, multiply the original pixel value by 2, set the threshold to 255, and set the pixel value exceeding 255 after multiplication to 255):

[0066] ;

[0067] ;

[0068] in, For the The mask-filtered instanced texture data of neurons, Mask-filtered instanced texture data for overlay marking. is the three-dimensional coordinate of the texture data block voxel, is the marking function, is a mask for subcellular structures, For example, two examples are given Red mark or highlight mark, The brightness value is usually 255. The highlight coefficient is generally greater than 1.

[0069] from Figure 1 It can be seen from the four biological neural tissues that since there are too many segmented objects and they span multiple three-dimensional slices, it is difficult to directly observe and analyze the objects in the data. Therefore, this embodiment proposes an instantiated projection map method to visualize each segmented object.

[0070] In one embodiment, if three-dimensional instanced texture data needs to be projected from any direction, the data needs to be redirected through spatial rotation to align the projection direction with the coordinate axis, thereby completing the projection operation along the new axis. That is, the process of generating an instanced projection image of the instanced texture data with superimposed marks on an arbitrary angle plane is as follows:

[0071] (a1) Data rotation;

[0072] The instanced texture data of the mask filter after superimposing the markers is , any rotation angle is expressed as a combination of Euler angles around three axes , the rotation matrix is Represents the Euler rotation operation in three-dimensional space, which can be decomposed into 、 、 Rotation matrix product about the axis:

[0073] ; ; ; ;

[0074] For each voxel coordinate in the rotated data block , first convert it to physical coordinates, and then use the inverse rotation matrix Map it back to the physical coordinate system, then map the rotated physical coordinates back to the original voxel coordinate space, and finally use the value of the position in the original 3D volume electron microscope data to perform interpolation calculation to obtain the rotated volume data value;

[0075] ;

[0076] ;

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] in, Represents the voxel coordinates in the data block after rotation The corresponding physical coordinates (three-dimensional), Indicates that the inverse rotation matrix Will The coordinates mapped back to the physical coordinate system, 、 、 They are The three-axis coordinates represented by 、 、 They are 、 、 the voxel spacing of the axis (physical voxel size), Represents an interpolation function for non-integer coordinates Get the data value.

[0082] (a2) Projection after rotation;

[0083] After rotation, the size and form of the data block remain unchanged, the value comes from the interpolation result, and the coordinates become , the three orthogonal directions of the data block are no longer along the original direction, X, Y, and Z become 、 、 At this time in the orthogonal plane In 、 and The planar projection is as follows:

[0084] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale;

[0085] ;

[0086] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale:

[0087] ;

[0088] exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale;

[0089] ;

[0090] in, for instantiated projection graph on a plane, for instantiated projection graph on a plane, for An instanced projection graph on a plane.

[0091] Projection calculation for instanced volume data , designed to fully utilize the space where the instanced texture data of the mask filter is located The texture and label information of the image are used to highlight the visual features of interest. Different projection functions can be used. Taking plane projection as an example, the implementation methods include but are not limited to the following:

[0092] (b1) Maximum projection:

[0093] ;

[0094] This is the space where the instanced texture data for mask filtering is located, which is used to highlight high-intensity signals, such as myelin sheath, lysosome and other textures.

[0095] (b2) Minimum projection:

[0096] ;

[0097] Suitable for reflecting brightness structures such as cavitation, low-density areas, etc.

[0098] (b3) Standard deviation projection:

[0099]

[0100] Used to highlight areas with drastic signal changes and enhance boundary and texture contrast.

[0101] (b4) Median projection:

[0102]

[0103] It is not sensitive to noise and is suitable for analysis of relatively stable tissue structures.

[0104] It should be noted that these projection methods, like the summation projection method described above, are computationally efficient, can achieve perspective effects, and can be accelerated using a GPU. All projection results can be normalized to the grayscale range of [0, 255] to meet the requirements of conventional image display.

[0105] Preferably, the conventional orthogonal directions When projecting 3D instanced texture data, summation and normalization are performed along a certain coordinate axis. That is, the instanced projection map of the instanced texture data with superimposed markers on three orthogonal planes is generated as follows:

[0106] Projecting on the XY orthogonal plane, summing the mask-filtered instantiated texture data along the Z direction, and obtaining an instantiated projection image on the XY orthogonal plane, so that the boundaries of all slices in the original 3D volume electron microscopy data in the X and Y directions are preserved, while the thickness and texture in the Z direction can be displayed in grayscale;

[0107] ;

[0108] in, It is an instantiated projection image on the XY orthogonal plane.

[0109] Projecting on the XZ orthogonal plane, summing the mask-filtered instantiated texture data along the Y direction, and obtaining an instantiated projection image on the XZ orthogonal plane, so that the boundaries of all slices in the original 3D volume electron microscopy data in the X and Z directions are preserved, while the thickness and texture in the Y direction can be displayed in grayscale;

[0110] ;

[0111] in, It is an instantiated projection image on the XZ orthogonal plane.

[0112] Projecting on the YZ orthogonal plane, summing the mask-filtered instantiated texture data along the X direction, and obtaining an instantiated projection image on the YZ orthogonal plane, so that the boundaries of all slices in the original 3D volume electron microscopy data in the Y and Z directions are preserved, while the thickness and texture in the X direction can be displayed in grayscale;

[0113] ;

[0114] in, It is an instanced projection image on the YZ orthogonal plane.

[0115] According to the above projection method, the conversion from three-dimensional data to two-dimensional data is realized. See the schematic diagram Figure 3 A, by combining the segmented object in the volume electron microscope data with the electron microscope image and projecting it along multiple directions to obtain an instantiated projection image. The instantiated projection image designed in this embodiment retains the boundary of the object and retains the texture features of the object to a certain extent ( Figure 3 B. Since the electron microscope image is used in the projection, if the mitochondrial mask is set, the mitochondrial area will appear red or highlighted, and the mitochondrial texture can be observed in the instantiated projection image (mitochondrial mask is not set).

[0116] like Figure 4 As shown, (a), (b), and (c) are three practical variants of the segmentation projection map, which are respectively projection using only segmentation (segmentation projection), projection using neuron segmentation and then superimposing synaptic segmentation (red) (marker projection), and projection using segmentation + texture dot product (texture projection). Figure 4 It can be clearly seen in.

[0117] It should be noted that if the texture is not multiplied, a simple morphological projection can also be obtained by directly projecting the neuron segmentation mask block, see Figure 4 (a) is just the morphological projection image obtained in this way, which lacks some texture features.

[0118] Normalize the instantiated projection images on each orthogonal plane obtained above to the range of 0 to 255:

[0119] ;

[0120] in, Represents the instantiation projection on the current orthogonal plane, specifically corresponding to 、 、 ,

[0121] It can be understood that when XYZ are directly orthogonal planes, then Can be directly corresponded to 、 、 ,thus according to Specific reference 、 or Corresponding respectively 、 or .

[0122] In one embodiment, step three is specifically as follows:

[0123] Respectively Plane projection, Plane projection and The normalized projection images on the plane projection are scaled to match the actual physical space ratio;

[0124] ;

[0125] in, For The projection image after the scaling operation. refers to the difference function, is the anisotropic acquisition coefficient, They are 、 or , that is, when for hour, Pick ,when for hour, Pick ,when for hour, Pick . They are 、 or , that is, when for hour, Pick ,when for hour, Pick ,when for hour, Pick , and Different parameters are not taken at the same time, because It is composed of planes, so and Represents the same plane, and similarly for other planes, it finally corresponds to three planes, namely, Corresponding respectively 、 or , Corresponding respectively 、 or .

[0126] When XYZ are directly orthogonal planes, XY usually has the same resolution on the electron microscope image according to the acquisition resolution, and there may be differences in the Z direction. In many cases, the single layer thickness, that is, the resolution value in the Z direction is greater than that in the X and Y directions. The *Z plane projection is scaled. Can be X or Y to make it consistent with the actual physical space ratio:

[0127] ;

[0128] in, For The projection image after the telescopic operation, refers to the difference function, is the anisotropic acquisition coefficient (e.g. the ratio of the Z-direction resolution to the X or Y-direction resolution), They are or , that is, when When X, Pick ,when When Y, Pick In addition, this embodiment does not exclude stretching or scaling of the instantiated projection image on the XY orthogonal plane.

[0129] After completing the view space ratio correction, there are two options. One is to store the three corrected views as three channels of the image and fill the blank areas of the image size differences with 0 values. The other is to combine the three scaled projected images into a composite view. middle( Figure 3 B):

[0130] ;

[0131] in, 、 and Respectively represent 、 and Directional length (number of voxels), 、 、 This is the instantiated projection image after spatial scale correction.

[0132] Preferably, when XYZ are directly orthogonal planes, the three projected images after the scaling operation are combined into a comprehensive view middle( Figure 3 B)

[0133] ;

[0134] in, 、 and Represents the length (number of voxels) in the X, Y, and Z directions respectively.

[0135] In the above formula It refers to the instantiation projection diagram on the normalized XY orthogonal plane. and This is the instantiated projection image after spatial scale correction.

[0136] Save the comprehensive view as an image file. The file name contains the corresponding neuron ID, which is convenient for subsequent neuron indexing.

[0137] In this embodiment, compared with operations such as 3D rendering, the instantiated projection algorithm is simple. Except for extracting the ID in step 1, all other operations are simple matrix batch operations such as matrix dot product and summation. Therefore, GPU can be used to accelerate matrix operations, thereby quickly generating a large number of instantiated projection images. Figure 3 Views can be expanded to multiple views, effectively improving the accuracy of structural descriptions. Similar to how CT uses multiple viewpoints to reconstruct 3D data, 3D data can also be reconstructed using instantiated projections from multiple viewpoints, demonstrating that the description of structures using instantiated projections is stable and deterministic.

[0138] As an example:

[0139] See Figure 4 , taking a neural tissue texture data block as an example, based on the existing body electron microscopy data and the segmentation of all neuronal segments (segmentation can be obtained by manual annotation or automatic segmentation algorithm), the segmentation projection map can be obtained by calculating the projection of the simple segmentation mask, the projection of the segmentation mask and texture after dot multiplication, and the superposition of synaptic segmentation projection. Figure 4 (a)), Marked projection diagram ( Figure 4 (b)) and texture projection map ( Figure 4 (c)).

[0140] in, Figure 4The brightness and darkness of the segmentation projection in (a) show the thickness variation of the segmented object, with bright areas being thicker and dark areas being thinner. From three perspectives, the segmented object can be seen extending from the upper, deep, and left side of the data block to the lower, shallow, and left side. The object has multiple small branches, which are dendritic spines on the dendrites. Texture projection has more texture details than segmentation projection. Its brightness and darkness are no longer simply thickness information, but also include projections of the internal structure of the segmented object, so the projections of structures such as mitochondria can be distinguished ( Figure 3 B).

[0141] Figure 4 The (b) labeled projection image shows whether a synapse has formed on the dendrite. The red area is the projection of the postsynaptic region, which shows that multiple synapses have been formed on the dendritic spines and dendrites, but there is one dendritic spine that is suspected of not forming a postsynapse. This view can be used to analyze the relationship between neuronal fragments and neuronal substructures based on the existing neuronal object segmentation and to verify possible segmentation errors.

[0142] Figure 5 This example shows more instanced projections of segments from the neural tissue data block. (a), (b), (c), and (d) from left to right are computed texture projections of multiple segmented objects (with superimposed subcellular structures labeled) from the same texture data block. The segmented objects can be observed as belonging to dendritic forks, dendritic trunks, axon terminals, and thin axon terminals, respectively. These segments belong to different locations on the neuron. This example demonstrates that instanced projections retain sufficient morphological features to distinguish the neuronal location of the segmented object. This is often impossible to determine using a single electron microscopy image, as the field of view is too small and requires observation of multiple sections. This can also sometimes lead to misidentification due to the lack of 3D information. Once these segmented objects are observed and distinguished, the instanced projections can be used to locate the desired object (retrieval) from a large number of neuronal segmentations. For example, to extract a larger structure for further analysis, the dendritic trunk can be selected. To examine the connections between neuronal terminals, the axon terminals and thin axon terminals can be selected for further analysis.

[0143] After superimposing the subcellular structure markers, they can be inspected and verified. For example, subcellular structure markers derived from manual or automated segmentation methods may contain errors. According to common sense in the field, axon terminals may have synapses. If a synaptic marker appears on them, the marker is likely correct. Conversely, if it appears on other segments, it may be incorrect.

[0144] It is often difficult to make similar judgments using a single electron microscope image, while three-dimensional rendering is more complex, software-dependent, and prone to occlusion (for example, when 3D rendering a single neuron, the subcellular structures on its back or inside cannot be observed from a single perspective). Figure 5 This demonstrates that the method of this embodiment can display the morphological and texture characteristics of different neuronal segments in instantiated projection images. The instantiated projection images generated by this embodiment are highly efficient and require minimal data. The instantiated projection images of all objects can be slightly compressed and stored as retrieval thumbnails, enabling rapid retrieval of objects of interest from a 3D data block.

[0145] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A method for visualizing volume electron microscopy data by generating instantiated projections, characterized in that: include: Multiplying the texture data block of the volume electron microscopy data by a segmentation mask block corresponding to the volume electron microscopy data to obtain mask-filtered instantiated texture data, and superimposing the marked subcellular structure on the mask-filtered instantiated texture data to obtain superimposed marked instantiated texture data; The instanced texture data of the superimposed marker on the plane of any angle is converted to the orthogonal plane by data rotation, and the instanced projection image on the current orthogonal plane is obtained and normalized using the projection function; The normalized projection image is synthesized after spatial scale correction to output the visualization result corresponding to the current ID.

2. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 1, wherein: The texture data block and the segmentation mask block have the same size and are aligned; Extract all non-zero ID values ​​from the segmentation mask block by removing duplicate values ​​to form a set , where the set Each ID represents an instance object; Set the The voxel with the first ID value is 1, and the others are 0, thus creating the The segmentation mask block of IDs, .

3. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 1, wherein: Overlay labeled subcellular structures on the mask-filtered instanced texture data, specifically: The intersection area between the mask-filtered instantiated texture data and the set subcellular structure segmentation mark mask is obtained and marked; If there is no subcellular structure to be marked, all values ​​of the subcellular structure segmentation mark mask are set to 0; If there are multiple subcellular structures to be marked, the instantiated texture data after superimposing the current subcellular structure segmentation mark mask will be superimposed on the next subcellular structure segmentation mark mask in sequence.

4. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 3, wherein: Objects are marked by marking functions. When marking multiple objects, each marking function is independent. The marking functions are as follows: ； ； in, For the The mask-filtered instanced texture data for each object. For instanced texture data with mask filtering after overlay marking, is the 3D coordinate of the voxel in the texture data block, is the marking function, is a mask for subcellular structures, It's a red mark. To highlight, is the marker brightness value, is the highlight coefficient.

5. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 1, wherein: In the process of converting the instanced texture data of the superimposed mark on the arbitrary angle plane into the orthogonal plane by data rotation, the orthogonal plane is 、 and flat; Using the projection function, we can get the instantiation projection image on the current orthogonal plane, specifically: exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale; exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale; exist Plane projection, along Direction sums the mask-filtered instanced texture data to get The instantiation projection on the orthogonal plane makes all slices in the original 3D electron microscope data and The boundaries of the directions are preserved, and Directional thickness and texture are displayed in grayscale.

6. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 5, wherein: The normalized projection image is corrected by spatial scale, specifically: right Plane projection, Plane projection and The normalized projection images on the plane projection are scaled to match the actual physical space ratio.

7. The method for visualizing volume electron microscopy data by generating instantiated projections according to claim 1, wherein: Use GPU to accelerate the calculation of volume electron microscopy data visualization method.

8. A volume electron microscope data visualization device for generating instantiated projections, characterized in that: It includes data preprocessing module, projection map calculation module and visualization module; The data preprocessing module is used to perform a dot product of a texture data block of the volume electron microscopy data and a segmentation mask block corresponding to the volume electron microscopy data to obtain instantiated texture data filtered by the mask, and to superimpose a marked subcellular structure on the instantiated texture data filtered by the mask to obtain an instantiation of the superimposed mark; The projection map calculation module is used to convert the instanced texture data of the superimposed mark on the plane of any angle into an orthogonal plane by data rotation, and use the projection function to obtain the instanced projection map on the current orthogonal plane and normalize it; The visualization module is used to synthesize the normalized projection image after spatial scale correction, thereby outputting the visualization result corresponding to the current ID.

9. The device for visualizing volume electron microscope data for generating instantiated projections according to claim 8, characterized in that: The data preprocessing module includes a mask superposition module, a first marking module and a second marking module; The mask superposition module is used to obtain and mark the intersection area between the instantiated texture data filtered by the mask and the set subcellular structure segmentation mark mask; If there is no subcellular structure to be marked, all values ​​of the subcellular structure segmentation mark mask are set to 0 through the first marking module; If there are multiple subcellular structures to be marked, the instantiated texture data after superimposing the current subcellular structure segmentation mark mask is sequentially superimposed with the next subcellular structure segmentation mark mask through the second marking module.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a plurality of classification programs, which are used to be called by a processor and execute the volume electron microscope data visualization method according to any one of claims 1 to 7.

Images