Body electron microscope data visualization method and device for generating instantiated projection and storage medium

Abstract

The invention discloses a body electron microscope data visualization method and device for generating instantiated projection, and a storage medium, and relates to the technical field of image processing, and the method comprises the steps: carrying out the dot product of a texture data block of body electron microscope data and a segmentation mask block corresponding to the body electron microscope data, obtaining instantiated texture data filtered by a mask, and carrying out the instantiated texture data; superposing and marking a subcellular structure on the instantiated texture data filtered by the mask to obtain instantiated texture data with superposed and marked subcellular structures; the instantiated texture data with the superimposed marks on the plane at any angle are converted into an orthogonal plane through data rotation, and an instantiated projection drawing on the current orthogonal plane is obtained and normalized by using a projection function; the normalized projection drawing is synthesized after being subjected to space proportion correction, so that a visualization result corresponding to the current ID is output; according to the body electron microscope data visualization method, the visualization formats of forms and textures contained in multiple objects in the body electron microscope data can be rapidly generated, and rapid retrieval, verification and analysis of the body electron microscope data are achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a method, apparatus, and storage medium for visualizing volume electron microscopy data by generating instantiated projections. Background Art

[0002] The image sequence of nerve tissue obtained by electron microscopy technology has high resolution but a large amount of data. How to effectively extract and visualize neuron structure information from the segmentation data is a key technical challenge.

[0003] Browsing or visualizing volume electron microscopy data requires traversing all layers or using three-dimensional rendering to display. Taking sequential images as an example, one picture is one layer, and the three-dimensional structure in volume electron microscopy data often spans multiple layers. Therefore, if you need to browse the structural object, you need to traverse all the slices where the structure exists. Taking the electron microscopy imaging of biological tissues such as nerve cells as an example, the cell structures have too large a span in volume electron microscopy data, so only small data blocks can be read for browsing. At this time, all layers of the small data block often need to be accessed, resulting in low efficiency.

[0004] Another method using three-dimensional rendering can generate a table grid for rendering the surface of each object in the data block through an algorithm. However, this grid generation algorithm usually needs to be specifically designed to handle different surface topographies, and different results will be produced depending on the rendering tool and rendering settings. Therefore, it cannot be simply implemented for wide application. And, rendering according to the surface grid will completely lose the image texture, which is very important for understanding and subsequent processing of data objects.

[0005] Generally speaking, the prior art lacks an efficient and intuitive visualization processing method for multiple objects existing in volume electron microscopy data. Summary of the Invention

[0006] Based on the technical problems existing in the background art, the present invention proposes a method, apparatus, and storage medium for visualizing volume electron microscopy data by generating instantiated projections, which can quickly generate a visualization format including morphology and texture of multiple objects in volume electron microscopy data, and realize rapid retrieval, verification, and analysis of volume electron microscopy data.

[0007] The method for visualizing volume electron microscopy data by generating instantiated projections proposed by the present invention includes: Multiplying the texture data block of the volume electron microscopy data and the corresponding segmentation mask block of the volume electron microscopy data to obtain masked-filtered instantiated texture data, and superimposing a marked subcellular structure on the masked-filtered instantiated texture data to obtain superimposed-marked instantiated texture data; Converting the superimposed-marked instantiated texture data on an arbitrary angle plane to an orthogonal plane through data rotation, and using a projection function to obtain an instantiated projection map on the current orthogonal plane and normalizing it; The normalized projection images are synthesized after spatial ratio correction, so as to output the visualization result corresponding to the current ID.

[0008] Furthermore, the texture data block and the segmentation mask block have the same size and are aligned; All non-zero ID values are extracted from the segmentation mask block by the method of removing duplicate values to form a set , where the set each ID in represents an instance object; Set the voxels with the th ID value to 1 and the others to 0, so as to create the segmentation mask block of the th ID, .

[0009] Furthermore, marker subcellular structures are superimposed on the masked filtered instantiated texture data, specifically: Take the intersection area between the masked filtered instantiated texture data and the set subcellular structure segmentation marker mask and mark it; If there is no subcellular structure to be marked, set all the values of the subcellular structure segmentation marker mask to 0; If there are multiple subcellular structures to be marked, superimpose the instantiated texture data after superimposing the current subcellular structure segmentation marker mask with the next subcellular structure segmentation marker mask in sequence.

[0010] Furthermore, objects are marked by a marking function. When multiple objects are marked, each marking function is independent. The marking function is as follows: ; ; Among them, is the masked filtered instantiated texture data of the th object, is the masked filtered instantiated texture data after superimposing the mark, is the three-dimensional coordinate of the voxel in the texture data block, is the marking function, is the mask of the subcellular structure, For example, is a red mark, is a highlighting mark, is the marking brightness value which is often 255, is the highlighting coefficient.

[0011] Furthermore, when converting the instantiated texture data with superimposed marks on an arbitrary angle plane to an orthogonal plane through data rotation, the orthogonal plane includes , and Plane; Using a projection function, a projection map on the current orthogonal plane is obtained, specifically: In plane projection, sum the masked-filtered instantiated texture data along direction to obtain an instantiated projection map on the orthogonal plane, so that the boundaries on all slices in the original three-dimensional electron microscopy data in and directions are retained, while the thickness and texture in the direction are displayed in gray scale; In plane projection, sum the masked-filtered instantiated texture data along direction to obtain an instantiated projection map on the orthogonal plane, so that the boundaries on all slices in the original three-dimensional electron microscopy data in and directions are retained, while the thickness and texture in the direction are displayed in gray scale; In plane projection, sum the masked-filtered instantiated texture data along direction to obtain an instantiated projection map on the orthogonal plane, so that the boundaries on all slices in the original three-dimensional electron microscopy data in and directions are retained, while the thickness and texture in the direction are displayed in gray scale.

[0012] Furthermore, perform spatial scale correction on the normalized projection map, specifically: For plane projection, plane projection, and the projection maps on the plane projection are respectively scaled to match the actual physical space scale.

[0013] Furthermore, use the GPU to accelerate the operation of the three-dimensional electron microscopy data visualization method.

[0014] A three-dimensional electron microscopy data visualization device for generating an instantiated projection includes a data preprocessing module, a projection map calculation module, and a visualization module; The data preprocessing module is used to multiply the texture data block of the three-dimensional electron microscopy data and the segmentation mask block corresponding to the three-dimensional electron microscopy data to obtain masked-filtered instantiated texture data, and superimpose a marked subcellular structure on the masked-filtered instantiated texture data to obtain superimposed-marked instantiated texture data; The projection map calculation module is used to convert the instantiated texture data of the superimposed markers on an arbitrary - angle plane into an orthogonal plane through data rotation, and use the projection function to obtain the projection map on the current orthogonal plane and normalize it; The visualization module is used to synthesize the normalized projection map after spatial - scale correction, so as to output the visualization result corresponding to the current ID.

[0015] Furthermore, the data pre - processing module includes a mask - overlay module, a first marking module, and a second marking module; The mask - overlay module is used to take the intersection area between the instantiated texture data filtered by the mask and the set sub - cellular structure segmentation marker mask and mark it; If there is no sub - cellular structure to be marked, the first marking module sets all the values of the sub - cellular structure segmentation marker mask to 0; If there are multiple sub - cellular structures to be marked, the second marking module sequentially superimposes the next sub - cellular structure segmentation marker mask on the instantiated texture data after superimposing the current sub - cellular structure segmentation marker mask.

[0016] A computer - readable storage medium stores a number of classification programs, and the number of classification programs is used to be called by a processor and execute the volume electron microscopy data visualization method as described above.

[0017] The advantages of the volume electron microscopy data visualization method, device, and storage medium for generating instantiated projections provided by the present invention are as follows: It can quickly generate a visualization format of multiple objects including morphology and texture in volume electron microscopy data, realizing the rapid retrieval, verification, and analysis of volume electron microscopy data. Moreover, the instantiated projection map in this embodiment contains structural and texture projection information, which is close to the visual effects of X - rays and CT imaging, providing an intuitive representation of neuron morphology and helping researchers quickly understand complex three - dimensional structures. The instantiated projection map can calculate the projections of multiple objects simultaneously. In addition, it can combine sub - cellular structures (such as synapses and mitochondria), realizing the resolution of the positions, morphologies, and spatial relationships of multiple three - dimensional objects in the instantiated projection map. It can be applied to volume electron microscopy data and segmentation results from different sources, supporting the visualization requirements of various neuron morphologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of the present invention; Figure 2 It is a schematic diagram of input data display. (a) is the volume electron microscopy data of biological neural tissue, (b) is the segmentation mask, (c) is the rendering of a small number of objects, and (d) is a single object in the slice; Figure 3To instantiate the projection map and the three-view drawings, where A is the instantiated projection map obtained by projecting the segmented objects and electron microscopy images in the volume electron microscopy data along multiple directions; B is the three-view drawing. The general display form of the instantiated projection map is the three-view drawing form. Since electron microscopy images are used in the projection, mitochondrial textures can be observed in the instantiated projection map (without setting a mitochondrial mask); Figure 4 Schematic diagrams of three practical variants of the segmented projection map; (a) is the segmented projection map using only segmentation for projection, (b) is the labeled projection map using neuron segmentation projection and then superimposing synaptic segmentation (red), and (c) is the texture projection map using segmentation and texture dot product for projection; Figure 5 Instantiated projection maps of multiple objects in the same region; (a) is the projection map of dendritic bifurcations, (b) is the projection map of dendritic trunks, (c) is the projection map of axon terminals, and (d) is the projection map of thin axon terminals. Detailed implementation manners

[0019] Next, the technical solutions of the present invention will be described in detail through specific embodiments. Many specific details are set forth in the following description in order to fully understand the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the connotation of the present invention. Therefore, the present invention is not limited by the specific implementations disclosed below.

[0020] In this embodiment, there are some professional terms, for example: volume electron microscopy data → sequence images, data blocks. Each of the sequence images is a layer, and when combined, they form a volume block; the three-dimensional structure in the volume electron microscopy data → object, segmented object. The volume electron microscopy data mentioned in this embodiment often contains multiple three-dimensional structure objects, and each object may span the data block at any angle.

[0021] As Figures 1 to 5 shown, the method for visualizing volume electron microscopy data for generating instantiated projections proposed by the present invention includes the following steps: Step 1: Multiply the texture data block of the volume electron microscopy data and the corresponding segmentation mask block of the volume electron microscopy data to obtain the instantiated texture data filtered by the mask, and superimpose the labeled subcellular structures on the instantiated texture data filtered by the mask to obtain the instantiated texture data with labels superimposed; Step 2: Convert the instantiated texture data with labels superimposed on any angle plane into an orthogonal plane through data rotation, and use the projection function to obtain the instantiated projection map on the current orthogonal plane and normalize it; Step 3: Synthesize the normalized projection map after spatial scale correction, so as to output the visualization result corresponding to the current ID.

[0022] According to Steps 1 to 3, it is possible to quickly generate a visual format of the morphology and texture of multiple objects in volume electron microscopy data, realizing the rapid retrieval, verification, and analysis of volume electron microscopy data. Moreover, the instantiated projection map in this embodiment contains structural and texture projection information, which is close to the visual effects of X-ray and CT imaging, providing an intuitive representation of neuron morphology and helping researchers quickly understand complex three-dimensional structures. The instantiated projection map can calculate the projections of multiple objects simultaneously. Additionally, it can combine subcellular structures (such as synapses and mitochondria), enabling the resolution of the positions, morphologies, and spatial relationships of multiple three-dimensional objects in the instantiated projection map. It can be applied to volume electron microscopy data and segmentation results from different sources, supporting the visualization requirements of various neuron morphologies.

[0023] It should be noted that the volume electron microscopy data visualization method in this embodiment can be used for neurons or other structural objects such as glial cells and muscle cells that densely occupy the three-dimensional space of tissues, thereby realizing visualization of neuron morphology or type cell morphology in nerve tissues. The following will use neurons for illustration, and other types of cells can be obtained directly in a similar manner.

[0024] In one embodiment, Step 1 is specifically as follows: Texture data block ( Figure 2 of (a)): A three-dimensional data block composed of an electron microscopy image sequence, where the gray value of each pixel represents the brightness of the electron microscopy imaging of nerve tissue. Due to the heavy metal staining process in the sample preparation process, regions containing lipids such as cell membranes and organelle membranes have a higher brightness, where is the three-dimensional data coordinate.

[0025] Neuron segmentation mask block ( Figure 2 of (b)): A three-dimensional data block with the same size as the texture data block, where each neuron object is represented by a unique ID, and the background is usually set to 0. Among them, the texture data block and the segmentation mask block have the same size and are aligned. This mask can be generated by manual annotation or existing automatic segmentation algorithms.

[0026] Subcellular structure segmentation label mask : A three-dimensional data block with the same size as the texture data block, where the background is usually set to 0, and the subcellular structures to be labeled such as mitochondria and synapses have non-zero values. A type of subcellular structure is often represented by an ID (for example, later, a single subcellular structure label will be used as an example, with ID = 1); this mask can be generated by manual annotation or existing automatic segmentation algorithms.

[0027] The specific process of loading electron microscopy data is as follows: Read the three-dimensional electron microscopy image sequence, take the inverse as the texture data block (the pixel value range is 0 to 255, take the inverse: pixel value = 255 - original pixel value); Read the corresponding neuron segmentation mask block and the data of the subcellular structure segmentation marker mask; In this embodiment, first, extract the neuron ID from the neuron segmentation mask block: Extract all non-zero ID values from the neuron segmentation mask block by the method of deleting duplicate values to form a set , and each ID in it represents a neuron object. For each neuron ID (denoted as ).

[0028] Then, retain the neuron mask area in the three-dimensional texture data. Specifically: Set the voxels with the th ID value to 1, and the others to 0, so as to create the neuron segmentation mask block with the th ID , multiply the neuron segmentation mask block with the texture data block to perform a dot product operation to obtain the masked and filtered instantiated texture data , where each element is obtained by multiplying the mask with the value at the corresponding position in the three-dimensional data of the texture data block: ; In the masked and filtered instantiated texture data, with the cell membrane of the neuron as the boundary, the pixel values of the target neuron are retained, and the pixel values of the points outside the target neuron are 0. This operation can separate the instantiated texture data of the target neuron.

[0029] Finally, overlay the marks (three-dimensional) on the masked and filtered instantiated texture data: For the subcellular structures (specific structure segmentation inside the neuron segmentation object, such as synapses, mitochondria, etc.) in the masked and filtered instantiated texture data , according to the subcellular structure segmentation marker mask , perform a marking process on the intersection area of the two. If there is no subcellular structure to be marked, set all the values of the subcellular structure segmentation marker mask to 0; if there are multiple subcellular structures to be marked, overlay the instantiated texture data after overlaying the current subcellular structure segmentation marker mask with the next subcellular structure segmentation marker mask in sequence until all object markings are completed. In each overlay marking, the marking function can be different, that is, multiple different types of subcellular structures can be marked on the same neuron.

[0030] Among them, the marking process function is , and the function can be to set the corresponding pixel to red (Figure 4 of (b)) or other highlighting operations (such as multiplying the original pixel value by 2, setting the threshold to 255, and setting values exceeding 255 after multiplying by 2 to 255): ; ; wherein, is the instantiated texture data of the th neuron after mask filtering, is the instantiated texture data after mask filtering with superimposed markers, is the three - dimensional coordinates of the voxel of the texture data block, is the marking function, is the mask of the sub - cellular structure, For example, two examples for illustration are a red marker or a highlighting marker respectively, is that the marker brightness value is often 255, and the highlighting coefficient is generally greater than 1.

[0031] From Figure 1 the four biological nerve tissues, it can be seen that since there are too many segmentation objects and they span multiple three - dimensional slices, it is difficult to directly observe and analyze the objects in the data. Therefore, this embodiment proposes an instantiated projection map method to visualize each segmentation object.

[0032] In one of the embodiments, if it is necessary to project three - dimensional instantiated texture data from any direction, data redirection needs to be achieved through spatial rotation to align the projection direction with the coordinate axes, so as to complete the projection operation along the new axis. That is, the generation process of the instantiated projection map of the instantiated texture data with superimposed markers on an arbitrary - angle plane is as follows: (a1) Data rotation; The instantiated texture data after mask filtering with superimposed markers is , and the arbitrary rotation angle is represented as a combination of Euler angles around three axes , and the rotation matrix is represents the Euler rotation operation in three - dimensional space, which can be decomposed into the product of rotation matrices around , , axes: ; ; ; ; For each voxel coordinate in the rotated data block, first convert it to physical coordinates, and then through the inverse rotation matrix Map it back to the physical coordinate system, then map the rotated physical coordinates back to the original voxel coordinate space, and finally perform interpolation calculations using the value at this position in the original three-dimensional electron microscopy data to obtain the rotated volume data value; ; ; ; ; ; ; Among them, represents the physical coordinates (three-dimensional) corresponding to the voxel coordinates in the rotated data block , represents the coordinates mapped back to the physical coordinate system through the inverse rotation matrix of , , , are respectively the three-axis coordinates represented by , , are respectively , , the voxel spacings (voxel physical sizes) of the axes, represents the interpolation function used to obtain the data value at the non-integer coordinate .

[0033] (a2) Rotated projection; The size and form of the rotated data block remain unchanged, and the numerical values come from the interpolation results. The coordinates become , and the three orthogonal directions of the data block no longer follow the original directions. X, Y, and Z become , , . At this time, in the orthogonal plane , project onto , and planes as follows: In the projection on the plane, sum the instantiated texture data filtered by the mask along the direction to obtain the instantiated projection map on the orthogonal plane, so that the boundaries in the and directions on all slices of the original three-dimensional electron microscopy data are retained, while the thickness and texture in the direction are displayed in gray scale; ; In a planar projection, sum the instantiated texture data filtered by a mask along a direction to obtain an instantiated projection map on an orthogonal plane, such that the boundaries on all slices in the original three-dimensional electron microscopy data in and directions are retained, while the thickness and texture in the direction are displayed in grayscale: ; In a planar projection, sum the instantiated texture data filtered by a mask along a direction to obtain an instantiated projection map on an orthogonal plane, such that the boundaries on all slices in the original three-dimensional electron microscopy data in and directions are retained, while the thickness and texture in the direction are displayed in grayscale; ; wherein, is an instantiated projection map on the plane, is an instantiated projection map on the plane.

[0034] Projection calculation for the instantiated volume data , aiming to make full use of the texture and marking information in the space where the instantiated texture data filtered by a mask is located, highlight the visualized features of interest, and different forms of projection functions can be adopted. Taking the projection of the rotated instantiated texture data onto the plane as an example, the implementation methods include but are not limited to the following: (b1) Maximum projection: ; is the space where the instantiated texture data filtered by a mask is located, used to highlight high-intensity signals, such as textures of myelin sheaths, lysosomes, etc.

[0035] (b2) Minimum projection: ; is applicable to inverse-display brightness structures, such as vacuoles, low-density regions, etc.

[0036] (b3) Standard deviation projection:

[0037] Used to highlight areas with drastic signal changes and enhance boundary and texture contrast.

[0038] (b4) Median projection:

[0039] Insensitive to noise and suitable for the analysis of relatively stable tissue structures.

[0040] It should be noted that these projection methods, like the summation projection method described above, have the characteristics of high-efficiency calculation, can all achieve a perspective effect, and can be accelerated by GPU. All projection results can be mapped to the [0, 255] gray range through a normalization operation to meet the requirements of conventional image display.

[0041] Preferably, the conventional orthogonal direction When performing three-dimensional instantiated texture data projection, sum and normalize along a certain coordinate axis. That is, the generation process of the instantiated projection maps of the superimposed marked instantiated texture data on three orthogonal planes is as follows: Project on the XY orthogonal plane, sum the masked and filtered instantiated texture data along the Z direction to obtain the instantiated projection map on the XY orthogonal plane, so that the boundaries in the X and Y directions on all slices of the original three-dimensional electron microscopy data are retained, and the thickness and texture in the Z direction are displayed in gray scale; ; Among them, is the instantiated projection map on the XY orthogonal plane.

[0042] Project on the XZ orthogonal plane, sum the masked and filtered instantiated texture data along the Y direction to obtain the instantiated projection map on the XZ orthogonal plane, so that the boundaries in the X and Z directions on all slices of the original three-dimensional electron microscopy data are retained, and the thickness and texture in the Y direction are displayed in gray scale; ; Among them, is the instantiated projection map on the XZ orthogonal plane.

[0043] Project on the YZ orthogonal plane, sum the masked and filtered instantiated texture data along the X direction to obtain the instantiated projection map on the YZ orthogonal plane, so that the boundaries in the Y and Z directions on all slices of the original three-dimensional electron microscopy data are retained, and the thickness and texture in the X direction are displayed in gray scale; ; Among them, is the instantiated projection map on the YZ orthogonal plane.

[0044] The conversion from three-dimensional data to two-dimensional data is achieved according to the above projection method. For a schematic diagram, see Figure 3 A, where an instantiated projection map is obtained by projecting the segmented objects and electron microscopy images in the volume electron microscopy data along multiple directions. The instantiated projection map designed in this embodiment retains the boundaries of the objects and, to a certain extent, retains the texture features of the objects ( Figure 3 B. Since electron microscopy images are used in the projection, if a mitochondrial mask is set, the mitochondrial region will appear red or highlighted, and mitochondrial texture can be observed in the instantiated projection map (when the mitochondrial mask is not set)).

[0045] As Figure 4 shown, where (a), (b), and (c) are three practical variants of the segmented projection map, namely projection using only segmentation (segmentation projection), projection using neuron segmentation followed by superposition of synaptic segmentation (red) (labeled projection), and projection using segmentation + texture dot product (texture projection). It can be clearly seen from Figure 4 .

[0046] It should be noted that if the texture is not multiplied, but the neuron segmentation mask block is directly projected, a simple morphological projection map can also be obtained. See Figure 4 a, but the morphological projection map obtained in this way lacks some texture features.

[0047] The instantiated projection maps obtained on each orthogonal plane are normalized to the range of 0 to 255: ; where represents the instantiated projection map on the current orthogonal plane, specifically corresponding to , , , It can be understood that when XYZ is directly the orthogonal plane, then can directly correspond to , , . Therefore, According to specifically refers to , or respectively corresponding to , or .

[0048] In one embodiment, step three is specifically as follows: Respectively for plane projection, plane projection, and The normalized projection diagrams on the planar projection are respectively subjected to stretching and shrinking operations to make them conform to the ratio of the actual physical space; ; where is the projection diagram after the stretching and shrinking operation on , refers to the difference function, is the anisotropic acquisition coefficient, are respectively , or , that is, when is , takes , when is , takes , when is , takes . are respectively , or , that is, when is , takes , when is , takes , when is , takes , and do not take the same parameters simultaneously. Since constitutes a plane, thus and represent the same plane. Similarly for other planes. Finally, it corresponds to three planes, that is, respectively correspond to , or , respectively correspond to , or .

[0049] When the XYZ is directly an orthogonal plane, according to the acquisition resolution, XY usually has the same resolution on the electron microscope image, and there may be differences in the Z direction. In many cases, the single-layer thickness, that is, the Z-direction resolution value, is greater than that in the X and Y directions. The *Z-plane projection is subjected to stretching and shrinking operations, where It can be X or Y to match the actual physical space ratio: ; Among them, is the projected image after the scaling operation on , refers to the difference function, is the anisotropic acquisition coefficient (e.g., the ratio of the resolution in the Z direction to the resolution in the X or Y direction), are respectively or , that is, when is X, takes , when is Y, takes . In addition, this embodiment does not exclude stretching or scaling the instantiated projected image on the XY orthogonal plane.

[0050] After completing the view space ratio correction, there are two subsequent options. One is to store the three corrected views as the three channels of the image and fill the blank areas with different sizes between the images with 0 values. The other way is to combine the three projected images after the scaling operation into a comprehensive view in ( Figure 3 B): ; Among them, , and respectively represent , and direction lengths (number of voxels), , , are the instantiated projected images after the space ratio correction.

[0051] Preferably, when XYZ is directly an orthogonal plane, combine the three projected images after the scaling operation into a comprehensive view in ( Figure 3 B) ; Among them, , and respectively represent the X, Y, and Z direction lengths (number of voxels).

[0052] In the above formula, refers to the instantiated projected image on the XY orthogonal plane after normalization, and are the instantiated projected images after the space ratio correction.

[0053] Save the comprehensive view as an image file with the corresponding neuron ID in the file name, which is convenient for subsequent indexing according to neurons.

[0054] In this embodiment, compared with operations such as 3D rendering, the instantiated projection graph algorithm is simple. Except for extracting the ID in step one, other operations are simple matrix batch operations such as dot product and summation of matrices. Therefore, the GPU can be used to accelerate matrix operations, thereby quickly generating a large number of instantiated projection graphs. The instantiated projection Figure 3 view can be extended to multi-views, thereby effectively improving the accuracy of structure description. Similar to CT using multiple perspectives to reconstruct 3D data, 3D data can also be reconstructed using instantiated projection graphs from multiple perspectives, indicating that the description of the structure by the instantiated projection graph is stable and definite.

[0055] As an embodiment: See Figure 4 , taking a texture data block of a neural tissue as an example. Based on the existing volume electron microscopy data and the segmentation of all neuron segments therein (the segmentation can be obtained through manual annotation or automatic segmentation algorithms), by calculating the projection of a simple segmentation mask, the projection after multiplying the segmentation mask and the texture, and superimposing the synaptic segmentation projection, the segmentation projection graph ( Figure 4 (a) of Figure 4 ), the labeled projection graph ( Figure 4 (b) of

[0056] Among them, Figure 4 the light and dark of the (a) segmentation projection graph in Figure 3 show the thickness change of the segmentation object. The bright areas are thick, and the dark areas are thin. From three perspectives, it can be seen that the segmentation object extends from above, deep inside, and left side of the data block to below, shallow layer, and left side. There are multiple small bifurcations on this object, which are dendritic spine structures on the dendrite. The texture projection has more texture details compared to the segmentation projection. Its light and dark are no longer simply thickness information, but contain the projection of the internal structure of the segmentation object. Therefore, the projection of structures such as mitochondria can be distinguished therein (

[0057] Figure 4 (b) of

[0058] Figure 5Shows the instantiated projection diagrams of more segments in the neural tissue data block. From left to right, (a), (b), (c), and (d) are the computed texture projection diagrams of multiple segmented objects (overlay-marked subcellular structures) in the same texture data block. It can be observed that the segmented objects belong to dendritic bifurcations, dendritic trunks, axon terminals, and thin axon terminals respectively. Among them, dendritic bifurcations, dendritic trunks, axon terminals, and thin axon terminals are all at different positions on the neuron. This example shows that sufficient morphological features are retained in the instantiated projection diagram, and it can be distinguished which part of the neuron the neuron segment belongs to. However, it is often impossible to judge through a single electron microscope image because the field of view of a single electron microscope image is too small, and multiple slices must be continuously observed. Sometimes, misjudgment may also occur due to the lack of three-dimensional information. After these segmented objects can be observed and distinguished, the instantiated projection diagram can be used to find the required objects (retrieval) from a large number of neuron segmentations. For example, if you want to extract larger structures for further analysis, you can select the dendritic trunk. If you want to see the connections between neuron terminals, you can select the axon terminals and thin axon terminals for further analysis.

[0059] After overlay-marking the subcellular structures, inspection and verification can be carried out. For example, the subcellular structure markings come from manual or automated segmentation methods, but there may be errors. According to the common knowledge in the field, synaptic distributions may exist at axon terminals. If synaptic markings appear on them, then these markings may be correct. On the contrary, if they appear on other segments, they may be incorrect.

[0060] It is often difficult to make similar judgments through a single electron microscope image, and three-dimensional rendering is relatively complex, relying on software, and it is also prone to occlusion (for example, when rendering a single neuron in 3D, the subcellular structures on its back or inside cannot be observed from a single perspective). Figure 5 Illustrates that the method of this embodiment can show the morphological and texture features of different neuron segments in the instantiated projection diagram. The instantiated projection diagram generated by the method of this embodiment is highly efficient and generates a small amount of data. The instantiated projection diagrams of all objects can be slightly compressed and stored as retrieval thumbnails, so as to quickly find the analysis objects of interest from the three-dimensional data block and achieve fast retrieval.

[0061] As mentioned above, it is only the preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent substitutions or changes, and should be covered by the protection scope of the present invention.

Claims

1. A method for visualizing volume electron microscope data for generating an instantiated projection, characterized in that Including: Multiplying the texture data block of the volume electron microscopy data and the corresponding segmentation mask block of the volume electron microscopy data to obtain the masked-filtered instantiated texture data, and superimposing and marking subcellular structures on the masked-filtered instantiated texture data to obtain the superimposed and marked instantiated texture data; Converting the superimposed and marked instantiated texture data on an arbitrary-angle plane into an orthogonal plane through data rotation, and using a projection function to obtain and normalize the instantiated projection map on the current orthogonal plane; Performing synthesis on the normalized projection map after spatial scale correction, so as to output the visualization result corresponding to the current ID.

2. The method for visualizing volume electron microscope data for generating an instantiated projection according to claim 1, wherein The texture data block and the segmentation mask block have the same size and are aligned; Extract all non-zero ID values from the segmentation mask block by removing duplicate values to form a set , where each ID in the set represents an instance object; Set the voxel with the th ID value to 1 and the others to 0, thereby creating the th ID's segmentation mask block, .

3. The method for visualizing volume electron microscope data for generating an instantiated projection according to claim 1, wherein Superimposing and marking subcellular structures on the masked-filtered instantiated texture data, specifically: Taking the intersection region between the masked-filtered instantiated texture data and the set subcellular structure segmentation marker mask and marking it; If there are no subcellular structures to be marked, setting all the values of the subcellular structure segmentation marker mask to 0; If there are multiple subcellular structures to be marked, sequentially superimposing the next subcellular structure segmentation marker mask on the instantiated texture data after superimposing the current subcellular structure segmentation marker mask.

4. The method for visualizing volume electron microscope data for generating an instantiated projection according to claim 3, wherein Marking an object through a marking function. When marking multiple objects, each marking function is independent. The marking function is as follows: ； ； Among them, is the instantiated texture data of the th object after mask filtering, is the instantiated texture data after mask filtering with superimposed marks, is the three-dimensional coordinates of the voxels within the texture data block, is the marking function, is the mask of the subcellular structure, For example, is a red mark, is a highlight mark, is the marking brightness value, is the highlight coefficient.

5. The method for visualizing volume electron microscope data for generating an instantiated projection according to claim 1, wherein In the conversion of instantiated texture data with superimposed markers on an arbitrarily angled plane to an orthogonal plane through data rotation, the orthogonal plane includes , and planes; Using a projection function to obtain the instantiated projection map on the current orthogonal plane, specifically: In a planar projection, summing the instantiated texture data filtered by a mask along a direction to obtain an instantiated projection map on an orthogonal plane, such that the boundaries in the and directions on all slices in the original three-dimensional electron microscope data are preserved, while the thickness and texture in the direction are displayed in grayscale; In a planar projection, summing the instantiated texture data filtered by a mask along a direction to obtain an instantiated projection map on an orthogonal plane, such that boundaries in all slices of the original three-dimensional electron microscope data in and directions are retained, while the thickness and texture in the direction are displayed in grayscale; In a planar projection, summing the instantiated texture data filtered by a mask along a direction to obtain an instantiated projection image on an orthogonal plane, such that the boundaries in the and directions on all slices in the original three-dimensional electron microscopy data are retained, while the thickness and texture in the direction are displayed in grayscale.

6. The method for visualizing volume electron microscope data for generating an instantiated projection according to claim 5, wherein Performing spatial scale correction on the normalized projection map, specifically: Perform respectively on planar projections, planar projections and the normalized projection diagrams on the planar projections stretching operations respectively, so as to conform to the actual physical space ratio.

7. The method for visualizing volume electron microscopy data for generating an instantiated projection according to claim 1, wherein Using a GPU to perform accelerated operations on the volume electron microscopy data visualization method.

8. A volume electron microscope data visualization device for generating an instantiated projection, characterized in that, Including a data preprocessing module, a projection map calculation module, and a visualization module; The data preprocessing module is used to multiply the texture data block of the volume electron microscopy data and the corresponding segmentation mask block of the volume electron microscopy data to obtain the masked-filtered instantiated texture data, and superimpose and mark subcellular structures on the masked-filtered instantiated texture data to obtain the superimposed and marked instantiation; The projection map calculation module is used to convert the superimposed and marked instantiated texture data on an arbitrary-angle plane into an orthogonal plane through data rotation, and use a projection function to obtain and normalize the instantiated projection map on the current orthogonal plane; The visualization module is used to perform synthesis on the normalized projection map after spatial scale correction, so as to output the visualization result corresponding to the current ID.

9. The volume electron microscope data visualization device for generating an instantiated projection according to claim 8, wherein The data preprocessing module includes a mask superimposing module, a first marking module, and a second marking module; The mask superimposing module is used to take the intersection region between the masked-filtered instantiated texture data and the set subcellular structure segmentation marker mask and mark it; If there are no subcellular structures to be marked, setting all the values of the subcellular structure segmentation marker mask to 0 through the first marking module; If there are multiple subcellular structures to be marked, sequentially superimposing the next subcellular structure segmentation marker mask on the instantiated texture data after superimposing the current subcellular structure segmentation marker mask through the second marking module.

10. A computer-readable storage medium, characterized in that, A number of classification programs are stored on the computer-readable storage medium, and the number of classification programs are used to be called by a processor and execute the volume electron microscopy data visualization method according to any one of claims 1 to 7.

Images