A cross-ratio three-dimensional image fusion visualization method and system

Abstract

The application discloses a kind of cross magnification three-dimensional image fusion visualization method and system, the method includes: obtaining the low magnification global three-dimensional image data and at least one high magnification local three-dimensional image data of same sample;With global image coordinates as unified reference space, local image is registered to global image, and forward and reverse mapping matrix and coverage are extracted;LOD pyramid based on four-tree division is constructed, and global and local image data and its each level down-sampling image are blocked, and image block index including data source, LOD level, equivalent step and coverage is established;Based on user field of view adaptive display, according to the equivalent voxel step of current field of view, the best image block is filtered, and the image block boundary is fused to maintain gray consistent and smooth transition.The application realizes seamless fusion and high-precision visualization of different magnification three-dimensional images in the same scene by registration and cross magnification LOD continuous connection, solves the positioning difficulty of local image in global space and cross magnification and the like.

Description

Technical Field

[0001] This invention belongs to the field of image enhancement, and more specifically, relates to a method and system for cross-magnification three-dimensional image fusion visualization. Background Technology

[0002] In biomedical microscopic observation and digital imaging, it is often difficult to obtain a full-field, high-resolution 3D image of the entire sample in a single operation, in order to simultaneously achieve a large field of view, imaging speed, and high-resolution detail acquisition. For biomedical samples such as tissue sections, cell samples, tissue microarrays, organ-on-a-chip, or microfluidic chips, low-magnification imaging can quickly obtain the overall structure and global spatial distribution, while high-magnification imaging can obtain local cellular, tissue microstructure, or pathological details. However, high-magnification imaging has a small field of view, long acquisition time, and large data volume. Therefore, in actual observation and acquisition, global 3D data is often obtained first through low-magnification imaging, followed by high-magnification local acquisition of regions of interest, lesion areas, or areas with significant signals, forming cross-magnification data where low-magnification global data and high-magnification local data coexist.

[0003] Different magnification data have different physical sampling intervals, spatial resolutions, coverage areas, and data sizes. Current methods for processing images at different magnifications typically involve separate browsing, switching between multiple magnification windows, or downsampling high-magnification images and registering them with low-magnification images. While these methods support a certain degree of multi-magnification observation, the global structure and high-magnification local details are difficult to comprehensively display within the same 3D viewpoint. This makes it impossible to directly observe and determine the location of high-magnification local images in global space, causing difficulties for users in identifying image content. Furthermore, hard stitching across magnification boundaries, inconsistencies in grayscale, and discontinuous data loading can still pose challenges to image content identification. Summary of the Invention

[0004] To address the aforementioned deficiencies or improvement needs of existing technologies, this invention achieves seamless fusion and high-precision visualization of 3D images at different magnifications in the same scene through registration and continuous LOD connection across magnifications. This solves problems such as the difficulty in locating local images in global space and inconsistent grayscale at cross-magnification boundaries.

[0005] To achieve the above objectives, according to one aspect of the present invention, a method for cross-magnification 3D image fusion visualization is provided, comprising the following steps: (1) For the same sample, the data source is obtained, which includes low-magnification global 3D image data and one or more local 3D image data with magnification higher than that of the global 3D image. The 3D image data of the data source is denoted as ; (2) Registering local 3D image data to global 3D image data: For the data source obtained in step (1) It uses the coordinate space of its global three-dimensional image data as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in ; (3) Constructing the LOD pyramid: Based on the registered 3D image data obtained in step (2), a block-based and layered pyramid data structure is constructed using a quadtree partitioning method. The global 3D image data and local 3D image data, as well as the downsampled images of each LOD layer, are organized into image blocks according to a preset block partitioning strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: ; (4) Adaptive display based on user field of view: Based on the initial field of view and the user's image transformation operation, obtain the display voxel range and equivalent voxel step size of the current observation field of view. For each voxel in the display voxel range of the current observation field of view, match the image block index obtained in step (3) to filter out the best image block and perform image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation and rotation.

[0006] Preferably, in the cross-magnification 3D image fusion visualization method, the coverage areas of the local 3D image data and the global 3D image data overlap. In the case of multiple local 3D image data, the local 3D image data can be independent of each other, or they can have overlapping, intersecting, or containing relationships.

[0007] Preferably, in the cross-magnification 3D image fusion visualization method, step (2) involves the forward mapping matrix. The specific extraction method is as follows:

[0008] in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization.

[0009] The reverse mapping matrix The specific extraction method is as follows: .

[0010] Preferably, in the cross-magnification 3D image fusion visualization method, step (2) involves selecting the magnification. Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. :

[0011] in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

[0012] Preferably, the cross-magnification 3D image fusion visualization method includes the following steps in step (3): (3-1) For registration to a unified reference space 3D image data According to LOD level The image data of the LOD layer is obtained by downsampling at the corresponding downsampling ratio, and the equivalent step size is calculated. :

[0013] in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer The corresponding downsampling factor; (3-2) For registration to a unified reference space 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. Its coverage is a unified reference space. Scope The calculation method is as follows: For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes :

[0014] Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view.

[0015] Preferably, the cross-magnification 3D image fusion visualization method includes the following steps in step (4): Specifically, the following steps are included: (4-1) Obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operation, and obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. ,in:

[0016] in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction.

[0017] (4-2) Map the pixels of the current field of view to a unified reference space. Display voxels Based on the image patch index obtained in matching step (3), the image patch whose equivalent voxel step size is closest to the screen sampling step size obtained in step (4-1) is selected, and the image patch with the highest LOD level is selected as the best image patch. ,in,

[0018] in, This represents the 3D image data to which the best data block belongs. , Its LOD level is the maximum LOD level that satisfies the closest equivalent voxel step size; (4-3) The optimal data block for all display voxels in the field of view will contain arbitrary three-dimensional image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

[0019] According to another aspect of the present invention, a cross-magnification 3D image fusion visualization system is provided, comprising the following modules: The image acquisition module is used to acquire data sources for the same sample. The data sources include low-magnification global 3D image data and one or more local 3D image data with magnification higher than the global 3D image. The 3D image data of the data sources is denoted as... Submitted to the image registration module; The image registration module is used for data source... Its global 3D image data uses the coordinate space as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in Submit to the LOD pyramid building module; The LOD pyramid construction module is used to construct a block-based and layered pyramid data structure based on the registered 3D image data using a quadtree partitioning method. It organizes the global 3D image data, local 3D image data, and downsampled images of each LOD layer into image blocks according to a preset block partitioning strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: Submitted to the adaptive display module; The adaptive display module is used to obtain the display voxel range and equivalent voxel step size of the current viewing field based on the initial field of view and the user's image transformation operation. For each voxel within the display voxel range of the current viewing field, the module matches the image block index to filter out the best image block and performs image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation, and rotation.

[0020] Preferably, in the cross-magnification 3D image fusion visualization system, the coverage areas of the local 3D image data and the global 3D image data overlap. In the case of multiple local 3D image data, the local 3D image data can be independent of each other, or they can have overlapping, intersecting, or containing relationships.

[0021] Preferably, in the cross-magnification 3D image fusion visualization system, the forward mapping matrix... The specific extraction method is as follows:

[0022] in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization.

[0023] The reverse mapping matrix The specific extraction method is as follows: ; The image registration module selects a magnification. Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. :

[0024] in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

[0025] Preferably, the LOD pyramid construction module of the cross-magnification 3D image fusion visualization system includes: The downsampling submodule is used for registration to a unified reference space. 3D image data According to LOD level The image is downsampled at the corresponding downsampling ratio to obtain its LOD layer image data, which is then submitted to the block segmentation submodule, where the equivalent step size is calculated. :

[0026] in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer The corresponding downsampling factor; The segmented sub-modules are used for registration to a unified reference space. 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. Its coverage is a unified reference space. Scope The calculation method is as follows: For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes :

[0027] Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view.

[0028] The adaptive display module includes: The field-of-view perception submodule is used to obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operations, and to obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. And submit it to the loading submodule, where:

[0029] in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction.

[0030] The loading submodule is used to map the pixels of the current field of view to a unified reference space. Display voxels Based on the matching image patch index, the image patch whose equivalent voxel step size is closest to the screen sampling step size is selected, and the image patch with the highest LOD level is selected as the best image patch. Display and load, where,

[0031] in, This represents the 3D image data to which the best data block belongs. , Its LOD level is the maximum LOD level that satisfies the closest equivalent voxel step size; The fusion submodule is used to select the optimal data block for all display voxels in the field of view, which will contain arbitrary 3D image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

[0032] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: This invention first registers images at different magnifications, and then performs cross-magnification LOD continuity from high magnification to low magnification. It can directly call up the original 3D data at different magnifications and perform a visualization method in the same 3D scene, including field-of-view partitioning display, cross-magnification LOD continuity connection, boundary fusion, grayscale consistency processing, and low-magnification fallback display. It is convenient for relative positioning and switching-free observation of images at different magnifications.

[0033] In a preferred embodiment, when transitioning from low-magnification to high-magnification display, the present invention does not start displaying from the lowest resolution LOD level of the high-magnification data, nor does it switch according to a fixed LOD level order. Instead, it calculates the equivalent voxel step size of candidate LOD levels based on the current screen sampling step size, the current magnification display accuracy, the physical sampling interval of the original data at each magnification, and the downsampling factor of each LOD level. The magnification and LOD level closest to the current display requirements are then selected as the starting display level. As the user continues to zoom in, the display gradually switches to a higher resolution LOD level, thereby avoiding sudden sparseness, blurring, abrupt changes, or premature loading of excessively high-resolution data. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the data source for collecting samples in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the construction of a LOD pyramid according to an embodiment of the present invention; Figure 3 This embodiment of the invention demonstrates the field of view at 10X magnification; Figure 4 This embodiment of the invention demonstrates the field of view at a 20X magnification. Figure 5 This embodiment of the invention demonstrates the field of view at 40X magnification; Figure 6 This is an example of the observation field of view at 100X magnification, as shown in this embodiment of the invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0036] The method for visualizing three-dimensional images across magnification provided by this invention includes the following steps: (1) For the same sample, the data source is obtained, which includes low-magnification global 3D image data and one or more local 3D image data with magnification higher than that of the global 3D image. The 3D image data of the data source is denoted as ; The coverage areas of the local three-dimensional image data and the global three-dimensional image data overlap. In the case of multiple local three-dimensional image data, the local three-dimensional image data can be independent of each other, or they can have overlapping, intersecting or containing relationships. For the same spatial location in the same sample, there is at least a low-magnification global 3D image data, and there may also be one or more local 3D image data.

[0037] (2) Registering local 3D image data to global 3D image data: For the data source obtained in step (1) It uses the coordinate space of its global three-dimensional image data as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in ; The positive mapping matrix The specific extraction method is as follows:

[0038] in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization.

[0039] The reverse mapping matrix The specific extraction method is as follows:

[0040] The local three-dimensional image data is in a unified reference space. Coverage in The specific steps to obtain it are as follows: Select the multiplier Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. :

[0041] in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

[0042] Used to determine whether the current field of view intersects with the magnification.

[0043] (3) Constructing the LOD pyramid: Based on the registered 3D image data obtained in step (2), a block-based and layered pyramid data structure is constructed using a quadtree partitioning method. The global 3D image data and local 3D image data, as well as the downsampled images of each LOD layer, are organized into image blocks according to a preset block partitioning strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: ; Specifically, the following steps are included: (3-1) For registration to a unified reference space 3D image data According to LOD level The image data of the LOD layer is obtained by downsampling at the corresponding downsampling ratio, and the equivalent step size is calculated. :

[0044] in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer Corresponding downsampling factor (3-2) For registration to a unified reference space 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. Its coverage is a unified reference space. Scope The calculation method is as follows: For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes :

[0045] Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view.

[0046] (4) Adaptive display based on user field of view: Based on the initial field of view and the user's image transformation operation, obtain the display voxel range and equivalent voxel step size of the current observation field of view. For each voxel in the display voxel range of the current observation field of view, match the image block index obtained in step (3) to filter out the best image block and perform image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation and rotation.

[0047] Specifically, the following steps are included: (4-1) Obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operation, and obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. ,in:

[0048] in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction.

[0049] (4-2) Map the pixels of the current field of view to a unified reference space. Display voxels Based on the image patch index obtained in matching step (3), the image patch whose equivalent voxel step size is closest to the screen sampling step size obtained in step (4-1) is selected, and the image patch with the highest LOD level is selected as the best image patch. ,in,

[0050] in, This represents the 3D image data to which the best data block belongs. , Its LOD level is the maximum LOD level that satisfies the closest equivalent voxel step size.

[0051] (4-3) The optimal data block for all display voxels in the field of view will contain arbitrary three-dimensional image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

[0052] The following is an example: The method for visualizing three-dimensional images across magnification provided by this invention includes the following steps: (1) For the same sample, the data source is obtained, which includes low-magnification global 3D image data and one or more local 3D image data with magnification higher than that of the global 3D image. The 3D image data of the data source is denoted as ; Three-dimensional image data is recorded as The imaging magnification is used to denot ; Indicate its size, used to describe , , Number of voxels in the direction; This indicates the voxel sampling interval; Representing three-dimensional image data The grayscale range.

[0053] The coverage areas of the local three-dimensional image data and the global three-dimensional image data overlap. In the case of multiple local three-dimensional image data, the local three-dimensional image data can be independent of each other, or they can have overlapping, intersecting or containing relationships. For the same spatial location in the same sample, there is at least a low-magnification global 3D image data, and there may also be one or more local 3D image data.

[0054] In this embodiment, imaging of the same sample yielded 10× global 3D image data, 20× local 3D image data, 40× local 3D image data, and 100× local 3D image data, respectively. Figure 1 As shown.

[0055] (2) Registering local 3D image data to global 3D image data: For the data source obtained in step (1) It uses the coordinate space of its global three-dimensional image data as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in ; The coordinate mapping relationship can be expressed as:

[0056]

[0057] in, Indicates multiplier Local three-dimensional image data Homogeneous coordinates in the local coordinate system Indicates multiplier Homogeneous coordinates in a unified reference space coordinate system for global 3D image data, and a forward mapping matrix. Used to determine the location of local 3D image data in a low-magnification global 3D image; inverse mapping matrix This is used to retrieve the coordinates of a high-magnification local 3D image from the user's current field of view during the display phase, thereby determining the original data block that needs to be read.

[0058] The positive mapping matrix The specific extraction method is as follows:

[0059] in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization.

[0060] The reverse mapping matrix The specific extraction method is as follows:

[0061] The local three-dimensional image data is in a unified reference space. Coverage in The specific steps to obtain it are as follows: Select the multiplier Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. :

[0062] in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

[0063] Used to determine whether the current field of view intersects with the magnification.

[0064] (3) Constructing the LOD pyramid: Based on the registered 3D image data obtained in step (2), a block-based and layered pyramid data structure is constructed using a quadtree partitioning method, such as... Figure 2 As shown, the global 3D image data and local 3D image data, along with their LOD layer downsampled images, are organized into image blocks according to a preset block division strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: ; Specifically, the following steps are included: (3-1) For registration to a unified reference space 3D image data The image data of the LOD layer is obtained by downsampling according to the corresponding downsampling ratio of the LOD layer. In this embodiment, a 5-layer LOD (Level of Detail) hierarchy is used, with downsampling ratios corresponding to the LOD layers as follows: LOD1: 1x, LOD2: 2x, LOD3: 4x, LOD4: 8x, and LOD5: 16x. For 10× global 3D image data, 20× local image data, 40× 3D image data, and 100× 3D image data, downsampling is performed according to the corresponding downsampling ratio of the LOD layer, and the equivalent voxel step size is calculated. :

[0065] in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer The corresponding downsampling factor, in this embodiment, can be expressed as: ,in, .when hour, , indicating the original resolution level; when hour, ;when hour, And so on.

[0066] In this embodiment, the equivalent voxel step size of the 3D image data of each layer of the LOD pyramid is... As shown in the table below:

[0067] (3-2) For registration to a unified reference space 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. .

[0068] Based on the preset three-dimensional block size, the unified reference space The image is divided into blocks, with each block containing global or local 3D image data and its LOD layer data, forming an image block as the loading unit. For each image block... Establish its display index, image patch index For: the global 3D image data or the local 3D image data to which it belongs LOD level and equivalent step size and covering a unified reference space Scope ;writing: ; For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes :

[0069] Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view.

[0070] In this embodiment, 128×128×128 voxels are used as the block size to divide the three-dimensional block size into blocks for each LOD level.

[0071] (4) Adaptive display based on user field of view: Based on the initial field of view and the user's image transformation operation, obtain the display voxel range and equivalent voxel step size of the current observation field of view. For each voxel in the display voxel range of the current observation field of view, match the image block index obtained in step (3) to filter out the best image block and perform image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation and rotation.

[0072] Specifically, the following steps are included: (4-1) Obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operation, and obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. .

[0073] The system calculates the screen sampling step size based on the current field of view's display requirements on the screen:

[0074] in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction. The smaller the screen sampling step size, the higher the user's zoom level, requiring higher resolution data for display.

[0075] After determining the current field of view, a grayscale value is assigned to each display pixel in the current field of view; specifically, this includes the following steps: (4-2) Map the pixels of the current field of view to a unified reference space. Display voxels Based on the image patch index obtained in matching step (3), the image patch whose equivalent voxel step size is closest to the screen sampling step size obtained in step (4-1) is selected, and the image patch with the highest LOD level is selected as the best image patch. ,in,

[0076] in, This indicates the 3D image data to which the selected data block belongs. , To satisfy the requirement of the highest LOD level closest to the equivalent voxel step size, when multiple 3D image data blocks have the same equivalent step size, the priority rule is as follows: the one with the highest LOD level is selected first. For example, in this embodiment, if the equivalent voxel step size of the 10× global 3D image LOD layer 1 and the 20× local 3D image LOD layer 2 is the same, then the image block of the 20× local 3D image LOD layer 2 with the highest LOD level is selected.

[0077] (4-3) The optimal data block for all display voxels in the field of view will contain arbitrary three-dimensional image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

[0078] In this embodiment, when the field of view is the XY plane, image block b contains three-dimensional image data. If the corner point is found, then three image blocks adjacent to image block b in the XY plane are found, and three-dimensional image fusion is performed to ensure consistent grayscale. The fused image is then used to update the pixels of image block b and its adjacent image blocks in the XY plane.

[0079] When transitioning from a low-magnification display to a high-magnification display, the system does not start from the lowest resolution level of the high-magnification display. Instead, it begins at the high-magnification LOD level that matches the current display precision, and gradually switches to higher resolution levels as the user continues to zoom in. Figures 3 to 6 As shown.

[0080] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for cross-magnification 3D image fusion visualization, characterized in that, Includes the following steps: (1) For the same sample, the data source is obtained, which includes low-magnification global 3D image data and one or more local 3D image data with magnification higher than that of the global 3D image. The 3D image data of the data source is denoted as ; (2) Registering local 3D image data to global 3D image data: For the data source obtained in step (1) It uses the coordinate space of its global three-dimensional image data as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in ; (3) Constructing the LOD pyramid: Based on the registered 3D image data obtained in step (2), a block-based and layered pyramid data structure is constructed using a quadtree partitioning method. The global 3D image data and local 3D image data, as well as the downsampled images of each LOD layer, are organized into image blocks according to a preset block partitioning strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: ; (4) Adaptive display based on user field of view: Based on the initial field of view and the user's image transformation operation, obtain the display voxel range and equivalent voxel step size of the current observation field of view. For each voxel in the display voxel range of the current observation field of view, match the image block index obtained in step (3) to filter out the best image block and perform image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation and rotation.

2. The cross-magnification 3D image fusion visualization method as described in claim 1, characterized in that, The coverage areas of the local 3D image data and the global 3D image data overlap. In the case of multiple local 3D image data, the local 3D image data can be independent of each other, or they can have overlapping, intersecting, or containing relationships.

3. The cross-magnification 3D image fusion visualization method as described in claim 1, characterized in that, The forward mapping matrix in step (2) The specific extraction method is as follows: ； in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization. The reverse mapping matrix The specific extraction method is as follows: 。 4. The cross-magnification 3D image fusion visualization method as described in claim 1, characterized in that, Step (2) Select the multiplier Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. : ； in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

5. The cross-magnification 3D image fusion visualization method as described in claim 1, characterized in that, Step (3) includes the following steps: (3-1) For registration to a unified reference space 3D image data According to LOD level The image data of the LOD layer is obtained by downsampling at the corresponding downsampling ratio, and the equivalent step size is calculated. : ； in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer The corresponding downsampling factor; (3-2) For registration to a unified reference space 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. Its coverage is a unified reference space. Scope The calculation method is as follows: For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes : ； Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view.

6. The cross-magnification 3D image fusion visualization method as described in claim 1, characterized in that, Step (4) includes the following steps: Specifically, the following steps are included: (4-1) Obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operation, and obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. ,in: ； in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction; (4-2) Map the pixels of the current field of view to a unified reference space. Display voxels Based on the image patch index obtained in matching step (3), the image patch whose equivalent voxel step size is closest to the screen sampling step size obtained in step (4-1) is selected, and the image patch with the highest LOD level is selected as the best image patch. ,in, ； in, This represents the 3D image data to which the best data block belongs. , Its LOD level is the maximum LOD level that satisfies the closest equivalent voxel step size; (4-3) The optimal data block for all display voxels in the field of view will contain arbitrary three-dimensional image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

7. A cross-magnification 3D image fusion visualization system, characterized in that, Includes the following modules: The image acquisition module is used to acquire data sources for the same sample. The data sources include low-magnification global 3D image data and one or more local 3D image data with magnification higher than the global 3D image. The 3D image data of the data sources is denoted as... Submitted to the image registration module; The image registration module is used for data source... Its global 3D image data uses the coordinate space as a unified reference space. Each local 3D image data is registered with the global 3D image data, and the positive mapping matrix from the local coordinate system to the unified reference space of the local 3D image data is extracted. and a unified reference space Inverse mapping matrix to the local coordinate system of the local 3D image data Local 3D image data in a unified reference space Coverage in Submit to the LOD pyramid building module; The LOD pyramid construction module is used to construct a block-based and layered pyramid data structure based on the registered 3D image data using a quadtree partitioning method. It organizes the global 3D image data, local 3D image data, and downsampled images of each LOD layer into image blocks according to a preset block partitioning strategy. For each image block... Establish its display index, image patch index For: the corresponding 3D image data LOD level Equivalent step size and covering a unified reference space Scope ;writing: Submitted to the adaptive display module; The adaptive display module is used to obtain the display voxel range and equivalent voxel step size of the current viewing field based on the initial field of view and the user's image transformation operation. For each voxel within the display voxel range of the current viewing field, the module matches the image block index to filter out the best image block and performs image block boundary fusion to determine the gray value of each voxel for display. The user's image transformation operation includes scaling, translation, and rotation.

8. The cross-magnification 3D image fusion visualization system as described in claim 7, characterized in that, The coverage areas of the local 3D image data and the global 3D image data overlap. In the case of multiple local 3D image data, the local 3D image data can be independent of each other, or they can have overlapping, intersecting, or containing relationships.

9. The cross-magnification 3D image fusion visualization system as described in claim 6, characterized in that, The positive mapping matrix The specific extraction method is as follows: ； in, The local 3D image is obtained by registering the local 3D image with the global 3D image. The three-dimensional spatial transformation matrix, when using rigid registration When representing a rotation matrix and performing registration using similarity transformations... The product of the scale factor and the rotation matrix is ​​used for affine transformation registration. Represents a general three-dimensional affine matrix, which may include differences in scale, rotation, shearing, or orientation; To obtain by registering local 3D images with global 3D images Translation vector; express Zero vector. Results obtained from different registration methods can all be converted into a unified matrix form for subsequent visualization. The reverse mapping matrix The specific extraction method is as follows: ； The image registration module selects a magnification. Local three-dimensional image data The eight corner points , And use a forward mapping matrix to map it to a unified reference space. Then, based on the mapped point set, the 3D bounding box of the local 3D image data in the unified reference space is calculated. : ； in, Indicates multiplier The first local three-dimensional image data Corner points, This indicates that the 3D bounding box is calculated based on the mapped point set. Indicates multiplier Local three-dimensional image data In a unified reference space The coverage area.

10. The cross-magnification 3D image fusion visualization system as described in claim 7, characterized in that, The LOD pyramid construction module includes: The downsampling submodule is used for registration to a unified reference space. 3D image data According to LOD level The image is downsampled at the corresponding downsampling ratio to obtain its LOD layer image data, which is then submitted to the block segmentation submodule, where the equivalent step size is calculated. : ； in, Representing three-dimensional image data relative to a unified reference space The basic sampling interval is calculated using the following formula: ,in, Represents a unified reference space Corresponding imaging magnification; LOD layer The corresponding downsampling factor; The segmented sub-modules are used for registration to a unified reference space. 3D image data Its LOD layer data is in a unified reference space The image is divided into blocks of the same size to obtain three-dimensional image data and its LOD layer data. Its coverage is a unified reference space. Scope The calculation method is as follows: For magnification of The LOD layer is 3D image data A certain data block Covering a unified reference space The range is the bounding box The method to obtain it is as follows: Obtain the eight corner points of the data block in the local coordinate system. , And calculate the data block in the unified reference space using the forward mapping matrix. Block-level bounding boxes : ； Block-level bounding boxes are used to quickly filter the data blocks that need to be loaded within the current field of view; The adaptive display module includes: The field-of-view perception submodule is used to obtain the display voxel range of the current field of view based on the initial field of view and the user's image transformation operations, and to obtain the display voxel range and equivalent voxel step size of the current field of view based on the scaling size of the observation field of view. And submit it to the loading submodule, where: ； in, Indicates the current screen sampling step size. This indicates that the current field of view is in a unified reference space. The actual length in the middle, This indicates the number of screen pixels in that direction; The loading submodule is used to map the pixels of the current field of view to a unified reference space. Display voxels Based on the matching image patch index, the image patch whose equivalent voxel step size is closest to the screen sampling step size is selected, and the image patch with the highest LOD level is selected as the best image patch. Display and load, where, ； in, This represents the 3D image data to which the best data block belongs. , Its LOD level is the maximum LOD level that satisfies the closest equivalent voxel step size; The fusion submodule is used to select the optimal data block for all display voxels in the field of view, which will contain arbitrary 3D image data. The corner data blocks are used as edge data blocks. The best data blocks of all display voxels in the adjacent viewing field of the edge data blocks are used as superpixels. Image fusion is performed to ensure grayscale consistency and smooth image transition. The grayscale of the three-dimensional image after superpixel fusion is obtained and the grayscale of all pixels in the superpixel is updated accordingly.

Images