A method for simultaneous generation of digital slice streaming stitching and sparse pyramid based on content rating

Abstract

This invention discloses a method for streaming digital tile stitching and synchronous generation of sparse pyramids based on content scoring, belonging to the technical field of digital image processing and storage. The method includes: acquiring image blocks and establishing an initial two-dimensional grid layout; classifying the image blocks; performing differential displacement estimation and fusion based on the category combination of the image blocks, and writing the fusion result to an output buffer; performing overall vertical registration; when stable regions exist, extracting image data within the stable regions to generate a sparse hierarchical multi-resolution tile pyramid, and synchronously writing the image data and pyramid tiles to a BigTIFF container. This invention can decouple the memory usage and output size of the stitching process as much as possible; it can adaptively select registration and fusion strategies based on the differences in image block content; and it synchronously generates a sparse hierarchical multi-resolution tile pyramid and writes it to containers such as BigTIFF during the stitching and writing process, thereby reducing redundancy and improving overall processing efficiency and subsequent browsing performance.

Description

Technical Field

[0001] This invention belongs to the technical field of digital image processing and storage, specifically relating to a method for simultaneous generation of digital slice streaming stitching and sparse pyramids based on content scoring. Background Technology

[0002] In digital pathological slide scanning, remote sensing image stitching, and other ultra-large format imaging applications, the target large-size image is often represented as a sequence of image blocks with spatial location identifiers (such as JPEG / PNG image blocks identified by row and column coordinates or filename encoding). A certain proportion of overlapping area is usually set between adjacent image blocks in order to perform registration and fusion and ensure the continuity of the final stitching result.

[0003] Taking digital pathology full-view slides as an example, due to the needs of tissue preparation and slide reading, slides are often stained with hematoxylin and eosin (HE), resulting in images exhibiting the typical characteristics of "concentrated stained tissue areas" and "large areas of low-color background areas": the tissue areas have rich texture and complex boundaries, while the background areas have low color saturation and texture intensity and sparse information. This strong non-uniformity leads to significant differences in the requirements for displacement estimation and fusion strategies in different regions during the stitching process.

[0004] In existing technologies, stitching algorithms typically employ a single registration and fusion strategy to process all image patches uniformly, such as using feature point matching throughout, frequency domain correlation throughout, or the same fusion method throughout. Because low-color background regions lack stable textures, feature point methods are prone to insufficient features, mismatches, or unstable matching. Conversely, using only low-complexity correlation strategies in structurally complex regions may result in displacement estimation errors or seam artifacts. Adopting a single algorithm to simultaneously adapt to both types of regions often creates a trade-off between speed, robustness, and stitching quality, making it difficult to balance overall efficiency and stability.

[0005] On the other hand, the size of ultra-large format stitching results usually far exceeds the capacity of memory to hold at one time. Existing solutions often cache a large number of intermediate fusion results in memory, or generate intermediate result files first and then write them out in a unified manner. This leads to memory usage increasing with the output size and significant I / O amplification. In single-machine memory-constrained environments, this may result in problems such as excessively long processing time, excessively high memory peaks, or even failure to complete stitching.

[0006] Meanwhile, existing multi-resolution pyramids are usually generated layer by layer at a fixed scaling factor (e.g., shrinking by 1 / 2 each time). In fast browsing scenarios, not all intermediate layers are necessary. Layer-by-layer scaling will generate a lot of redundant layers and tile data, increasing preprocessing time and storage overhead. Furthermore, stitching and pyramid generation are usually performed separately, which can easily lead to repeated reading and calculation of the already generated large image. Summary of the Invention

[0007] The purpose of this invention is to address the aforementioned shortcomings in the prior art by providing a method for streaming digital slice splicing and synchronous generation of sparse pyramids based on content scoring. This method enables segmented splicing and segmented writing under limited memory conditions, reducing the memory dependence of the splicing process. Furthermore, considering the non-uniform content characteristics of digital pathological slices—rich texture in tissue areas, sparse texture in background areas, and low color—content scoring is used to adaptively switch between splicing and fusion strategies, thereby improving the stability and efficiency of splicing.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for simultaneous generation of digital slice streaming stitching and sparse pyramid based on content scoring includes the following steps:

[0010] S1. Obtain the image patch sequence with overlapping regions, and establish an initial two-dimensional grid layout based on the image patch size, nominal overlap rate and spatial location identifier, and calculate the initial step size;

[0011] S2. Calculate the content score of the image patch in terms of color saturation, and classify the image patch according to the content score;

[0012] S3. Based on the category combination of the current image block and the adjacent image blocks that have been stitched together, perform differential displacement estimation and fusion on two adjacent image blocks in the same row, and write the fusion result into the output buffer.

[0013] S4. After stitching together a row of image blocks, estimate the inter-row displacement of each column of image blocks in that row, and perform overall vertical registration of the current row based on the inter-row displacement.

[0014] S5. Determine whether there is a stable region in the output buffer; if there is a stable region, extract the image data within the stable region to generate a sparse hierarchical multi-resolution tile pyramid, and write the image data and pyramid tiles synchronously into the BigTIFF container.

[0015] S6. Repeat S2-S5 in a predetermined order until all image blocks are stitched together and the target large-size image is generated.

[0016] Furthermore, in S1, the initial step size is calculated based on the nominal overlap rate, which is expressed as:

[0017]

[0018] In the formula, , For the pixel width and height of the image block, , The nominal overlap rate, The initial horizontal step size, This is the initial vertical step size.

[0019] Furthermore, S2 specifically includes:

[0020] Calculate the content score of an image patch in terms of color saturation:

[0021]

[0022] In the formula, Color score, The average saturation value. These are the weighting coefficients. For saturation greater than the first threshold pixel ratio, Indicates the saturation channel;

[0023] If color score If so, the image patch is classified as class A. The second threshold;

[0024] If color score If so, the image block is classified as class B.

[0025] Furthermore, S3 specifically includes:

[0026] When a combination of adjacent image blocks contains at least one Class A image block, phase-correlation-based displacement estimation is performed on the overlapping region of the image blocks, and a linear feathering method is used for fusion.

[0027] When adjacent image patch combinations are all class B image patches, a translational displacement estimation based on feature point matching is adopted, and a multi-band fusion method is used to suppress seam artifacts.

[0028] Then, the fusion result is written to the output buffer.

[0029] Furthermore, S4 specifically includes:

[0030] After stitching together a row of image blocks, the inter-row displacement of each corresponding image block in each column is estimated, and the inter-row displacement estimation results of each column are robustly aggregated. The median of the inter-row displacement estimation results is selected as the overall inter-row displacement, and the entire row is aligned based on the overall inter-row displacement.

[0031] Furthermore, in step S5, determining whether a stable region exists in the output buffer includes:

[0032] Define the stable write boundary as:

[0033]

[0034] In the formula, To ensure stable writing to the boundary, To splice the boundaries of the event, R is the maximum overlap width, and R is the fusion influence radius;

[0035] When the coordinates of the image block in the output buffer If a stable region exists, the distance between the boundary of the stable region and the boundary of the active splicing is greater than the maximum overlap width and the fusion influence radius.

[0036] Furthermore, in step S5, if a stable region exists, the image data within the stable region is extracted to generate a sparse hierarchical multi-resolution tile pyramid, and the image data and pyramid tiles are synchronously written into a BigTIFF container. This specifically includes the following sub-steps:

[0037] S501. Define the target level set L of the pyramid. The target level set L is a sparse level set, which includes the base level and at least one scaling level.

[0038] S502. Use equivalent downsampling filtering to verify the stable region and generate multi-resolution image data corresponding to the target hierarchical set L;

[0039] S503. Divide the multi-resolution image data of each level into tiles according to the preset tile size, and write them into the target BigTIFF file with the BigTIFF multi-directory structure or equivalent index structure, so as to support the reading end to independently locate the tiles by level and perform local reading and fast browsing.

[0040] Furthermore, in S502, the logic for generating multi-resolution image data is as follows: the target scaling layer is directly generated from the reference layer using an equivalent downsampling filter kernel, which is expressed as:

[0041]

[0042] In the formula, Scale the image data at the target level. As the baseline layer image data, For convolution, This is the equivalent downsampling filter kernel, ↓ represents the downsampling operation, and ↓s indicates downsampling by the scaling factor s.

[0043] The method for simultaneous generation of digital slice streaming and sparse pyramids based on content scoring provided by this invention has the following beneficial effects:

[0044] 1. By writing to the stable region in segments through a bounded memory output buffer, the memory usage of the stitching process is decoupled from the output image size as much as possible, which is suitable for single-machine memory-constrained environments;

[0045] 2. By using content scoring and A / B classification, differentiated displacement estimation and fusion strategy switching are achieved, which reduces the risk of mismatch in low-color background areas and improves the seam suppression and splicing stability in high-texture areas.

[0046] 3. By estimating the inter-row column by column and aggregating the median, the overall inter-row displacement is obtained, which reduces the misalignment caused by abnormal column bias and improves the robustness of inter-row registration;

[0047] 4. Reduce intermediate level redundancy by using sparse level sets (e.g., 1, 1 / 4, 1 / 16) and equivalent kernel direct downsampling strategy, thereby reducing pyramid generation time and storage space;

[0048] 5. Multi-resolution data structures written to BigTIFF via tiled format enable subsequent fast browsing and partial reading. Attached Figure Description

[0049] Figure 1 This is a flowchart of a method for synchronous generation of digital slices and sparse pyramids based on content scoring, according to an embodiment of the present invention.

[0050] Figure 2 This is a schematic flowchart of an embodiment of the present invention;

[0051] Figure 3 This is a schematic diagram of the content scoring and A / B classification process in an embodiment of the present invention;

[0052] Figure 4 This is a schematic diagram illustrating the selection of splicing and fusion strategies based on the combination of categories A and B in an embodiment of the present invention;

[0053] Figure 5 This is a schematic diagram of the registration process of estimating the offset between rows and columns and aggregating by taking the median in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram illustrating the writing of the output buffer and stable region in an embodiment of the present invention;

[0055] Figure 7 This is a schematic diagram of the processing route for constructing a sparse hierarchical pyramid across levels in an embodiment of the present invention;

[0056] Figure 8 This is a schematic diagram illustrating the generation of a sparse hierarchical pyramid according to an embodiment of the present invention. Detailed Implementation

[0057] The specific embodiments of the present invention are described below to enable those skilled in the art to understand the present invention. However, it should be understood that the present invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the present invention as defined and determined by the appended claims. All inventions utilizing the concept of the present invention are protected.

[0058] The content-scoring-based digital slice streaming stitching and sparse pyramid synchronous generation method in this embodiment can avoid repeated reading and multiple generation of large images in the post-processing stage, reduce intermediate level redundancy, reduce storage and computing overhead, and at the same time meet the needs of subsequent fast browsing and local reading.

[0059] This embodiment uses digital pathology whole-view section (WSI) stitching as an example. The input is a sequence of image blocks identified by spatial location. Each image block is a JPEG / PNG image, with a size of, for example, 1200×1920 pixels. There is a preset overlap rate between adjacent image blocks (e.g., horizontal overlap rate 0.10, vertical overlap rate 0.06). The image blocks are arranged into a regular grid (e.g., 2×64, 31×64, etc.) according to their spatial location. The spatial location identifier can be obtained from filename parsing or provided by an index table. Furthermore, since digital pathology stained sections typically exhibit non-uniformity with "high texture / high color in tissue areas and low color / low texture in background areas," this embodiment classifies image blocks into categories A and B through content scoring and switches displacement estimation and fusion strategies accordingly to balance efficiency and stability. (Reference) Figure 1 and Figure 2 This embodiment specifically includes the following:

[0060] S1. Obtain a sequence of image blocks with overlapping regions. Each input image block has a spatial location identifier, and adjacent input image blocks have overlapping regions. Establish an initial two-dimensional grid layout based on the image block size, nominal overlap rate, and spatial location identifier, and calculate the initial step size. The two-dimensional grid index is... The image patch (where i is the column number and j is the row number) can be initially placed at the following coordinates: The initial placement coordinates mentioned above are used to provide the search center for displacement estimation, and the final placement coordinates are corrected by the relative displacement obtained from subsequent registration estimation.

[0061] The initial step size is calculated based on the nominal overlap rate and is expressed as follows:

[0062]

[0063] In the formula, , For the pixel width and height of the image block, , The nominal overlap rate, The initial horizontal step size, This is the initial vertical step size.

[0064] S2. Calculate the content score of the image patch in terms of color saturation, and classify the image patch according to the content score. Taking color score as an example: convert the image to HSV space, refer to... Figure 3 Specifically, it is as follows:

[0065] Calculate the content score of an image patch in terms of color saturation:

[0066]

[0067] In the formula, Color score, The average saturation value. These are the weighting coefficients. For saturation greater than the first threshold pixel ratio, Indicates the saturation channel;

[0068] If color score If so, the image patch is classified as class A. The second threshold;

[0069] If color score If so, the image block is classified as class B.

[0070] In this class, Class A represents low color / low texture blocks, and Class B represents high color / high texture blocks;

[0071] In one specific embodiment, Take 30, Take 50, and T can be obtained from sample statistics; the above values ​​are only examples and do not constitute a limitation.

[0072] S3. Based on the category combination of the current image block and the adjacent image blocks that have been stitched together, perform differential displacement estimation and fusion on two adjacent image blocks in the same row, and write the fusion result into the output buffer.

[0073] For details, please refer to Figure 4 Perform the following operation on two adjacent blocks within the same row:

[0074] When a combination of adjacent image blocks contains at least one Class A image block, such as AA, AB, or BA, a phase-correlation-based displacement estimation is performed on the overlapping region of the image blocks (the overlapping region can be downsampled first, for example, to 0.5), and a linear feathering method is used for fusion.

[0075] When adjacent image patch combinations are all B-class image patches, i.e. BB, a translational displacement estimation based on feature point matching (SURF / ORB optional) is adopted, and a multi-band fusion method (e.g., Laplacian pyramid) is used to suppress seam artifacts.

[0076] Then, the fusion result is written to the output buffer.

[0077] The preferred method for merging overlapping regions is as follows:

[0078] Linear feathering blending preferably uses window function weights for pixel-wise weighted blending. The blending method is: Output pixel = stitched pixels × (1 w) + pixels to be stitched × w, where w is the weight value calculated based on the position of the overlapping area.

[0079] Multi-band fusion can employ the Laplacian pyramid to achieve smooth transitions across multiple frequency bands while preserving detail, thereby reducing seam artifacts.

[0080] S4. After stitching together a row of image blocks, estimate the inter-row displacement of each column of image blocks in that row, and perform overall vertical registration of the current row based on the inter-row displacement.

[0081] Specifically, after stitching together one row of image blocks, the following steps are performed on adjacent rows:

[0082] refer to Figure 5 Estimate the inter-row displacement for each corresponding image block in each column, and analyze the inter-row displacement estimation results for each column. Robust aggregation is performed, the median of the inter-line displacement estimation results is selected as the overall inter-line displacement, and the entire line is aligned based on the overall inter-line displacement.

[0083] The median is preferred. and the entire downward movement will be arranged according to Alignment.

[0084] S5. Determine whether there is a stable region in the output buffer; if there is a stable region, extract the image data within the stable region to generate a sparse hierarchical multi-resolution tile pyramid, and write the image data and pyramid tiles synchronously into the BigTIFF container.

[0085] For details, please refer to Figure 6 Maintain an output buffer (strip buffer / sliding window) with a preset memory limit, and write the fused output to the corresponding position in the buffer; when the write condition is met, write the stable region to the target image file and release or reuse the buffer;

[0086] The conditions for writing out include: the cumulative height of the output buffer reaches the preset strip height (e.g., an integer multiple of 256); and / or a stable region is formed, that is, the region is no longer affected by the overlapping and fusion of subsequent image blocks;

[0087] Determining whether a stable region exists in the output buffer includes:

[0088] Define the stable write boundary as:

[0089]

[0090] In the formula, To ensure stable writing to the boundary, To splice the boundaries of the event, R is the maximum overlap width, and R is the fusion influence radius;

[0091] When the coordinates of the image block in the output buffer If a stable region exists, the distance between the boundary of the stable region and the boundary of the active splicing is greater than the maximum overlap width and the fusion influence radius, in order to avoid seams or misalignments caused by "writing first and then being overlaid and fused and modified later".

[0092] During streaming stitching, when a stable region forms in the output buffer and is ready to be written to the target image file, a sparse hierarchical multi-resolution tile pyramid is simultaneously generated and written to the BigTIFF container. This avoids a second reading and layer-by-layer generation of the entire large image after stitching is complete. To execute this process, it is first necessary to determine whether a stable region exists, i.e.:

[0093] If a stable region exists, the image data within that region is extracted to generate a sparse hierarchical multi-resolution tile pyramid. The image data and pyramid tiles are then synchronously written into a BigTIFF container. (See reference...) Figure 7 and Figure 8 The process specifically includes the following sub-steps:

[0094] S501. Set the target level set L of the pyramid. The target level set L is a sparse level set, which includes the base level and at least one scaling level, such as 1, 1 / 4, and 1 / 16, to reduce redundancy in the intermediate levels.

[0095] S502. Determine the equivalent downsampling kernel parameters, and use the equivalent downsampling filter kernel or the same type of low-pass filter kernel to generate multi-resolution image data corresponding to the target level set L for the stable region, so as to suppress aliasing and maintain visual continuity.

[0096] In this process, multi-resolution data generation uses an equivalent downsampling filter kernel to directly generate the target scaling layer from the baseline layer, rather than generating it layer by layer in a 1 / 2 recursive manner. Equivalent downsampling is used to suppress aliasing and ensure visual continuity and consistency, or near-consistency, without requiring it to be completely identical pixel-by-pixel to the result of layer-by-layer recursion. It is represented as follows:

[0097]

[0098] In the formula, Scale the image data at the target level. As the baseline layer image data, For convolution, For the equivalent downsampling filter kernel, the equivalent kernel Make In terms of aliasing suppression and visual continuity, it is consistent with or approximately consistent with the results of layer-by-layer 1 / 2 recursion, but does not require pixel-by-pixel completeness; ↓ represents downsampling operation, ↓s indicates downsampling by scaling factor s.

[0099] S503. Divide the multi-resolution image data of each level into tiles according to the preset tile size (preferably 1024×1024, but can also be configured as 512×512, etc.), and write them into the target BigTIFF file with a BigTIFF multi-directory (IFD) structure or an equivalent index structure, so as to support the reading end to independently locate the tiles by level and perform local reading and fast browsing.

[0100] S6. Repeat S2-S5 in a predetermined order until all image blocks are stitched together and the target large-size image is generated.

[0101] Traditional pyramid file storage with a sampling factor of 2 is typically 1.33 times the size of an image. This invention, by removing redundant layers, can minimize the image size, reducing it to 1.067 times the original size. Compared to traditional methods, this invention reduces overhead by 26.3 percentage points.

[0102] Although specific embodiments of the invention have been described in detail with reference to the accompanying drawings, this should not be construed as limiting the scope of protection of this patent. Various modifications and variations that can be made by a person skilled in the art without inventive effort within the scope described in the claims still fall within the scope of protection of this patent.

Claims

1. A method for simultaneous generation of digital slice streaming stitching and sparse pyramid based on content scoring, characterized in that, Includes the following steps: S1. Obtain the image patch sequence with overlapping regions, and establish an initial two-dimensional grid layout based on the image patch size, nominal overlap rate and spatial location identifier, and calculate the initial step size; S2. Calculate the content score of the image patch in terms of color saturation, and classify the image patch according to the content score; S3. Based on the category combination of the current image block and the adjacent image blocks that have been stitched together, perform differential displacement estimation and fusion on two adjacent image blocks in the same row, and write the fusion result to the output buffer; S3 specifically includes: When a combination of adjacent image blocks contains at least one Class A image block, phase-correlation-based displacement estimation is performed on the overlapping region of the image blocks, and a linear feathering method is used for fusion. When adjacent image patch combinations are all class B image patches, a translational displacement estimation based on feature point matching is adopted, and a multi-band fusion method is used to suppress seam artifacts. Then, the fusion result is written to the output buffer; S4. After stitching together a row of image blocks, estimate the inter-row displacement of each column of image blocks in that row, and perform overall vertical registration of the current row based on the inter-row displacement. S5. Determine whether there is a stable region in the output buffer; if there is a stable region, extract the image data within the stable region to generate a sparse hierarchical multi-resolution tile pyramid, and write the image data and pyramid tiles synchronously into the BigTIFF container. In step S5, determining whether a stable region exists in the output buffer includes: Define the stable write boundary as: In the formula, To ensure stable writing to the boundary, To splice the boundaries of the event, R is the maximum overlap width, and R is the fusion influence radius; When the coordinates of the image block in the output buffer If the distance between the boundary of the stable region and the boundary of the active splicing is greater than the maximum overlap width and the fusion influence radius, then a stable region is determined to exist. In step S5, if a stable region exists, the image data within the stable region is extracted to generate a sparse hierarchical multi-resolution tile pyramid, and the image data and pyramid tiles are synchronously written into a BigTIFF container. This specifically includes the following sub-steps: S501. Define the target level set L of the pyramid. The target level set L is a sparse level set, which includes the base level and at least one scaling level. S502. Use equivalent downsampling filtering to verify the stable region and generate multi-resolution image data corresponding to the target hierarchical set L; S503. Divide the multi-resolution image data of each level into tiles according to the preset tile size, and write them into the target BigTIFF file with the BigTIFF multi-directory structure or equivalent index structure to support the reading end to independently locate the tiles by level and perform local reading and fast browsing. S6. Repeat S2-S5 in a predetermined order until all image blocks are stitched together and the target large-size image is generated.

2. The method for simultaneous generation of digital slice streaming and sparse pyramids based on content scoring according to claim 1, characterized in that, In S1, the initial step size is calculated based on the nominal overlap rate, and it is expressed as follows: In the formula, , For the pixel width and height of the image block, , The nominal overlap rate, The initial horizontal step size, This is the initial vertical step size.

3. The method for simultaneous generation of digital slice streaming and sparse pyramids based on content scoring according to claim 1, characterized in that, S2 specifically includes: Calculate the content score of an image patch in terms of color saturation: In the formula, Color score, The average saturation value. These are the weighting coefficients. For saturation greater than the first threshold pixel ratio, Indicates the saturation channel; If color score If so, the image patch is classified as class A. The second threshold; If color score If so, the image block is classified as class B.

4. The method for simultaneous generation of digital slice streaming and sparse pyramids based on content scoring according to claim 1, characterized in that, S4 specifically includes: After stitching together a row of image blocks, the inter-row displacement of each corresponding image block in each column is estimated, and the inter-row displacement estimation results of each column are robustly aggregated. The median of the inter-row displacement estimation results is selected as the overall inter-row displacement, and the entire row is aligned based on the overall inter-row displacement.

5. The method for simultaneous generation of digital slice streaming and sparse pyramids based on content scoring according to claim 1, characterized in that, In step S502, the logic for generating multi-resolution image data is as follows: The target scaling layer is directly generated from the reference layer using an equivalent downsampling filter kernel, which is expressed as: In the formula, Scale the image data at the target level. As the baseline layer image data, For convolution, This is the equivalent downsampling filter kernel, ↓ represents the downsampling operation, and ↓s indicates downsampling by the scaling factor s.

Images