Image stitching method and storage medium

Abstract

This invention relates to an image stitching method and storage medium. The main steps include: S1: acquiring a local field-of-view image and its initial position information; S2: performing characterization processing on the local field-of-view image; S3: filtering candidate adjacent images based on the initial position information; S4: calculating the theoretical overlap region between the candidate adjacent images; S5: performing matching within the local overlap region to obtain effective relative displacement constraints between images; S6: establishing a relationship graph structure between multiple images based on the effective relative displacement constraints; S7: maintaining and merging the graph structure; S8: globally optimizing the image positions in the graph structure; and S9: outputting the stitching result under unified coordinates based on the optimization results. This method improves stitching efficiency and accuracy.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to image stitching methods and storage media. Background Technology

[0002] In the prior art, the solutions related to this invention mainly include patents CN121213375A, CN117237193A, and CN121391601A. Among them, CN121213375A mainly targets adjacent digital pathology images in the scanning sequence, and performs feature point matching, affine correction, color mapping, and stitching seam optimization based on a preset overlapping area; CN117237193A mainly uses a pathology slide scanner to preprocess, extract features, perform affine transformation, and perform weighted average fusion after acquiring the image; CN121391601A evaluates the adjacency relationship between sub-slices through texture flow field, multi-scale feature descriptors, and a global stitching relationship graph model, and solves the stitching parameters through iterative optimization.

[0003] However, the aforementioned existing technologies still have the following shortcomings: (1) CN121213375A and CN117237193A are mainly for local registration and local fusion of adjacent image pairs. They lack unified modeling and optimization of the overall spatial relationship of multiple scanned images, and are prone to cumulative errors in large-scale continuous scanning. (2) Although the above scheme utilizes local overlapping areas or location information, it still does not fully combine the prior information of the initial position of the image naturally existing during the scanning process to screen candidate adjacency relationships. Therefore, it is difficult to simultaneously take into account processing efficiency and global consistency in large-scale scanning scenarios. (3) Although CN121391601A considers global optimization, it mainly relies on texture flow field and multi-scale content features to infer the relationship between images. The computation link is long and the processing complexity is high. It does not make sufficient use of the position prior in the scanning continuous image acquisition scenario.

[0004] Therefore, existing technologies still struggle to effectively address issues such as cumulative error suppression, global geometric consistency assurance, and high-efficiency stitching when stitching multiple images during digital slicing scanning. Summary of the Invention

[0005] Therefore, it is necessary to provide an image stitching method to address the problems in existing technologies. This method is beneficial for improving stitching efficiency and accuracy.

[0006] The specific plan is as follows: An image stitching method, comprising: S1: Obtain the local field of view image and its initial position information. S2: Perform characterization processing on the local field of view image. S3: Filter candidate neighboring images based on initial location information. S4: Calculate the theoretical overlap region between the image and the candidate neighboring image. S5: Perform matching within locally overlapping regions to determine the effective relative displacement constraints between images. S6: Based on the effective relative displacement constraints between images, establish a relationship graph structure between multiple images. S7: Maintenance and merging of execution graph structures S8: Perform global optimization on the image positions within the graph structure. S9: Output the stitching result under unified coordinates based on the optimization results.

[0007] In one embodiment, step S3, which involves filtering candidate neighboring images based on initial position information, specifically includes: taking the initial position of the current image as the center, and combining the width, height, and preset neighborhood range of a single image to determine the spatial distance between existing images; only when two images fall within the preset neighborhood range in both the width and height directions are they determined as candidate neighboring images.

[0008] In one embodiment, S4, calculating the theoretical overlap region between the current image and each candidate neighboring image specifically includes: for the current image and each candidate neighboring image, calculating the theoretical overlap region between the two images based on their initial estimated positions and field of view sizes.

[0009] In one embodiment, in S4, the theoretical overlapping region is extended to obtain an extended local overlapping region for matching.

[0010] In one embodiment, in S5, only when the local matching result meets the preset reliability condition is it determined that there is a valid adjacent relationship between the candidate adjacent image and the current image, and the output result is: the valid relative displacement constraint between the current image and the candidate adjacent image.

[0011] In one embodiment, in S6, within the graph structure: Each image serves as a node. The effective relative displacement constraint between any two images is taken as an edge. The edges record the effective relative displacement constraints between corresponding image pairs and the reliability of the matching.

[0012] In one embodiment, S7, the maintenance and merging of the execution graph structure specifically includes: When the current image establishes a connection with an existing graph structure, the image is incorporated into the corresponding graph structure. When the current image establishes connections with multiple existing graph structures simultaneously, these existing graph structures are merged. When certain images are spatially distant from the currently scanned area, their activity level can be reduced or their status managed.

[0013] In one embodiment, step S8 specifically includes: during global optimization, the initial estimated position of each image node is used as the initial value for optimization, and the relative displacement constraint between image nodes is used as the constraint term in the optimization objective. Through iterative solution, the positions of all images are made to reach the optimal consistency state as a whole.

[0014] In one embodiment, in step S9, the stitching result includes the final position coordinates of each image or the final stitched image.

[0015] In one embodiment, the characterization processing of the local field of view image in S2 includes grayscale processing of the image.

[0016] A computer storage medium storing at least one executable instruction that causes a processor to perform an operation corresponding to the image stitching method.

[0017] The beneficial effects of this application are as follows: (1) Using the initial position information of the image to filter the candidate adjacent images avoids matching all images, which helps to reduce the amount of computation and improve the stitching efficiency.

[0018] (2) Matching constraints are established only within the effective overlapping area to reduce interference from the background and non-overlapping areas and improve the stability of local matching.

[0019] (3) Constructing the relative positional relationship between multiple images into a unified graph structure and performing global optimization can effectively suppress cumulative errors and improve the overall geometric consistency of the entire digital pathology image.

[0020] (4) When grayscale characterization is used in step S2, it can not only improve the image processing speed, but also reduce the dependence on rich tissue texture features in a single image. Therefore, even when the background accounts for a large proportion in the scanning field of view and contains only a small amount of tissue area or a small amount of impurities, relatively stable image stitching can still be achieved, which improves the applicability of the present invention in sparse tissue scenarios. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating an image stitching method according to an embodiment of this application. Detailed Implementation

[0022] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0023] like Figure 1 As shown, an embodiment of this application provides an image stitching method, which includes: S1: Obtain the local field of view image and its initial position information. S2: Perform characterization processing on the local field of view image. S3: Filter candidate neighboring images based on initial location information. S4: Calculate the theoretical overlap region between the image and the candidate neighboring image. S5: Perform matching within locally overlapping regions to determine the effective relative displacement constraints between images. S6: Based on the effective relative displacement constraints between images, establish a relationship graph structure between multiple images. S7: Maintenance and merging of execution graph structures S8: Perform global optimization on the image positions within the graph structure. S9: Output the stitching result under unified coordinates based on the optimization results.

[0024] The method described in this application can be applied to pathological slide image stitching scenarios.

[0025] Specifically, step S1: acquiring local field-of-view images and their initial position information, specifically includes: a digital slice scanner continuously scans the slide sample according to a preset scanning path to obtain multiple local images. While acquiring each local image, the initial estimated position corresponding to that image is recorded. The initial estimated position can be obtained from the displacement information of the scanning platform, the position of the stage, or other position parameters related to the scanning process, and is used to characterize the approximate spatial position of the image in the scanning coordinate system. Each local field-of-view image can be considered as a basic processing unit in the subsequent stitching process.

[0026] In one embodiment, step S2: characterizing the local field-of-view image, specifically including: characterizing the acquired local image to facilitate the subsequent establishment of matching relationships between images. The characterization process can be implemented in different ways as needed, for example: 1. Convert the image to grayscale to obtain a grayscale image suitable for local region matching; 2. Perform necessary preprocessing on the image to reduce the impact of noise, background, or brightness fluctuations on the matching results.

[0027] This invention does not strictly limit the specific representation method used, as long as it can be used for matching solutions in subsequent overlapping regions.

[0028] In one embodiment, step S3: filtering candidate neighboring images based on initial position information specifically includes: for the image to be included in the stitching process, filtering candidate images that may be spatially adjacent to it from the processed image set according to its initial estimated position. Preferably, the spatial distance of existing images is determined by taking the initial position of the current image as the center and combining the width, height, and preset neighborhood range of a single image. Only when two images fall within the preset neighborhood range in both the width and height directions are they determined to be candidate neighboring images. That is, the current image falls within the preset neighborhood range of the candidate neighboring image in both the width and height directions, and the candidate neighboring image falls within the preset neighborhood range of the current image in both the width and height directions.

[0029] This step allows for full utilization of the initial position information naturally present during the scanning process, eliminating a large number of obviously non-adjacent images in advance, thereby avoiding indiscriminate matching of all images and significantly reducing the overall computational load.

[0030] In one embodiment, step S4: calculating the theoretical overlap region between candidate images specifically includes: for the current image and each candidate adjacent image, calculating the theoretical overlap region range of the two images based on their initial estimated positions and field of view sizes. Specifically, the upper boundary, lower boundary, left boundary, and right boundary of the possible overlap region can be obtained based on the estimated positions of the two images in a unified coordinate system, thereby forming a corresponding local rectangular region. If the calculated width or height of the theoretical overlap region is less than a preset threshold, it is considered that there is no effective overlap relationship between the two, and the matching process is no longer performed.

[0031] To improve the robustness of actual matching, the theoretical overlapping region can be appropriately extended to obtain an extended local region for matching. By introducing a certain degree of boundary redundancy, the adverse effects of scanning platform position errors on subsequent local matching can be mitigated.

[0032] In one embodiment, step S5: performing matching within the locally overlapping region to determine the relative displacement relationship between images specifically includes: for the extended local region between the current image and the candidate adjacent image (if there is no extension, it is the theoretical overlapping region), performing local matching to determine the relative displacement relationship between the two images. The local matching can be implemented in various ways, for example: First, feature point-based matching method: Feature points and their descriptive information are extracted from two local regions, the feature points are matched, and the relative displacement relationship between the two images is estimated based on the matching results.

[0033] Second, a matching method based on regional grayscale similarity: Similarity calculations are performed on local regions of two images within a limited search range to determine the optimal alignment position, thereby obtaining the relative displacement relationship between the two images.

[0034] Preferably, a valid adjacency relationship is determined between the candidate neighboring image and the current image only when the local matching result meets the preset reliability conditions.

[0035] The reliability conditions may include: the number of matching points reaching a threshold, the local similarity score reaching a threshold, and the estimated displacement result being stable.

[0036] The output of this step is: the effective relative displacement constraint between the current image and the candidate neighboring images.

[0037] In one embodiment, step S6: establishing a relationship graph structure among multiple images, specifically includes: using the effective relative displacement constraints obtained in step S5 to construct the image relationship graph structure. In this graph structure: 1. Each local image is treated as a node; 2. The effective relative displacement relationship between any two images is taken as an edge; 3. Record the relative displacement constraints and matching reliability between corresponding image pairs in the edge.

[0038] If a valid constraint is established between the current image and a candidate neighboring image, a connection edge is established between them; if a valid constraint is established between the current image and multiple candidate neighboring images, it means that the current image can be connected to multiple existing local stitching regions at the same time, and these connection relationships should be included in the same graph structure; if the current image does not have a valid constraint with any existing candidate neighboring image, then the image is used as the starting point of a new independent subgraph.

[0039] Through the above method, the present invention can receive images frame by frame during the scanning process and dynamically maintain the graph structure, thereby realizing incremental stitching relationship construction.

[0040] In one embodiment, step S7: performing graph structure maintenance and merging specifically includes: during the graph structure construction process, the graph structure can also be dynamically maintained. Specifically, this includes: 1. When the current image establishes a connection with an existing graph structure, the image is merged into the corresponding graph structure; 2. When the current image establishes connections with multiple existing graph structures simultaneously, these existing graph structures are merged. 3. When certain images are spatially significantly far from the current scanning area, their activity level can be reduced or state management can be implemented to minimize unnecessary repetitive processing. Specifically, if the distance between certain images and the current scanning area is greater than the width and height distance of n images, they can be considered significantly far apart, i.e., (x1-x2)>nx*w, (y1-y2)>ny*h, where nx is the width direction (nx=w / step_x+1), and ny is the height direction (n), denoted as ny = h / step_y+1. step_x and step_y are the scanning intervals. Then, according to the scanning rules (scanning line by line), subsequent scanning areas will only move further away from this image, so subsequent scanning areas will not overlap with these images. Reducing their activity level or implementing state management specifically includes: marking this image so that it can be skipped directly when determining whether it overlaps or matches later. For example, if the distance between images a and b in the five images {a, b, c, d, e} and the previous scan area is greater than n width and height, then when searching for images adjacent to the current scan area, a and b can be skipped directly.

[0041] Through the above-mentioned graph structure maintenance mechanism, the present invention can not only handle continuous image stitching under regular scanning paths, but also adapt to complex acquisition situations such as cross-line scanning, partial retracement, or partial repeated coverage.

[0042] In one embodiment, step S8, which involves globally optimizing the image positions in the graph structure, specifically includes: after establishing local relative displacement constraints between multiple images, performing global optimization on the entire graph structure to uniformly solve for the final positions of each image in the global coordinate system. During global optimization, the initial estimated positions of each image node are used as initial optimization values, and the relative displacement constraints between image nodes are used as constraints in the optimization objective. Through iterative solving, the positions of all images are made to reach an optimal and consistent state overall. Global optimization achieves the following effects: 1. Make full use of the redundant constraints between multiple images; 2. Reduce the impact of single local matching errors on the overall results; 3. Suppress image drift caused by cumulative errors of the scanning platform; 4. Improve the overall geometric consistency when splicing large areas.

[0043] Compared to methods that rely solely on local registration of neighboring images, this global optimization step significantly improves the spatial consistency of the entire digital pathology image.

[0044] In one implementation, step S9, outputting the stitching result under a unified coordinate system based on the optimization result, specifically includes: after global optimization is completed, obtaining the final position of each local image in a unified coordinate system. Subsequently, all images can be uniformly arranged according to the final positions, and subsequent image fusion, rendering, or output processing can be performed to generate the entire digital pathology image.

[0045] In one embodiment, the present invention can directly output the final position coordinates of each image for subsequent image stitching module to perform image synthesis.

[0046] In another implementation, a fusion process can be further combined with the above to directly output the final stitched image.

[0047] It should be noted that: The "width and height of a single image" in this application are not necessarily limited to either actual physical distance or pixel value; their essential meaning is the coverage size of a single local field-of-view image in a unified stitching coordinate system. If the initial position output by the scanning device is a physical quantity such as stage displacement or scanning platform coordinates, it can be converted to pixel coordinates or equivalent sampling coordinates according to the scanning resolution or calibration ratio; if the system directly records the initial position in pixels, then the image width and height are the pixel width and pixel height.

[0048] For example, in one implementation, the local image size can be expressed as: , in, This represents the number of pixels along the width of the image. This represents the number of pixels in the image's height direction. If physical coordinates are used, it can be expressed as: , in, , These are the physical resolutions in the width and height directions, respectively. In other words, as long as the initial position, image width, height, and preset neighborhood range are in the same coordinate system and within the same unit, they can be used for subsequent spatial relationship determination.

[0049] This application does not simply rely on the initial position to directly determine the final adjacency relationship. Instead, it first uses the initial position, image width, height, and a preset neighborhood range to filter "candidate adjacent images," and then confirms the existence of valid adjacency relationships through local overlapping region matching. This step differs from the prior art in that it directly maps the hardware position to the canvas. In this application, the initial position is only used as a spatial prior to reduce invalid matching objects.

[0050] Specifically, for the i-th image, its initial estimated position can be denoted as: , If the width and height of a single image are W and H respectively, then the theoretical coverage area of ​​the image in a unified coordinate system can be represented as: , For the current image i and the existing image j, let the initial position difference between them be: , Let the preset neighborhood ranges in the width and height directions be respectively. , When the following conditions are met, for example: , , Furthermore, the two are located near adjacent or overlapping zones in at least one direction, for example: , If two images are spatially adjacent or overlapping, they can be considered as candidate adjacent images. , It is used to absorb scanning platform positioning errors, mechanical backlash, sampling errors, and boundary redundancy required for image matching.

[0051] Therefore, the reason why the initial position combined with width and height can be used for candidate adjacency determination is that each local image is not a point without size, but occupies a rectangular region in a unified coordinate system determined by the initial position and the field of view size. If the distance between two rectangular regions in the width or height direction exceeds the size of a single image and the preset neighborhood range, then the two cannot have a valid overlap or adjacency relationship and can be eliminated in advance; otherwise, it only indicates that the two have a candidate relationship, and further matching confirmation is still needed.

[0052] For candidate adjacent images, this application further calculates the theoretical overlapping region. Two images , The theoretical overlapping region boundary can be represented as: , , , , When the following conditions are met: , If a preset minimum overlap threshold is met, it indicates that the two have a theoretical overlap area. To compensate for the initial position error, the overlap area can be expanded outwards from the theoretical overlap area. , This forms an extended local region for matching. Subsequently, gray-level similarity matching, phase correlation matching, or feature point matching are performed within this local region to obtain the relative displacement between the two images. , and matching reliability Only when A valid adjacency relationship is established only when the preset reliability conditions are met. Therefore, the candidate selection in S3 is not equivalent to the final adjacency determination; the final adjacency relationship is confirmed by the local matching results in S5.

[0053] Regarding "how to achieve image position optimization through iterative solution of relative displacement of image nodes and constraint set", the implementation logic of this application is as follows: Each image is treated as a node in the graph structure, and the node state represents the position of that image to be optimized in a unified coordinate system: , Alternatively, when rotation needs to be considered, it can be extended to a two-dimensional rigid body pose: , Use the effective relative displacement obtained through local matching between any two images as edge constraints: , The weight of an edge is determined by the reliability of the matching, and can be expressed as: , Where I is the identity matrix, The larger the value, the more reliable the relative displacement constraint is, and the higher its weight in the optimization.

[0054] The goal of global optimization is to ensure that all node positions simultaneously satisfy multiple relative displacement constraints. Taking a translation model as an example, the following least-squares objective function can be established: , Where E is the set of edge constraints formed by all valid adjacent image pairs. If a two-dimensional pose model is used, it can be written as: , , Where ρ(·) can be a robust kernel function, used to reduce the impact of abnormal matching constraints on the overall result.

[0055] During optimization, the initial estimated positions of each image are used as the initial values ​​for the nodes: , A reference node or stable subgraph is fixed to eliminate the overall translational degrees of freedom. Then, the node positions are updated using nonlinear least-squares iterative methods such as the Gauss-Newton method or the Levenberg-Marquardt method. , The process is iterated repeatedly until the maximum number of iterations is reached or the error converges. After optimization, the position of each node is the final globally consistent position of multiple images in a unified coordinate system.

[0056] The method described in this application is applicable to multiple local images obtained when a digital slide scanner scans a glass slide sample. It can utilize the initial spatial position information of each image during the scanning process to filter images that may be spatially adjacent, establish image matching constraints only in local areas with effective overlap, and further construct a unified graph structure for the relative constraint relationships between multiple images. By globally optimizing the solution, the final position of each image in the unified coordinate system is obtained, thereby suppressing the cumulative drift caused by local matching errors and improving the overall geometric consistency, stability, and reliability of the entire digital pathology image stitching result.

[0057] Unlike existing methods that primarily focus on local registration and fusion of adjacent image pairs, this invention considers not only the local matching relationships between adjacent images but also integrates the relationships between multiple images into a unified global constraint framework for solution. This makes it more suitable for pathological image stitching in large-scale, continuous scanning scenarios. This application employs a processing chain of "initial position prior screening of candidate images, matching of locally overlapping regions to generate effective relative displacement constraints, and global optimization based on graph structure." This chain can improve candidate screening efficiency by utilizing the initial position provided by the scanning device, while simultaneously correcting initial position errors through local matching and global constraint optimization, suppressing the accumulation of local matching errors, thereby obtaining a more stable stitching result.

[0058] Embodiments of this application also provide a computer storage medium storing at least one executable instruction that causes a processor to perform operations corresponding to the image stitching method described above.

[0059] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0060] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. An image stitching method, characterized in that, include: S1: Obtain the local field of view image and its initial position information. S2: Perform characterization processing on the local field of view image. S3: Filter candidate neighboring images based on initial location information. S4: Calculate the theoretical overlap region between the image and the candidate neighboring image. S5: Perform matching within locally overlapping regions to determine the effective relative displacement constraints between images. S6: Based on the effective relative displacement constraints between images, establish a relationship graph structure between multiple images. S7: Maintenance and merging of execution graph structures S8: Perform global optimization on the image positions within the graph structure. S9: Output the stitching result under unified coordinates based on the optimization results.

2. The image stitching method according to claim 1, characterized in that, In step S3, the selection of candidate neighboring images based on initial position information specifically includes: taking the initial position of the current image as the center, combining the width and height of a single image with a preset neighborhood range, performing spatial distance judgment on existing images, and only when two images fall within the preset neighborhood range in both the width and height directions, they are judged as candidate neighboring images.

3. The image stitching method according to claim 1, characterized in that, In step S4, calculating the theoretical overlap region between the current image and each candidate neighboring image specifically includes: for the current image and each candidate neighboring image, calculating the theoretical overlap region between the two images based on their initial estimated positions and field of view sizes.

4. The image stitching method according to claim 3, characterized in that, In step S4, the theoretical overlapping region is extended to obtain an extended local overlapping region for matching.

5. The image stitching method according to claim 1, characterized in that, In step S5, only when the local matching result meets the preset reliability condition is it determined that there is a valid adjacent relationship between the candidate adjacent image and the current image, and the output result is: the valid relative displacement constraint between the current image and the candidate adjacent image.

6. The image stitching method according to claim 1, characterized in that, In S6, within this diagram structure: Each image serves as a node. The effective relative displacement constraint between any two images is taken as an edge. The edges record the effective relative displacement constraints between corresponding image pairs and the reliability of the matching.

7. The image stitching method according to claim 1, characterized in that, In S7, the maintenance and merging of the execution graph structure specifically includes: When the current image establishes a connection with an existing graph structure, the image is incorporated into the corresponding graph structure. When the current image establishes connections with multiple existing graph structures simultaneously, these existing graph structures are merged. When certain images are spatially distant from the current scan area, their activity level is reduced or their status is managed.

8. The image stitching method according to claim 1, characterized in that, Specifically, S8 includes: during global optimization, the initial estimated position of each image node is used as the initial value for optimization, and the relative displacement constraint between image nodes is used as the constraint term in the optimization objective. Through iterative solution, the positions of all images are made to reach the optimal consistency state as a whole.

9. The image stitching method according to claim 1, characterized in that, In S9, the stitching result includes the final position coordinates of each image or the final stitched image. In S2, the characterization processing of the local field of view image includes grayscale processing of the image.

10. A computer storage medium, characterized in that, The computer storage medium stores at least one executable instruction that causes the processor to perform the operation corresponding to the image stitching method as described in any one of claims 1 to 9.

Images