Image stitching methods, apparatus and equipment

Abstract

This invention relates to the field of image processing technology and discloses an image stitching method, apparatus, and device. The method includes: identifying multiple target structures in images to be stitched; determining a homography matrix using target feature points of the multiple target structures in different images to be stitched; the target feature points are determined based on the geometric shape of the target structures; performing a perspective transformation on one of the images to be stitched using the homography matrix to obtain a distorted image; and stitching the distorted image with another image to be stitched, which serves as a reference image. This invention can improve the accuracy of feature point recognition and registration, thereby ensuring the quality of image stitching.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and specifically to an image stitching method, apparatus, and device. Background Technology

[0002] In image stitching, homography transformation is a common model for distorting images, encompassing translation, rotation, scaling, and viewpoint transformation. Traditional image stitching methods utilize manually defined feature detection methods to calculate the homography matrix. The core idea behind these methods is to design optimal features (points, lines, or energy functions) to achieve image alignment. However, due to the weak features and repetitive textures in some images, such as spinal X-ray images captured by a C-arm X-ray machine (intraoperative acquisition of spinal images is generally achieved using small to medium-sized, easily portable C-arm X-ray machines; however, limited by their finite field of view (FOV), surgeons can only obtain image slices of the spine, thus requiring image stitching techniques to combine these slices into a panoramic image), accurate identification and matching of feature points becomes particularly difficult. The homography matrix used for perspective transformation during image stitching is determined based on feature points; that is, the accuracy of the homography matrix directly determines the effectiveness of the perspective transformation. If feature point identification and matching are inaccurate, the homography matrix will be imprecise, leading to misalignment between the distorted image and the reference image, thus affecting the stitching quality. Summary of the Invention

[0003] In view of this, the present invention provides an image stitching method, apparatus and device to solve the problem of low stitching quality caused by the difficulty in accurately identifying and matching feature points in image stitching.

[0004] In a first aspect, the present invention provides an image stitching method, the method comprising:

[0005] Identify multiple target structures in the image to be stitched;

[0006] The homography matrix is ​​determined by using target feature points of multiple target structures in different images to be stitched; the target feature points are determined based on the geometry of the target structures.

[0007] A perspective transformation is performed on one of the images to be stitched together using the homography matrix to obtain a distorted image;

[0008] The distorted image is stitched together with another image to be stitched, which serves as the reference image.

[0009] In one alternative implementation, the target structure is a pedicle screw.

[0010] In one alternative implementation, identifying multiple target structures in the image to be stitched includes:

[0011] Using a target neural network model, multiple target structures in the image to be stitched are identified separately;

[0012] The target neural network model includes at least: a patch embedding layer, an encoder, a decoder, and a parameterless attention module;

[0013] The patch embedding layer is used to divide the image to be stitched into multiple image regions;

[0014] The encoder includes a first VSS block module and a patch merge layer. Multiple image regions divided by the patch embedding layer are input to the first VSS block module. The first VSS block module is used to extract features from the image regions respectively. The extracted feature information is input to the patch merge layer for downsampling.

[0015] The decoder includes a patch expanding layer and a second VSS block module; the patch expanding layer is used for upsampling, and the second VSS block module is used for feature extraction from the output of the patch expanding layer.

[0016] The feature information downsampled by the patch merge layer in the encoder is passed to the decoder through a parameterless attention module.

[0017] In one optional implementation, the homography matrix is ​​determined using target feature points of multiple target structures in different images to be stitched, including:

[0018] Obtain the initial homography matrix;

[0019] A registration energy function is constructed, which is the sum of multiple first distances. Each first distance is the distance between a first target feature point in the first image to be stitched and a second target feature point in the distorted second image to be stitched. The first and second images to be stitched are two separate images. The first and second target feature points are target feature points within the overlapping portion of the first and distorted second images, with the second target feature point being the closest target feature point to the first target feature point in the distorted second image. The distorted second image to be stitched is obtained by transforming the second image using the initial homography matrix.

[0020] The initial homography matrix is ​​optimized by minimizing the value of the registration energy function, thus obtaining the optimal homography matrix.

[0021] In one optional implementation, the homography matrix is ​​determined using target feature points of multiple target structures in different images to be stitched, including:

[0022] If there are three or more images to be stitched, then obtain the minimum registration energy function value between each pair of images to be stitched.

[0023] For an image to be stitched, the image with the smallest minimum registration energy function value is taken as the adjacent image to be stitched.

[0024] The homography matrix corresponding to the minimum registration energy function value between two adjacent images to be stitched is determined as the homography matrix between the two adjacent images to be stitched.

[0025] In one optional implementation, the homography matrix is ​​determined using target feature points of multiple target structures in different images to be stitched, including:

[0026] If there are three or more images to be stitched, then obtain the homography matrix between each pair of adjacent images to be stitched.

[0027] Using one of the images to be stitched as the reference image, the homography matrix of the third image to be stitched relative to the reference image is determined by multiplying the homography matrices between two adjacent images to be stitched; the third image to be stitched is an image other than the reference image and its adjacent images to be stitched.

[0028] In one alternative implementation, stitching the distorted image with another image to be stitched, which serves as a reference image, includes:

[0029] Construct a hybrid energy function that includes pixel differences, gradient differences, and depth feature differences between the distorted image and the reference image in grayscale space;

[0030] Initialize the energy at the starting boundary of the seam;

[0031] Extend the path from different starting points to the other end of the image. Each time the path is extended, select the pixel with the smallest mixed energy function value as the next path node until the boundary of the overlapping part of the distorted image and the reference image is reached to obtain the optimal path.

[0032] Backtrack along the optimal path to determine the best seam.

[0033] The distorted image and the reference image are stitched together according to the optimal stitching seam.

[0034] In one alternative implementation, stitching the distorted image with another image to be stitched, which serves as a reference image, includes:

[0035] Determine the transition region on both sides of the seam between the distorted image and the reference image;

[0036] The pixel values ​​of the distorted image and the reference image within the transition region are weighted and summed to obtain the pixel values ​​of the stitched image.

[0037] In a second aspect, the present invention provides an image stitching device, the device comprising:

[0038] The target structure recognition module is used to identify multiple target structures in the image to be stitched together;

[0039] The homography matrix determination module is used to determine the homography matrix using target feature points of multiple target structures in different images to be stitched together;

[0040] The transformation module is used to perform perspective transformation on one of the images to be stitched using the homography matrix, resulting in a distorted image.

[0041] The stitching module is used to stitch a distorted image with another image to be stitched, which serves as the reference image.

[0042] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the image stitching method described in the first aspect or any corresponding embodiment thereof.

[0043] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the image stitching method described in the first aspect or any corresponding embodiment thereof.

[0044] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the image stitching method described in the first aspect or any corresponding embodiment thereof.

[0045] The image stitching method, apparatus, and device provided in this embodiment first identify multiple target structures in the image to be stitched, then determine the homography matrix based on the target feature points of the identified target structures, and transform the image to be stitched based on the determined homography matrix. Finally, the transformed distorted image is stitched with the reference image. In this process, since the target feature points are determined based on the geometry of the target structures, their determination accuracy and matching accuracy are guaranteed, thus ensuring the calculation accuracy of the homography matrix and consequently guaranteeing the image stitching quality. Furthermore, compared to methods that use manually defined feature detection methods to calculate the homography matrix, manually designed features exhibit lower robustness due to their complex design. In contrast, the target feature point determination method in this embodiment is simple, resulting in a highly robust and versatile image stitching method. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0047] Figure 1 This is a flowchart illustrating an image stitching method according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the architecture of a target neural network model according to an embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of the feature processing process of SimAM according to an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the registration of target feature points within the overlapping portion of two images to be stitched according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of the fusion process of stitching together truncated spinal X-ray images according to an embodiment of the present invention;

[0052] Figure 6 This is a schematic diagram of the overall process of stitching X-ray images of spinal transection using the image stitching method in an embodiment of the present invention;

[0053] Figure 7 This is a structural block diagram of an image stitching device according to an embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0056] Regarding the stitching of spinal X-ray images taken by a C-arm X-ray machine, the following solutions have been proposed in related technologies:

[0057] I. A common X-ray ruler used in orthopedics is used to calculate the relative positions between images through the scale on the ruler, thus achieving image stitching.

[0058] 2. Use a non-perspective panel placed below the patient and use the absolute coordinates on the panel to stitch the images together.

[0059] Third, by installing a camera and a double-sided mirror on the C-arm, the optical center of the camera is made to coincide with the focal point of the X-ray source, thereby realizing the fusion of X-ray images and video images.

[0060] However, the methods mentioned above rely on external labels or structures, which often increases operational costs. In addition, some methods utilize deep learning-based stitching strategies, such as an end-to-end long bone image stitching scheme. This scheme uses convolutional neural networks (CNNs) to reconstruct two-dimensional images from multiple images, employing the Structural Similarity Index Measure (SSIM) and adversarial loss to force the network to generate visually similar images to the ground truth. However, this method is currently only designed for the femur and has not yet been extended to other bone types.

[0061] According to an embodiment of the present invention, an image stitching method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of executable computer instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0062] This embodiment provides an image stitching method that can be used in various computer devices, including medical devices. Figure 1 This is a flowchart of an image stitching method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0063] Step S101: Identify multiple target structures in the image to be stitched.

[0064] In this embodiment of the invention, there are no restrictions on the image to be stitched and the target structure. The image to be stitched can be a spinal X-ray image (also known as a spinal X-ray image) or other images. The target structure can be a pedicle screw or other target structures, depending on the specific structure in the image to be stitched.

[0065] The images to be stitched together must be at least two. The target structure within each image needs to be identified. For example, image segmentation methods such as deep learning models can be used to segment the target structure within the images to be stitched together, obtaining a target structure mask.

[0066] In some optional implementations, step S101, namely identifying multiple target structures in the image to be stitched, includes:

[0067] Step S1011: Using the target neural network model, identify multiple target structures in the image to be stitched.

[0068] Among them, such as Figure 2 As shown, the target neural network model includes at least: a block embedding (or patch embedding) layer, an encoder, a decoder, and a parameterless attention module.

[0069] The patch embedding layer is used to divide the image to be stitched (with dimensions of H×W×3) into multiple image regions (patches); for example, dividing it into 4×4 non-overlapping image regions (patches).

[0070] In this embodiment of the invention, features are calculated independently for each image region (patches), and the number of channels of the image is mapped to C.

[0071] The encoder includes a first VSS block module and a patch merge layer. Multiple image regions divided by the patch embedding layer are input to the first VSS block module. The first VSS block module is used to extract features from each image region. The extracted feature information is input to the patch merge layer for downsampling and to expand the feature channels to twice the original size.

[0072] The encoder may include, for example, four sets of first VSS block modules and patch merge layers, each set of which may include two first VSS block modules.

[0073] The decoder includes a block expanding (also known as patch expanding) layer and a second VSS block module; the patch expanding layer is used for upsampling, and the second VSS block module is used for feature extraction from the output of the patch expanding layer.

[0074] The decoder may include, for example, four sets of patch expanding layers and a second VSS block module, each set of which may include two second VSS block modules.

[0075] The feature information downsampled by the patch merge layer in the encoder is passed to the decoder through a parameterless attention module. In this embodiment of the invention, the feature information downsampled by the encoder is not directly passed to the decoder, but is passed to the decoder through a lightweight parameterless attention module (Simple Attention Module, SimAM) to further focus on the key features of the model.

[0076] The decoder is followed by a projection layer, which restores the feature size to the original number of channels.

[0077] In this embodiment of the invention, to achieve accurate segmentation of the target structure (e.g., pedicle screw) region, a neural network is constructed to perform the semantic segmentation task. Specifically, the VM-UNet (Vision Mamba UNet) network architecture is adopted, and a lightweight attention module, SimAM, is integrated. The VM-UNet network architecture is a novel UNet architecture based on a state-space model, which can not only capture extensive contextual information but also maintain linear computational complexity, providing an efficient solution for image segmentation.

[0078] In this embodiment of the invention, the VSS block module (including a first VSS block module and a second VSS block module) each comprises two branches. One branch includes a linear layer, a depthwise separable convolution, an activation function, and a 2D selective scan (SS2D) module for further feature extraction from the input. The other branch includes a linear layer and an activation function. The outputs of the two branches are combined element-wise through a product, and then a linear layer is used to blend the features. Finally, the combined features are added to the residual connections to form the output of the VSS block module.

[0079] In this embodiment of the invention, SimAM can generate three-dimensional attention weights by calculating the local self-similarity of feature maps. Specifically, SimAM defines a neural energy function to adaptively estimate the importance of each neuron, and this neural energy function is:

[0080]

[0081] Where x i And t is x∈R C×H×W The input and output of a single channel, ω t It is the weight, M = H × W.

[0082] The model assumes that all pixels in a single channel follow the same distribution. Therefore, calculating the mean and variance of neurons in a single channel can be applied to all channels, and the final minimum energy function can be expressed as:

[0083]

[0084] in,

[0085] Each neuron can calculate its attention weights. All channels are set to E, and finally activated using sigmoid(·), as shown in the following formula:

[0086]

[0087] Where i represents the feature activated after the i-th encoding stage, which is then used as a downsampled feature and the feature passed through the VSS block in the decoder stage is added to achieve a skip connection. The connected feature is then input into Patch Expanding for upsampling.

[0088] like Figure 3 The diagram shows how SimAM generates 3D attention weights. The cube represents the feature map H*W*C. Attention weights are generated during the Fusion stage, while Expansion refers to recalculating the weighted feature values ​​within the feature channels.

[0089] This invention proposes a novel Unet architecture—VMS-Unet—which is a U-shaped network based on the Mamba state-space model and integrates the SimAM attention mechanism, enabling efficient semantic segmentation of images to be stitched together. This segmentation method exhibits good robustness in weakly textured scenes and is applicable to images of different resolutions.

[0090] Step S102: Determine the homography matrix using target feature points from multiple target structures in different images to be stitched. The target feature points are determined based on the geometry of the target structures.

[0091] Specifically, a target structure can have one or more target feature points; this is not limited here. If a target structure has only one target feature point, that feature point can be the geometric centroid, geometric center, etc., of the target structure; this is also not limited here. In this embodiment of the invention, the subsequent determination of the homography matrix is ​​based on the target feature point.

[0092] In one specific embodiment, the target feature points of the target structure are the geometric centroids of the target structure. After obtaining the mask of the target structure through image segmentation, the geometric centroids of the target structure can be determined based on the mask. Specifically, the geometric centroid of the l-th target structure... The calculation method could be, for example:

[0093]

[0094] Among them, A l Let be the area of ​​the mask (polygon) for the target structure. i represents the vertex coordinates of the segmented target structure mask. These represent the coordinates of the i-th vertex of the mask for the l-th target structure.

[0095] In this embodiment of the invention, to ensure that identical screws in two images (I1, I2) to be stitched together are precisely aligned, the non-reference image I2 can be distorted using a homography matrix H. The homography matrix H is 3*3, homogenizing the coordinates of each pixel in the non-reference image I2 to P = (x, y, 1), with a shape of 3*1. The new coordinates of each pixel are P′ = H·P. Finally, all pixels are mapped to new positions, forming the distorted image I′2. Not all positions in I′2 correspond to the original pixels; pixels without corresponding positions are generated using bilinear interpolation.

[0096] In some optional implementations, step S102, which involves determining the homography matrix using target feature points of multiple target structures in different images to be stitched, includes:

[0097] Step S1021: Obtain the initial homography matrix.

[0098] Step S1022: Construct a registration energy function, which is the sum of multiple first distances. One first distance is the distance between a first target feature point in the first image to be stitched I1 (as the reference image) and a second target feature point in the distorted (i.e., homography-transformed) second image to be stitched I2′; for example... Figure 4 As shown, the first image to be stitched I1 and the second image to be stitched I2 are two images to be stitched; the first target feature point and the second target feature point are target feature points within the overlapping part 401 (the part within the red box) of the first image to be stitched and the distorted second image to be stitched, and the second target feature point is the target feature point in the distorted second image to be stitched that is closest to the first target feature point; the second image to be stitched is transformed using the initial homography matrix to obtain the distorted second image to be stitched.

[0099] In this embodiment of the invention, a new registration energy function is introduced to optimize the homography matrix H, thereby minimizing the differences between the two images to be stitched together. This new registration energy function is expressed by the formula:

[0100]

[0101] Where i and j represent the indexes of the target feature points in the two images to be stitched, and H(·) is the homography matrix. Let I1 and I2 represent the coordinates of the i-th target feature point in the image to be stitched, and the coordinates of the j-th target feature point in the image to be stitched, respectively. I1∪H(I2) represents the overlapping portion between the image to be stitched, I1, and the image to be stitched after homography transformation, H(I2) (i.e., I′2). This indicates that the i-th target feature point in the image to be stitched and the j-th target feature point in the image to be stitched belong to the overlapping part. This represents the distance between the i-th target feature point in the image to be stitched I1 and the j-th target feature point in the distorted image to be stitched I′2.

[0102] In this embodiment of the invention, it is necessary to calculate the distance between each target feature point in the overlapping part of the image I1 to be stitched and each target feature point in the overlapping part of the image H(I2) to be stitched after homography transformation. Then, for each target feature point in the overlapping part of the image I1 to be stitched, the minimum distance (the distance between the target feature point in the overlapping part of the image H(I2) to be stitched after homography transformation) is selected and summed to obtain the value of the registration energy function, thereby achieving the highest degree of alignment.

[0103] Step S1023: The initial homography matrix is ​​optimized by minimizing the value of the registration energy function to obtain the optimal homography matrix.

[0104] Specifically, embodiments of the present invention optimize the homography matrix by minimizing the value of the registration energy function, so as to achieve registration between images to be stitched using the optimal homography matrix.

[0105] In some optional implementations, there may be more than two images to be stitched together, and the order of the images is uncertain. That is, it is necessary to stitch together multiple unordered images (three or more). In this case, step S102, which uses the target feature points of multiple target structures in different images to be stitched together to determine the homography matrix, includes:

[0106] Step S102a: If there are three or more images to be stitched, the minimum registration energy function value between each pair of images to be stitched is obtained. The process for obtaining the minimum registration energy function value between each pair of images to be stitched is described above.

[0107] Step S102b: For an image to be stitched, the image with the smallest minimum registration energy function value is selected as the adjacent image to be stitched.

[0108] Step S102c: The homography matrix corresponding to the minimum registration energy function value between two adjacent images to be stitched is determined as the homography matrix between the two adjacent images to be stitched.

[0109] When the stitching process is extended to multiple unordered images, the primary task is to determine the stitching order between the images, i.e., to solve the pairing problem among multiple images. To achieve this goal, this embodiment assumes that when all images are arranged in the correct order, the energy function L for pairwise registration between them is... Registration This will reach a minimum. Based on this assumption, the cumulative sum of the registration energy function will also be minimized. Therefore, the optimization objective of this embodiment is to minimize this cumulative sum, where i and j represent the image indices, respectively. During the optimization process, each image can be considered as a node in the graph, and the registration energy function value between image pairs can be used as the distance between nodes. After selecting an image (a node), it is necessary to select the next node that has the smallest registration energy function value (i.e., the closest distance).

[0110] In this embodiment, when determining the order of multiple images to be stitched together, it is not necessary to check all possible image combinations. Instead, the concept of a minimum path algorithm is adopted, treating each image as a node in the graph and using the registration energy function value between image pairs as the distance between nodes. In this way, the problem of optimizing the total energy function is transformed into finding the shortest path. Therefore, a greedy algorithm-based strategy can be adopted: after selecting an image (a node), the image with the smallest registration energy function value (i.e., the closest distance) is selected as the next node. After determining the image order, the homography matrix of adjacent images is calculated using the registration energy function.

[0111] In some optional implementations, when stitching multiple images, only one image to be stitched can be used as the reference image (also called the reference image), while the other images to be stitched are used as non-reference images, requiring homography transformation. In this case, step S102, which involves determining the homography matrix using target feature points of multiple target structures in different images to be stitched, includes:

[0112] If there are three or more images to be stitched, then obtain the homography matrix between each pair of adjacent images to be stitched.

[0113] Using one of the images to be stitched as the reference image, the homography matrix of the third image to be stitched relative to the reference image is determined by multiplying the homography matrices between two adjacent images to be stitched; the third image to be stitched is an image other than the reference image and its adjacent images to be stitched.

[0114] For example, when dealing with multiple truncated spinal X-ray images, the topmost spinal X-ray image can be used as the reference image. Third image to be stitched: I k The transformation can be calculated based on cumulative homography:

[0115]

[0116] Where i and j are the sequence numbers of two adjacent images after the order is determined, that is, j = i + 1.

[0117] Step S103: Use the homography matrix to perform perspective transformation on one of the images to be stitched to obtain a distorted image.

[0118] Specifically, perspective transformation can be homography transformation, which utilizes the homography matrix for transformation. The images to be stitched together that require transformation are non-reference images.

[0119] Step S104: The distorted image is stitched together with another image to be stitched, which serves as the reference image.

[0120] In some optional implementations, step S104, which involves stitching the distorted image with another image to be stitched, serving as a reference image, includes:

[0121] Step S1041: Construct a hybrid energy function, which includes pixel differences, gradient differences, and depth feature differences between the distorted image and the reference image in the grayscale space.

[0122] Specifically, based on the pixel differences between the distorted image and the reference image in the grayscale space, the color difference energy E can be obtained. ρ (u, v):

[0123] E ρ =(u,v)=(I1(u,v)-I2(u,v)) 2

[0124] Where u,v are pixel coordinates, I1(u,v) is the pixel value of the reference image I1 at (u,v), and I2(u,v) is the pixel value of the distorted image I2 at (u,v). E ρ This is a feature map with the same size as images I1 and I2.

[0125] Based on the gradient difference between the distorted image and the reference image in grayscale space, the geometric structure energy E can be obtained. δ (u, v):

[0126] E δ (u, v) = (Δ1 (u, v)-Δ2 (u, v)) 2

[0127] Where Δ1(u,v) is the sum of squares of the gradients in the x and y directions of image I1 at (u,v) (which is the sum of squares of the subtraction of pixels in the x and y directions), and Δ2(u,v) is the sum of squares of the gradients in the x and y directions of image I2 at (u,v).

[0128] Based on the difference in depth features between the distorted image and the reference image in grayscale space, the feature energy can be obtained.

[0129]

[0130] Where Φ1(u,v) represents the feature of image I1 at (u,v), and Φ2(u,v) represents the feature of image I2 at (u,v). For example, the 24th layer of a pre-trained ResNet50 can be used as the semantic content representation (i.e., features) of the image.

[0131] The mixed energy function is:

[0132]

[0133] Where λ is the weighting factor for each energy level.

[0134] Step S1042: Initialize the energy of the starting boundary of the seam. The starting boundary can be the left boundary.

[0135] Step S1043: Extend the image to the other end from different starting points (e.g., each pixel at the boundary of the starting point). Each time the image is extended, select the pixel with the smallest mixed energy function value as the next path node until the boundary of the overlapping part of the distorted image and the reference image is reached to obtain the optimal path.

[0136] Step S1044: Backtrack along the optimal path to determine the best seam.

[0137] Step S1045: Stitch the distorted image and the reference image according to the optimal stitching seam.

[0138] In this embodiment of the invention, a stitching path optimization algorithm based on a hybrid energy function is designed for the image fusion process.

[0139] In another optional implementation, step S104, which involves stitching the distorted image with another image to be stitched, serving as a reference image, includes:

[0140] Step S104a: Determine the transition regions on both sides of the seam between the distorted image and the reference image. Please refer to the above for the process of determining the seam.

[0141] Step S104b: The pixel values ​​belonging to the transition region on the distorted image and the reference image are weighted and summed to obtain the pixel values ​​of the stitched image.

[0142] Traditional fusion methods that rely on seams directly replace the original image on both sides of the seam, which may create obvious boundary lines. To eliminate these boundary lines, this invention designs a fusion splicing domain on both sides of the seam, in which the image will smoothly transition from I1 to I2.

[0143] Taking image stitching as an example, the formula for the stitching process is as follows:

[0144] I seam [:, v]=σ(kv)⊙I1[:,v]+(1-σ(kv))⊙I2[:,v],v∈[0,ξ]

[0145] Where v represents the transition area on both sides of the seam, ξ is the total length of the transition area, σ is the sigmoid function, and k is the amplification factor, which can be, for example, 10. seam The final stitched image is represented by I1 and I2, which are the baseline image and the distorted image, respectively.

[0146] In this embodiment of the invention, in the transition region of the stitching, the area closer to the stitching seam is weighted by the average weight of the stitched images, while the area farther from the stitching seam is dominated by the images they occupy. The stitching process of truncated spinal X-ray images is as follows: Figure 5 As shown. Using the image stitching method provided in this embodiment of the invention, the stitched image quality is high, achieving artifact-free and seamless stitching.

[0147] The image stitching method provided in this embodiment first identifies multiple target structures in the image to be stitched, then determines the homography matrix based on the target feature points of the identified target structures, and transforms the image to be stitched based on the determined homography matrix. Finally, the transformed distorted image is stitched together with the reference image. In this process, since the target feature points are determined based on the geometry of the target structures, their determination accuracy and matching accuracy are guaranteed, thus ensuring the calculation accuracy of the homography matrix and consequently guaranteeing the image stitching quality. Furthermore, compared to methods that use manually defined feature detection methods to calculate the homography matrix, manually designed features exhibit lower robustness due to their complex design. The target feature point determination method in this embodiment is simple, resulting in a highly robust and versatile image stitching method.

[0148] The image stitching method provided in this invention is adaptable to stitching truncated spinal X-ray images, achieving end-to-end full-length image stitching. Therefore, this image stitching method can be applied to pedicle screw implantation surgery, where the target structure can be the pedicle screw. The advantages of this method lie in the high quality of the stitched images, achieving artifact-free and seamless stitching, and its ability to rapidly process input disordered truncated spinal image slices. Its stability and speed make it very suitable for professional clinical surgical environments, supporting real-time intraoperative imaging.

[0149] This invention also proposes a novel Unet architecture—VMS-Unet—which is a U-shaped network based on the Mamba state-space model and integrates the SimAM attention mechanism, enabling efficient semantic segmentation of spinal X-ray images. This segmentation method exhibits good robustness in weakly textured scenes and is applicable to images of different resolutions.

[0150] Compared with current stitching methods based on traditional detection operators, the algorithm proposed in this invention has significant advantages in homography estimation and image stitching in real-world scenarios, and has been favored by professional clinical surgeons.

[0151] like Figure 6 As shown, the overall process for stitching X-ray images of a truncated spine using the image stitching method provided in this embodiment of the invention is as follows:

[0152] 1. Obtain multiple disordered X-ray images of spinal transection.

[0153] 2. Input the multiple disordered spinal segmentation X-ray images obtained in the previous step into the VM-Unet network including a parameterless attention module for image segmentation, identify the pedicle screws in each spinal segmentation X-ray image, and obtain the geometric centroid of the identified pedicle screws.

[0154] Third, the correct order of spinal truncated X-ray images is determined by using the principle of minimizing the minimum registration energy function value between two adjacent images. Specifically, image ordering is performed using pedicle screw registration.

[0155] IV. After determining the correct order among the truncated spinal X-ray images, use one of the truncated spinal X-ray images as the reference image to determine the homography matrix of the other truncated spinal X-ray images relative to the reference image.

[0156] 5. Based on the determined homography matrix, perform homography transformation on the corresponding truncated spinal X-ray images, then determine the optimal seam estimation based on the hybrid energy function, and finally stitch together the truncated spinal X-ray images in sequence.

[0157] For detailed steps of the above process, please refer to the text above; they will not be repeated here.

[0158] This embodiment also provides an image stitching device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0159] This embodiment provides an image stitching device, such as... Figure 7 As shown, it includes:

[0160] The target structure recognition module 701 is used to recognize multiple target structures in the image to be stitched together.

[0161] The homography matrix determination module 702 is used to determine the homography matrix using target feature points of multiple target structures in different images to be stitched together;

[0162] Transformation module 703 is used to perform perspective transformation on one of the images to be stitched using the homography matrix to obtain a distorted image;

[0163] The stitching module 704 is used to stitch the distorted image with another image to be stitched, which serves as a reference image.

[0164] In some alternative implementations, the target structure is a pedicle screw.

[0165] In some alternative implementations, multiple target structures in the images to be stitched are identified, including:

[0166] Using a target neural network model, multiple target structures in the image to be stitched are identified separately;

[0167] The target neural network model includes at least: a patch embedding layer, an encoder, a decoder, and a parameterless attention module;

[0168] The patch embedding layer is used to divide the image to be stitched into multiple image regions;

[0169] The encoder includes a first VSS block module and a patch merge layer. Multiple image regions divided by the patch embedding layer are input to the first VSS block module. The first VSS block module is used to extract features from the image regions respectively. The extracted feature information is input to the patch merge layer for downsampling.

[0170] The decoder includes a patch expanding layer and a second VSS block module; the patch expanding layer is used for upsampling, and the second VSS block module is used for feature extraction from the output of the patch expanding layer.

[0171] The feature information downsampled by the patch merge layer in the encoder is passed to the decoder through a parameterless attention module.

[0172] In some alternative implementations, the homography matrix determination module 702 includes:

[0173] The initial homography matrix acquisition unit is used to acquire the initial homography matrix;

[0174] The registration energy function construction unit is used to construct a registration energy function, which is the sum of multiple first distances. One first distance is the distance between a first target feature point in the first image to be stitched and a second target feature point in the distorted second image to be stitched. The first and second images to be stitched are two separate images. The first and second target feature points are target feature points within the overlapping portion of the first and distorted second images to be stitched, and the second target feature point is the target feature point in the distorted second image to be stitched that is closest to the first target feature point. The distorted second image to be stitched is obtained by transforming the second image using the initial homography matrix.

[0175] The homography matrix optimization unit is used to optimize the initial homography matrix by minimizing the value of the registration energy function, so as to obtain the optimal homography matrix.

[0176] In some alternative implementations, the homography matrix determination module 702 includes:

[0177] The minimum registration energy function value acquisition unit is used to acquire the minimum registration energy function value between every two images to be stitched if there are three or more images to be stitched.

[0178] The adjacent image determination unit is used to determine, for a given image to be stitched, the other image to be stitched with the smallest minimum registration energy function value as the adjacent image to be stitched.

[0179] The adjacent image homography matrix determination unit is used to determine the homography matrix corresponding to the minimum registration energy function value between two adjacent images to be stitched as the homography matrix between the two adjacent images to be stitched.

[0180] In some alternative implementations, the homography matrix determination module 702 includes:

[0181] The adjacent image homography matrix acquisition unit is used to acquire the homography matrix between each pair of adjacent images if there are three or more images to be stitched.

[0182] The cumulative multiplication unit is used to determine the homography matrix of the third image to be stitched relative to the reference image by multiplying the homography matrices between two adjacent images to be stitched, when one of the images to be stitched is used as the reference image; the third image to be stitched is an image to be stitched other than the reference image and its adjacent images to be stitched.

[0183] In some alternative implementations, the splicing module 704 includes:

[0184] A hybrid energy function construction unit is used to construct a hybrid energy function, which includes pixel differences, gradient differences, and depth feature differences between the distorted image and the reference image in the grayscale space.

[0185] An initialization unit is used to initialize the energy at the starting boundary of the seam.

[0186] The path search unit is used to expand from different starting points to the other end of the image. Each time it expands, it selects the pixel with the smallest mixed energy function value as the next path node until the boundary of the overlapping part of the distorted image and the reference image is reached, thus obtaining the optimal path.

[0187] The optimal seam determination unit is used to backtrack along the optimal path to determine the optimal seam.

[0188] The stitching unit is used to stitch the distorted image and the reference image together with the optimal stitch seam.

[0189] In some alternative implementations, the splicing module 704 includes:

[0190] The transition region determination unit is used to determine the transition region on both sides of the seam between the distorted image and the reference image;

[0191] The weighted unit is used to perform a weighted summation of pixel values ​​belonging to the transition region on the distorted image and the reference image, which is then used as the pixel value of the stitched image.

[0192] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0193] In this embodiment, the image stitching device is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0194] This invention also provides a computer device having the above-described features. Figure 7 The image stitching device shown.

[0195] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0196] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0197] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0198] The memory 20 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some alternative embodiments, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0199] The memory 20 may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0200] The computer device also includes an input device 30 and an output device 40. The processor 10, memory 20, input device 30, and output device 40 can be connected via a bus or other means. Figure 8 Taking the example of a connection between China and Israel via a bus.

[0201] Input device 30 can receive input numerical or character information, and generate key signal inputs related to user settings and function control of the computer device, such as a touchscreen, keypad, mouse, trackpad, touchpad, joystick, one or more mouse buttons, trackball, joystick, etc. Output device 40 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The aforementioned display devices include, but are not limited to, liquid crystal displays, light-emitting diodes, displays, and plasma displays. In some alternative embodiments, the display device may be a touchscreen.

[0202] The computer device also includes a communication interface for communicating with other devices or communication networks.

[0203] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0204] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0205] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. An image stitching method, characterized by, The method comprises: identifying a plurality of target structures in the images to be spliced; determining a homography matrix using target feature points of the plurality of target structures in different images to be spliced; the target feature points are determined based on the geometric shape of the target structure; performing perspective transformation on one of the images to be spliced using the homography matrix to obtain a distorted image; splicing the distorted image with another image to be spliced as a reference image; wherein the determination of the homography matrix using target feature points of the plurality of target structures in different images to be spliced comprises: obtaining an initial homography matrix; constructing a registration energy function, the registration energy function being the sum of a plurality of first distances, one of the first distances being the distance between a first target feature point in a first image to be spliced and a second target feature point in a distorted second image to be spliced; the first image to be spliced and the second image to be spliced being two of the images to be spliced; the first target feature point and the second target feature point being the target feature points within the overlapping part of the first image to be spliced and the distorted second image to be spliced, and the second target feature point being the target feature point in the distorted second image to be spliced closest to the first target feature point; the second image to be spliced being transformed using the initial homography matrix to obtain the distorted second image to be spliced; optimizing the initial homography matrix by minimizing the value of the registration energy function to obtain an optimal homography matrix.

2. The method of claim 1, wherein, The target structure is a pedicle screw.

3. The method of claim 1, wherein, The identification of a plurality of target structures in the images to be spliced comprises: using a target neural network model to identify a plurality of target structures in the images to be spliced respectively; wherein the target neural network model at least comprises: a Patch embedding layer, an encoder, a decoder and a parameter-free attention module; the Patch embedding layer is used to divide the images to be spliced into a plurality of image regions; the encoder comprises a first VSS block module and a patch merge layer, the plurality of image regions divided by the Patch embedding layer are input into the first VSS block module, the first VSS block module is used to extract features of the image regions respectively, and the extracted feature information is input into the patch merge layer for down-sampling; the decoder comprises a Patch expanding layer and a second VSS block module; the Patch expanding layer is used for up-sampling, and the second VSS block module is used for feature extraction on the output of the Patch expanding layer; the feature information down-sampled by the patch merge layer in the encoder is transmitted to the decoder through the parameter-free attention module.

4. The method of claim 1, wherein, The homographic matrix is determined by using target feature points of the target structures in different ones of the images to be stitched. If the images to be stitched are three or more, a minimum registration energy function value between each two of the images to be stitched is obtained. For one of the images to be stitched, another one of the images to be stitched with a minimum registration energy function value is taken as an adjacent image to be stitched. A homographic matrix corresponding to a minimum registration energy function value between the adjacent two images to be stitched is determined as a homographic matrix between the adjacent two images to be stitched.

5. The method of claim 1, wherein, The homographic matrix is determined by using target feature points of the target structures in different ones of the images to be stitched. If the images to be stitched are three or more, a homographic matrix between each adjacent two of the images to be stitched is obtained. A homographic matrix of a third image to be stitched relative to a reference image is determined in a manner of multiplication of the homographic matrices between the adjacent two images to be stitched, the third image to be stitched being the image to be stitched other than the reference image and an adjacent image to be stitched to the reference image.

6. The method of claim 1, wherein, The warped image is stitched with another one of the images to be stitched as a reference image, including: A mixed energy function is constructed, the mixed energy function including pixel difference, gradient difference and depth feature difference in a gray space between the warped image and the reference image; An energy is initialized for a start boundary of a seam; From different starts, an image other end is expanded, each time, a pixel with a minimum mixed energy function value is selected as a next path node until a boundary of an overlapping part of the warped image and the reference image, an optimal path is obtained; The optimal path is backtracked to determine a best seam; The warped image and the reference image are stitched according to the best seam.

7. The method according to claim 1 or 6, characterized in that, The warped image is stitched with another one of the images to be stitched as a reference image, including: Transition regions on two sides of a seam between the warped image and the reference image are determined; Pixel values belonging to the transition regions on the warped image and the reference image are weighted and summed to be pixel values of a stitched image.

8. An image stitching apparatus characterized by comprising: The apparatus includes: A target structure identification module is configured to identify a plurality of target structures in images to be stitched. A homographic matrix determination module is configured to determine a homographic matrix by using target feature points of the target structures in different ones of the images to be stitched. A transformation module is configured to perform perspective transformation on one of the images to be stitched by using the homographic matrix to obtain a warped image. A stitching module is configured to stitch the warped image with another one of the images to be stitched as a reference image. The homographic matrix determination module includes: An initial homographic matrix acquisition unit is configured to acquire an initial homographic matrix. The registration energy function construction unit is configured to construct a registration energy function, the registration energy function being a sum of a plurality of first distances, one of the first distances being a distance between a first target feature point in a first to-be-stitched image and a second target feature point in a warped second to-be-stitched image; the first to-be-stitched image and the second to-be-stitched image being two of the to-be-stitched images; the first target feature point and the second target feature point being the target feature points within an overlapping portion of the first to-be-stitched image and the warped second to-be-stitched image, and the second target feature point being a target feature point in the warped second to-be-stitched image closest to the first target feature point; the second to-be-stitched image being warped by using the initial homography matrix to obtain the warped second to-be-stitched image; The homography matrix optimization unit is configured to optimize the initial homography matrix by minimizing a value of the registration energy function, to obtain an optimal homography matrix.

9. A computer device, comprising: The method comprises the following steps: A memory and a processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the image stitching method according to any one of claims 1 to 7.

Images