Image reconstruction method and electronic device

Abstract

The application provides an image reconstruction method and an electronic device, which can be applied to the technical field of medical image processing. The method comprises the following steps: performing registration on a plurality of pathological images based on a preset registration mode to obtain a plurality of target registration images; extracting a pixel mapping relationship between an mth target registration image and an ith pathological image by using a graph matching model to obtain an mth initial transformation matrix corresponding to the mth target registration image; performing image block cropping on the mth initial registration image and the ith pathological image by using a sliding window to obtain an mth initial registration image block and an ith pathological image block having a mapping relationship; performing non-rigid registration on the mth initial registration image block and the ith pathological image block by using an optical flow estimation model to obtain an mth reference registration image; determining an ith target registration image from m reference registration images related to the ith pathological image; and performing sequence mapping on the plurality of target registration images and the plurality of pathological images to obtain a reconstructed image.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically to an image reconstruction method and electronic device. Background Technology

[0002] Pathological diagnosis plays a guiding role in clinical treatment. Pathological analysis mainly relies on observation of single or a small number of pathological images, which is insufficient to fully reflect the true morphological structure of tissues in three-dimensional space. In order to analyze the microstructure of tissues from a three-dimensional perspective, three-dimensional reconstruction of pathological tissues is necessary. However, there are technical problems with low reconstruction accuracy in the three-dimensional reconstruction methods of related serial pathological sections. Summary of the Invention

[0003] In view of the above problems, the present invention provides an image reconstruction method and an electronic device.

[0004] According to a first aspect of the present invention, an image reconstruction method is provided, comprising: acquiring multiple pathological images representing a tumor region of a target object; registering the multiple pathological images based on a preset registration pattern to obtain multiple target registration images; wherein obtaining the i-th target registration image includes: extracting the pixel mapping relationship between the m-th target registration image and the i-th pathological image using a graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image, where 0 < m < i, and the i-th target registration image is determined based on the i-th pathological image; cropping image blocks of the m-th initial registration image and the i-th pathological image using a sliding window to obtain the m-th initial registration image block and the i-th pathological image block with a mapping relationship, wherein the m-th initial registration image is obtained by transforming the i-th pathological image based on the m-th initial transformation matrix; performing non-rigid registration of the m-th initial registration image block and the i-th pathological image block using an optical flow estimation model to obtain the m-th reference registration image; determining the i-th target registration image from the m reference registration images related to the i-th pathological image; and performing sequence mapping on the multiple target registration images and the multiple pathological images to obtain a reconstructed image.

[0005] According to an embodiment of this application, the graph matching model includes an attention module, a graph construction module, and a graph matching module. The process of extracting the pixel mapping relationship between the m-th target registration image and the i-th pathological image using the graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image includes: inputting the m-th target registration image and the i-th pathological image into the attention module to obtain a first target enhancement feature and a first attention matrix corresponding to the m-th target registration image, and a second target enhancement feature and a second attention matrix corresponding to the i-th pathological image; performing matching degree detection on the first target enhancement feature and the second target enhancement feature to obtain a detection result; if the detection result indicates that the detection is successful, processing the first attention matrix and the second attention matrix using the graph construction module to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image; and extracting the pixel mapping relationship between the first graph structure and the second graph structure using the graph matching module to obtain the m-th initial transformation matrix.

[0006] According to an embodiment of this application, the attention module includes a self-attention unit and a cross-attention unit; wherein, inputting the m-th target registration image and the i-th pathological image into the attention module to obtain a first attention matrix corresponding to the m-th target registration image and a second attention matrix corresponding to the i-th pathological image includes: inputting the m-th target registration image and the i-th pathological image into the self-attention unit respectively to obtain a first initial enhancement feature corresponding to the m-th target registration image and a second initial enhancement feature corresponding to the i-th pathological image; fusing the first initial enhancement feature and the m-th target registration image to obtain a first initial fusion feature; fusing the second initial enhancement feature and the i-th pathological image to obtain a second initial fusion feature; determining a first query feature based on the first initial fusion feature, determining a first key feature and a first value feature based on the second initial fusion feature, fusing the first query feature, the first key feature, and the first value feature using the cross-attention unit to obtain a first attention matrix corresponding to the m-th target registration image; determining a second query feature based on the second initial fusion feature, determining a second key feature and a second value feature based on the first initial fusion feature, fusing the second query feature, the second key feature, and the second value feature using the cross-attention unit to obtain a second attention matrix corresponding to the i-th pathological image.

[0007] According to an embodiment of this application, a matching degree detection is performed on a first target enhancement feature and a second target enhancement feature to obtain a detection result, including: fusing the first target enhancement feature and a first initial fusion feature to obtain a first target fusion feature; fusing the second target enhancement feature and the second initial fusion feature to obtain a second target fusion feature; processing the first target fusion feature and the second target fusion feature using an activation function to obtain a confidence score, wherein the confidence score characterizes the reliability of the matching pixel pairs formed by the pixels of the m-th target registration image and the i-th pathological image; if the number of target matching pixel pairs is greater than a preset number threshold, a detection result characterizing that the detection has passed is obtained, wherein the confidence score of the target matching pixel is greater than a preset confidence threshold; if the number of target matching pixel pairs is less than or equal to the preset number threshold, a detection result characterizing that the detection has failed is obtained.

[0008] According to an embodiment of this application, a graph construction module processes a first attention matrix and a second attention matrix to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image. This includes: determining a first cross-attention matrix corresponding to the m-th target registration image based on the first attention matrix and the transposed second attention matrix, where the first cross-attention matrix characterizes the similarity between pixels in the m-th target registration image; constructing a first graph structure based on pixels in the first cross-attention matrix whose feature values ​​are greater than a preset similarity threshold; determining a second cross-attention matrix corresponding to the i-th pathological image based on the second attention matrix and the transposed first attention matrix, where the second cross-attention matrix characterizes the similarity between pixels in the i-th pathological image; and constructing a second graph structure based on pixels in the second cross-attention matrix whose feature values ​​are greater than a preset similarity threshold.

[0009] According to embodiments of this application, the optical flow estimation model includes optical flow estimation layers corresponding to each of the K-level resolutions; wherein, non-rigid registration of the m-th initial registered image block and the i-th pathological image block is performed using the optical flow estimation model to obtain the m-th reference registered image, including: stitching the (k-1)-th deformation field and the k-th pathological image block to obtain the k-th stitched image block, wherein the k-th pathological image block is obtained by cropping the i-th pathological image corresponding to the k-level resolution using a sliding window. The k-th optical flow estimation layer is used to perform non-rigid registration on the k-th stitched image block and the k-th matched image block with a mapping relationship to obtain the k-th deformation field. The k-th matched image block is obtained by cropping the m-th initial registered image corresponding to the k-th resolution based on a sliding window. The k-th deformation field represents the pixel position correspondence between the m-th initial registered image corresponding to the k-th resolution and the i-th pathological image. The i-th pathological image is transformed based on the target deformation field to obtain the m-th reference registered image. The target deformation field is obtained by fusing K deformation fields.

[0010] According to an embodiment of this application, determining the i-th target registration image from m reference registration images related to the i-th pathological image includes: calculating the similarity between the m-th reference registration image and the (m-1)-th target registration image to obtain the similarity corresponding to the m-th reference registration image; and determining the reference registration image corresponding to the maximum similarity among the m similarities as the i-th target registration image.

[0011] According to an embodiment of this application, the image reconstruction method further includes: segmenting the initial pathological image using a threshold segmentation algorithm to obtain a background mask image and a non-background mask image; performing a mask operation between the non-background mask image and the initial pathological image to obtain a non-background pathological image; and filtering the non-background pathological image to obtain a pathological image.

[0012] According to an embodiment of this application, a reconstructed image is obtained by sequential mapping of multiple target registration images and multiple pathological images, including: interpolating every two adjacent target registration images in the multiple target registration images to obtain multiple interpolated images; and sequentially mapping the multiple interpolated images, multiple pathological images, and non-background mask images of each of the multiple pathological images to obtain the reconstructed image.

[0013] A second aspect of the present invention provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the method described above.

[0014] According to the image reconstruction method provided by this invention, the graph matching model can effectively distinguish repetitive texture regions that are similar in appearance but different in spatial location by utilizing the spatial topological constraints between feature points, thereby significantly improving the robustness of the initial transformation matrix estimation and avoiding erroneous matching caused by repetitive textures. In the non-rigid registration stage, a multi-scale layer-by-layer refinement strategy is adopted, and an optical flow estimation network is combined to predict the registration relationship of image blocks with mapping relationships, realizing unified modeling of large deformation and detailed deformation. This can improve the precision of registration while ensuring the stability of the calculation process, so as to maintain the local topological relationship of the tissue and avoid forcibly distorting and destroying the original tissue structure. This achieves high robustness and high precision registration and three-dimensional reconstruction of continuous pathological images, providing support for the spatial analysis of tissue microenvironment and complex structures. Attached Figure Description

[0015] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings.

[0016] Figure 1 A flowchart of an image reconstruction method according to an embodiment of the present invention is shown.

[0017] Figure 2 A schematic diagram of a graph matching model according to an embodiment of the present invention is shown.

[0018] Figure 3 A schematic diagram of a graph construction module according to an embodiment of the present invention is shown.

[0019] Figure 4 A schematic diagram of an optical flow estimation model according to an embodiment of the present invention is shown.

[0020] Figure 5 A block diagram of an electronic device suitable for implementing an image reconstruction method according to an embodiment of the present invention is shown. Detailed Implementation

[0021] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the invention for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0022] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0023] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0024] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0025] In the process of realizing this invention, it was found that related three-dimensional reconstruction methods for continuous pathological sections typically employ a layer-by-layer registration and stacking approach to complete the reconstruction, including four parts: image preprocessing, initial affine registration, non-rigid registration, and continuous alignment. In the initial affine registration stage, only feature point matching is relied upon. However, due to the similarity of local textures in pathological images, considering only point features can lead to incorrect matching and registration failure. In the non-rigid registration stage, traditional iterative optimization methods are used, which do not consider the topological information of the tissue, resulting in forced image distortion and damage to the original tissue structure.

[0026] In view of this, embodiments of the present invention provide an image reconstruction method. The method includes: registering multiple pathological images based on a preset registration pattern to obtain multiple target registration images; extracting the pixel mapping relationship between the m-th target registration image and the i-th pathological image using a graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image; cropping image blocks of the m-th initial registration image and the i-th pathological image using a sliding window to obtain an m-th initial registration image block and an i-th pathological image block with a mapping relationship; performing non-rigid registration of the m-th initial registration image block and the i-th pathological image block using an optical flow estimation model to obtain an m-th reference registration image; determining the i-th target registration image from the m reference registration images related to the i-th pathological image; and performing sequence mapping on the multiple target registration images and the multiple pathological images to obtain a reconstructed image.

[0027] In the technical solution of this invention, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0028] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0029] Figure 1 A flowchart of an image reconstruction method according to an embodiment of the present invention is shown.

[0030] like Figure 1 As shown, the method includes operations S110 to S140.

[0031] In operation S110, multiple pathological images representing the tumor region of the target object are acquired.

[0032] In operation S120, multiple pathological images are registered based on a preset registration mode to obtain multiple target registration images.

[0033] In operation S130, the sequence mapping of multiple target registration images and multiple pathological images is performed to obtain reconstructed images.

[0034] The target group can be patients with tumor tissue.

[0035] Multiple pathological images are high-resolution whole-section images obtained by digitizing multiple consecutive pathological sections related to the tumor area of ​​the target object through a digital pathology scanner after fixation, staining, and other processes.

[0036] For example, there are I pathological images. The first pathological image is used as the first target registration image. The second pathological image is registered to the first target registration image to obtain the second target registration image. The third pathological image is registered to both the first and second target registration images to obtain the first and second reference registration images. The third target registration image is determined from the first and second reference registration images. The i-th pathological image is registered to the i-1 target registration images to obtain the i-1 reference registration images. The i-th target registration image is determined from the i-1 reference registration images. The I-th pathological image is registered to the I-1 target registration images to obtain the I-1 reference registration images. The I-th target registration image is determined from the I-1 reference registration images.

[0037] m is a positive integer less than i.

[0038] Specifically, registering the i-th pathological image to i-1 target registration images to obtain the i-th target registration image includes the following process.

[0039] For the m-th target registration image among the i-1 target registration images, the pixel mapping relationship between the m-th target registration image and the i-th pathological image is extracted using a graph matching model, and the m-th initial transformation matrix corresponding to the m-th target registration image is obtained.

[0040] Graph matching models can be constructed based on a fusion attention mechanism that combines self-attention and cross-attention mechanisms.

[0041] For example, the Scale-Invariant Feature Transform (SIFT) algorithm with embedded fusion attention mechanism is used to extract feature points from the m-th target registration image and the i-th pathological image, respectively, to obtain the scale-invariant features of local feature points. The Fast Library for Approximate Nearest Neighbors (FLANN) algorithm is used to perform feature matching on the scale-invariant features of local feature points, and the Least Median of Squares (LMedS) algorithm is used to select matching points to obtain the pixel mapping relationship between local feature points. The m-th initial transformation matrix is ​​estimated based on the pixel mapping relationship between local feature points, and the i-th pathological image is transformed based on the m-th initial transformation matrix to obtain the m-th initial registration image after rigid transformation.

[0042] The m-th initial transformation matrix represents the initial spatial mapping relationship, and the m-th initial registration image represents the spatial alignment result after rigid transformation.

[0043] By using a sliding window, image blocks are cropped from the m-th initial registration image and the i-th pathological image to obtain the m-th initial registration image block and the i-th pathological image block with a mapping relationship.

[0044] The optical flow estimation model can be constructed based on the Upsampling Pyramid for Unsupervised Flow Learning (UpFlow) network. The optical flow estimation model is used to perform non-rigid registration of the m-th initial registered image patch and the i-th pathological image patch, obtaining a non-rigid registration relationship. This non-rigid registration relationship is then used to transform the i-th pathological image to obtain the m-th reference registered image.

[0045] The m-th reference registration image represents the fine alignment result after registration with nonlinear deformations such as local stretching, bending, and twisting.

[0046] The i-th pathological image is registered to m target registration images, resulting in m reference registration images. The similarity between each reference registration image and the previous target registration image can be calculated to evaluate the calibration effect. The reference registration image with the best calibration effect is selected from the m reference registration images as the i-th target registration image.

[0047] The target registration image represents the registration result after eliminating the accumulation of errors during the successive registration process.

[0048] Linear interpolation algorithms are used to interpolate multiple target registration images to obtain intermediate interpolated images. These intermediate interpolated images and multiple pathological images are then sequentially mapped to a unified three-dimensional Cartesian coordinate system to obtain a three-dimensional reconstructed image. The reconstructed image can accurately reflect the microscopic structure within the tissue.

[0049] In the embodiments of this application, the graph matching model can effectively distinguish repetitive texture regions that are similar in appearance but different in spatial location by utilizing the spatial topological constraints between feature points, thereby significantly improving the robustness of the initial transformation matrix estimation and avoiding erroneous matching caused by repetitive textures. In the non-rigid registration stage, a multi-scale layer-by-layer refinement strategy is adopted, and an optical flow estimation network is combined to predict the registration relationship of image blocks with mapping relationships, realizing unified modeling of large deformation and detailed deformation. This can improve the precision of registration while ensuring the stability of the calculation process, so as to maintain the local topological relationship of the tissue and avoid forcibly distorting and destroying the original tissue structure. This achieves high robustness and high precision registration and three-dimensional reconstruction of continuous pathological images, providing support for the spatial analysis of tissue microenvironment and complex structures.

[0050] According to an embodiment of this application, the image reconstruction method further includes: segmenting the initial pathological image using a threshold segmentation algorithm to obtain a background mask image and a non-background mask image; performing a mask operation between the non-background mask image and the initial pathological image to obtain a non-background pathological image; and filtering the non-background pathological image to obtain a pathological image.

[0051] The initial pathological image in Red Green Blue (RGB) format is converted to HueSaturation Value (HSV) space, the saturation channels are extracted, and a threshold segmentation algorithm is used to automatically segment the pathological image into a binary background mask and a non-background mask by calculating the threshold that maximizes the inter-class variance.

[0052] The non-background mask image is bitwise ANDed with the initial pathological image to remove background interference, and then grayscale processing is performed to retain only the non-background pathological image.

[0053] The non-background pathological image was converted from an RGB three-channel image to a single-channel grayscale image. Because the saturation of tissue areas is typically significantly higher than that of background areas, it is important to use [specific techniques] on single-channel grayscale images. The Sobel edge detection operator performs convolution operations on the saturation channel to enhance tissue edge features and suppress background interference, resulting in a filtered pathological image.

[0054] According to an embodiment of this application, the graph matching model includes an attention module, a graph construction module, and a graph matching module. The process of extracting the pixel mapping relationship between the m-th target registration image and the i-th pathological image using the graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image includes: inputting the m-th target registration image and the i-th pathological image into the attention module to obtain a first target enhancement feature and a first attention matrix corresponding to the m-th target registration image, and a second target enhancement feature and a second attention matrix corresponding to the i-th pathological image; performing matching degree detection on the first target enhancement feature and the second target enhancement feature to obtain a detection result; if the detection result indicates that the detection is successful, processing the first attention matrix and the second attention matrix using the graph construction module to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image; and extracting the pixel mapping relationship between the first graph structure and the second graph structure using the graph matching module to obtain the m-th initial transformation matrix.

[0055] The attention module integrates self-attention and cross-attention mechanisms.

[0056] Input the m-th target registration image into the attention module to obtain the first target enhancement feature and the first attention matrix corresponding to the m-th target registration image.

[0057] The first target enhancement feature represents the feature enhancement representation that incorporates global contextual information of the m-th target registered image; the first attention matrix represents the correlation weight distribution between pixel positions in the first target enhancement feature.

[0058] The i-th pathological image is input into the attention module to obtain the second target enhancement feature and the second attention matrix corresponding to the i-th pathological image.

[0059] The second target enhancement feature is a feature enhancement representation that incorporates global contextual information of the i-th pathological image; the second attention matrix represents the correlation weight distribution between pixel positions in the second target enhancement feature.

[0060] Calculate the matching score of the feature points corresponding to the first and second target enhancement features to obtain the matching result. The detection result indicates whether the number of matched feature points satisfies the feature pruning condition.

[0061] The detection result indicates that the detection passed, meaning that there are enough matched feature points. The first and second attention matrices can be processed by the graph construction module through the iterative proportional fitting algorithm (Sinkhorn algorithm, Sinkhorn), and points with high matching degree can be selected to construct the graph structure, resulting in the first graph structure corresponding to the m-th target registration image and the second graph structure corresponding to the i-th pathological image.

[0062] The first graph structure represents the global association between different spatial regions (or feature locations) in the m-th target registration image, and is a topological graph organized in the form of nodes and edges.

[0063] The second graph structure represents the global relationships between different spatial regions (or feature locations) in the i-th pathological image, and is a topological graph organized in the form of nodes and edges.

[0064] If the detection result indicates that the detection fails, it means that the number of matching feature points is too small. In this case, feature pruning needs to be performed to remove feature points with low confidence. The remaining features are then sent to the next attention module to continue attention calculation and optimization until a high-precision detection result is obtained.

[0065] The pixel mapping relationship between the first and second graph structures is extracted using the graph matching module to obtain the m-th initial transformation matrix.

[0066] According to embodiments of this application, a graph structure of the m-th target registration image and the i-th pathological image is constructed using a first attention matrix and a second attention matrix. By selecting point pairs with high response values ​​to establish edge connections, the originally isolated point matching is elevated to structural matching that includes adjacency relationships and topological constraints. It considers not only the similarity of individual feature points but also the consistency of the correlation between feature points, thus more effectively suppressing false matching and random noise interference, and enhancing the overall consistency and structural rationality of the matching results. This solves the technical problem that the feature matching process only focuses on the similarity between discrete points, making it difficult to effectively utilize the spatial relationships of tissue structures, thus easily leading to local correctness but overall structural distortion.

[0067] According to an embodiment of this application, the attention module includes a self-attention unit and a cross-attention unit; wherein, inputting the m-th target registration image and the i-th pathological image into the attention module to obtain a first attention matrix corresponding to the m-th target registration image and a second attention matrix corresponding to the i-th pathological image includes: inputting the m-th target registration image and the i-th pathological image into the self-attention unit respectively to obtain a first initial enhancement feature corresponding to the m-th target registration image and a second initial enhancement feature corresponding to the i-th pathological image; fusing the first initial enhancement feature and the m-th target registration image to obtain a first initial fusion feature; fusing the second initial enhancement feature and the i-th pathological image to obtain a second initial fusion feature; determining a first query feature based on the first initial fusion feature, determining a first key feature and a first value feature based on the second initial fusion feature, fusing the first query feature, the first key feature, and the first value feature using the cross-attention unit to obtain a first attention matrix corresponding to the m-th target registration image; determining a second query feature based on the second initial fusion feature, determining a second key feature and a second value feature based on the first initial fusion feature, fusing the second query feature, the second key feature, and the second value feature using the cross-attention unit to obtain a second attention matrix corresponding to the i-th pathological image.

[0068] Self-attention units are constructed based on the self-attention mechanism, and cross-attention units are constructed based on the cross-attention mechanism.

[0069] The m-th target registration image is subjected to feature extraction and feature enhancement using a self-attention unit to obtain the first initial enhanced features corresponding to the m-th target registration image.

[0070] The i-th pathological image is subjected to feature extraction and feature enhancement using a self-attention unit to obtain the second initial enhanced feature corresponding to the i-th pathological image.

[0071] The first initial enhancement feature and the m-th target registration image are stitched together to obtain the first initial fusion feature.

[0072] The second initial enhancement feature and the i-th pathological image are stitched together to obtain the second initial fusion feature.

[0073] In the cross-attention unit, the second initial fused feature Convolution is performed to obtain the second query feature, and the first initial fused feature is then processed. Convolution is performed to obtain the second key feature and the second value feature. The second query feature, the second key feature, and the second value feature are then fused to obtain the second attention matrix. The second attention matrix represents the similarity of feature points between the i-th pathological image and the m-th target registration image, obtained by using the features of the i-th pathological image as a query.

[0074] For the first initial fusion feature Convolution yields the first query feature, which is then used to fuse the second initial feature. Convolution is performed to obtain the first key feature and the first value feature. The first query feature, the first key feature, and the first value feature are then fused to obtain the first attention matrix. The first attention matrix represents the similarity of feature points between the m-th target registration image and the i-th pathological image, obtained by using the features of the m-th target registration image as a query.

[0075] According to embodiments of this application, matching degree detection is performed on the first target enhancement feature and the second target enhancement feature to obtain a detection result, including: fusing the first target enhancement feature and the first initial fusion feature to obtain a first target fusion feature; fusing the second target enhancement feature and the second initial fusion feature to obtain a second target fusion feature; processing the first target fusion feature and the second target fusion feature using an activation function to obtain a confidence score, wherein the confidence score characterizes the reliability of the matching pixel pairs formed by the pixels of the m-th target registration image and the i-th pathological image; if the number of target matching pixel pairs is greater than a preset number threshold, a detection result characterizing the detection passed is obtained, wherein the confidence score of the target matching pixel is greater than a preset confidence threshold; if the number of target matching pixel pairs is less than or equal to the preset number threshold, a detection result characterizing the detection failed is obtained.

[0076] The first target enhancement feature is obtained by extracting features from the first initial fusion feature using a cross-attention unit; the second target enhancement feature is obtained by extracting features from the second initial fusion feature using a cross-attention unit.

[0077] The first target enhancement feature and the first initial fusion feature are stitched together to obtain the first target fusion feature of the m-th target registration image; the second target enhancement feature and the second initial fusion feature are stitched together to obtain the second target fusion feature of the i-th pathological image.

[0078] The confidence score represents the degree of trustworthiness of a pixel pair's match. A confidence score greater than a preset confidence threshold indicates that the matching degree of the pixel pair is trustworthy.

[0079] If the number of target matching pixel pairs is greater than a preset threshold, it means that there are enough target matching pixel pairs used to construct the graph structure, and the graph structure construction result is relatively accurate. When the number of target matching pixel pairs is greater than the preset threshold, a detection result indicating that the detection has passed is obtained.

[0080] If the number of target matching pixel pairs is less than or equal to a preset threshold, it indicates that the number of target matching pixel pairs used to construct the graph structure is insufficient, resulting in inaccurate graph structure construction. When the number of target matching pixel pairs is less than or equal to the preset threshold, a detection result indicating a failed detection is obtained.

[0081] If a detection result fails the representation detection, feature pruning is performed to remove low-confidence feature points, and the remaining features are sent to the next attention module to continue attention calculation and optimization until a detection result that passes the representation detection is obtained.

[0082] Figure 2 A schematic diagram of a graph matching model according to an embodiment of the present invention is shown.

[0083] like Figure 2 As shown, the graph matching model consists of N layers, each including an attention module, a graph construction module, and a graph matching module. The attention module includes self-attention units and cross-attention units. The m-th target registration image 202 and the i-th pathological image 201 are fed into a pre-trained SuperPoint network with frozen parameters. This network is responsible for extracting the coordinates of key points from the images. and and the corresponding feature descriptors and Keypoint coordinates and feature descriptors are fed into N stacked self-attention units and cross-attention units. In the self-attention units, a first initial enhancement feature corresponding to the m-th target registration image and a second initial enhancement feature corresponding to the i-th pathological image are obtained. The first initial enhancement feature and the m-th target registration image are fused to obtain a first initial fused feature. The second initial enhancement feature and the i-th pathological image are fused to obtain a second initial fused feature. The first and second initial fused features are processed using cross-attention units to obtain a first target enhancement feature, a second target enhancement feature, a first attention matrix, and a second attention matrix. The first target enhancement feature and the first initial fused feature are then fused to obtain the first target enhancement feature. The first target fusion feature is fused with the second target enhancement feature and the second initial fusion feature to obtain the second target fusion feature. Matching degree detection is performed on the first target fusion feature and the second target fusion feature to obtain the detection result. If the detection result indicates that the representation detection fails, feature pruning is performed, and the remaining features are sent to the next layer for further attention calculation and optimization until a detection result indicating that the representation detection passes is obtained. If the detection result indicates that the detection passes, the first attention matrix and the second attention matrix are processed using the graph construction module to obtain the first graph structure and the second graph structure. The pixel mapping relationship between the first graph structure and the second graph structure is extracted using the graph matching module to obtain the m-th initial transformation matrix.

[0084] According to the embodiments of this application, the detection result is determined based on the confidence score. If the detection result fails the characterization detection, the feature points with low confidence are removed, and the remaining features are sent to the next layer attention module to continue attention calculation and optimization until the detection result passes the characterization detection is obtained, thereby obtaining more target matching pixel pairs for constructing the graph structure and obtaining a high-precision graph structure and matching result.

[0085] According to an embodiment of this application, a graph construction module processes a first attention matrix and a second attention matrix to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image. This includes: determining a first cross-attention matrix corresponding to the m-th target registration image based on the first attention matrix and the transposed second attention matrix, where the first cross-attention matrix characterizes the similarity between pixels in the m-th target registration image; constructing a first graph structure based on pixels in the first cross-attention matrix whose feature values ​​are greater than a preset similarity threshold; determining a second cross-attention matrix corresponding to the i-th pathological image based on the second attention matrix and the transposed first attention matrix, where the second cross-attention matrix characterizes the similarity between pixels in the i-th pathological image; and constructing a second graph structure based on pixels in the second cross-attention matrix whose feature values ​​are greater than a preset similarity threshold.

[0086] Multiplying the first attention matrix by the transpose of the second attention matrix yields the first cross-attention matrix. .

[0087] The first cross-attention matrix represents the similarity between pixels in the m-th target registration image after the feature weights of the i-th pathological image have been corrected.

[0088] Multiplying the second attention matrix by the transpose of the first attention matrix yields the second attention matrix. ;

[0089] The second cross-attention matrix represents the similarity between pixels in the i-th pathological image after the feature weights of the m-th target registration image have been corrected.

[0090] If the eigenvalues ​​in the first cross-attention matrix are greater than the preset similarity threshold, it means that the two pixels in the corresponding m-th target registration image are highly similar. In the current feature space, the feature vectors of these two points are the closest and most matched, and there is an edge relationship between the two pixels.

[0091] For example, select P pairs of pixels in the first cross-attention matrix whose feature values ​​are greater than a preset similarity threshold, and construct a first graph structure based on the P pairs of pixels and the edges.

[0092] If the eigenvalues ​​in the second cross-attention matrix are greater than the preset similarity threshold, it means that the two pixels in the corresponding i-th pathological image are highly similar. In the current feature space, the feature vectors of these two points are the closest and most matched, and there is an edge relationship between the two pixels.

[0093] For example, in the second cross-attention matrix, select P pairs of pixels whose feature values ​​are greater than a preset similarity threshold, and construct a second graph structure based on the P pairs of pixels and the edges.

[0094] Figure 3 A schematic diagram of a graph construction module according to an embodiment of the present invention is shown.

[0095] like Figure 3 As shown, based on the first attention matrix 301 and the transposed second attention matrix 302, a first cross-attention matrix 303 corresponding to the m-th target registration image is determined; based on the pixels in the first cross-attention matrix 303 whose feature values ​​are greater than a preset similarity threshold, a first graph structure 304 is constructed; based on the second attention matrix 302 and the transposed first attention matrix 301, a second cross-attention matrix 305 corresponding to the i-th pathological image is determined; based on the pixels in the second cross-attention matrix 305 whose feature values ​​are greater than a preset similarity threshold, a second graph structure 306 is constructed.

[0096] According to embodiments of this application, a graph structure is constructed between the i-th pathological image and the m-th target registration image using a bidirectional cross-attention matrix. Edge connections are established by selecting pixel pairs in the cross-attention matrix whose eigenvalues ​​are greater than a preset similarity threshold, thus elevating the originally isolated pixel matching to a structural matching that includes adjacency relationships and topological constraints. This approach considers not only the similarity of individual feature points but also the consistency of the relationships between feature points, thereby more effectively suppressing false matching and random noise interference, and enhancing the overall consistency and structural rationality of the matching results.

[0097] According to an embodiment of this application, determining the i-th target registration image from m reference registration images related to the i-th pathological image includes: calculating the similarity between the m-th reference registration image and the (m-1)-th target registration image to obtain the similarity corresponding to the m-th reference registration image; and determining the reference registration image corresponding to the maximum similarity among the m similarities as the i-th target registration image.

[0098] The similarity between the m-th reference registration image and the (m-1)-th target registration image represents the degree of spatial alignment between the m-th reference registration image (the current registration result) after registration to the m-th target registration image and the adjacent (m-1)-th target registration image (the previous reference image).

[0099] If the similarity is high, it means that the m-th reference registration image after registration is closer to the verified m-1-th target registration image, and the m-th reference registration image after registration is accurate and reliable.

[0100] For example, obtaining the fourth target registration image based on the registration of the fourth pathological image includes: registering the fourth pathological image to the first three target registration images respectively, obtaining three reference registration images respectively, and determining the fourth target registration image from the three reference registration images.

[0101] For example, taking the first, second, and third reference registration images mentioned above as examples, the similarity A between the third reference registration image and the second target registration image, and the similarity B between the second reference registration image and the first target registration image are calculated. If the similarity A is greater than the similarity B, then the third reference registration image corresponding to similarity A is determined as the fourth target registration image.

[0102] According to the embodiments of this application, in the continuous pathological slide registration process, by registering the i-th pathological image with the previous i-1 registered target registration images respectively to obtain i-1 reference registration images, and selecting the optimal reference registration image as the i-th target registration image based on the registration similarity, the accumulation of errors in the continuous registration process can be reduced, thereby improving the overall registration accuracy of continuous pathological slides and the spatial continuity and structural consistency of the three-dimensional reconstruction results. This overcomes the technical problem in related technologies where the adjacent previous target registration image is fixed as a reference, which is easily affected by slide damage, staining differences and local morphological changes.

[0103] According to embodiments of this application, the optical flow estimation model includes optical flow estimation layers corresponding to each of the K-level resolutions; wherein, non-rigid registration of the m-th initial registered image block and the i-th pathological image block is performed using the optical flow estimation model to obtain the m-th reference registered image, including: stitching the (k-1)-th deformation field and the k-th pathological image block to obtain the k-th stitched image block, wherein the k-th pathological image block is obtained by cropping the i-th pathological image corresponding to the k-level resolution using a sliding window. The k-th optical flow estimation layer is used to perform non-rigid registration on the k-th stitched image block and the k-th matched image block with a mapping relationship to obtain the k-th deformation field. The k-th matched image block is obtained by cropping the m-th initial registered image corresponding to the k-th resolution based on a sliding window. The k-th deformation field represents the pixel position correspondence between the m-th initial registered image corresponding to the k-th resolution and the i-th pathological image. The i-th pathological image is transformed based on the target deformation field to obtain the m-th reference registered image. The target deformation field is obtained by fusing K deformation fields.

[0104] The training set includes initial registration images and pathological images of samples, as well as sample graph structures, output by the graph matching model. For each pair of images in the training set, an image pyramid is constructed, and an optical flow estimation model is trained separately for each layer of the image pyramid. Using the initial registration images and pathological images of samples, and the correspondence between points and edges in the sample graph structures, the negative value of the normalized cross-correlation coefficient after deformation field distortion output by the optical flow estimation model is calculated as the image similarity loss function. The distance between corresponding points and edges in the graphs of the pathological image to be registered and the target pathological image after deformation field distortion output by the optical flow estimation model is calculated as the graph distance loss function. The edge-aware diffusion regularization value and the negative Jacobian determinant value of the deformation field are calculated as the regularization loss function. Based on the image similarity loss function, the graph distance loss function, and the regularization loss function, the target loss function is obtained. The optical flow estimation model is trained based on the target loss function to obtain the trained optical flow estimation model.

[0105] Scale the m-th initially registered image and the i-th pathological image to the original image size. This yields the m-th initial registration image and the i-th pathological image corresponding to the k-th resolution, and uses a step size of... Height and width are The sliding window extracts image blocks from the m-th initial registration image and the i-th pathological image corresponding to the k-th resolution, resulting in several one-to-one corresponding k-th matching image blocks and k-th pathological image blocks. The k-th matching image block and the k-th pathological image block are both squares with height H and width W, and k is less than K.

[0106] Figure 4 A schematic diagram of an optical flow estimation model according to an embodiment of the present invention is shown.

[0107] like Figure 4 As shown, K=3, the optical flow estimation model includes 3 optical flow estimation layers. The matching image blocks and pathological image blocks corresponding to the 3 resolutions are stitched together along the channel dimension and input into the optical flow estimation layers corresponding to the 3 resolutions respectively.

[0108] The registration is refined layer by layer starting from the first optical flow estimation layer: the first optical flow estimation layer is used to perform non-rigid registration on multiple first matching image blocks and first pathological image blocks with mapping relationships to obtain the first deformation field.

[0109] The first deformation field and the second pathological image block are stitched together to obtain the second stitched image block; the second optical flow estimation layer is used to perform non-rigid registration on multiple second stitched image blocks and second matched image blocks with mapping relationships to obtain the second deformation field.

[0110] The second deformation field and the third pathological image block are stitched together to obtain the third stitched image block; the third optical flow estimation layer is used to perform non-rigid registration on multiple third stitched image blocks and third matched image blocks with mapping relationships to obtain the third deformation field.

[0111] The first deformation field is upsampled and then spliced ​​with the second deformation field to obtain the initial deformation field fusion feature; the initial deformation field fusion feature is upsampled and then spliced ​​with the third deformation field to obtain the target deformation field.

[0112] The target deformation field is the same size as the original i-th pathological image.

[0113] The i-th pathological image is transformed using the target deformation field to obtain the m-th reference registration image.

[0114] According to the embodiments of this application, in view of the complex nonlinear deformations such as local stretching, compression, and bending that are common in pathological images, as well as the characteristics of high resolution and large size of pathological images, this application adopts a layer-by-layer refinement strategy based on multi-scale image pyramids in the non-rigid registration stage, and combines optical flow estimation network to predict the deformation field of corresponding image blocks. First, a large-scale coarse registration is completed at the low resolution level, and then it is gradually upsampled and refined to the high resolution level, thereby realizing a unified modeling of large deformation and detailed deformation. While ensuring the stability of the calculation process, it can improve the precision of deformation field estimation, so that the final registration result fully matches the actual tissue deformation.

[0115] According to an embodiment of this application, a reconstructed image is obtained by sequential mapping of multiple target registration images and multiple pathological images, including: interpolating every two adjacent target registration images in the multiple target registration images to obtain multiple interpolated images; and sequentially mapping the multiple interpolated images, multiple pathological images, and non-background mask images of each of the multiple pathological images to obtain the reconstructed image.

[0116] All target registration images are arranged in spatial order, and interpolation methods are used to supplement missing layer information between adjacent target registration images to obtain multiple interpolated images.

[0117] Since individual pathological images have a certain thickness, direct stacking can cause a sense of discontinuity in the Z-axis direction. Linear interpolation is used to generate intermediate voxels between two adjacent registered target images. By calculating the motion vectors of pixels between sequences, missing spatial layer information is filled in, ensuring that the reconstructed 3D data volume has a spatial resolution in the Z-axis direction that matches the XY plane.

[0118] All original pathological images and interpolated image sequences are mapped to a unified three-dimensional Cartesian coordinate system. Combined with the non-background mask image extracted during preprocessing, the voxel coordinates of each pixel in three-dimensional space and its corresponding grayscale or color attribute are determined.

[0119] By setting a uniform voxel density threshold and eliminating redundant background noise, a three-dimensional voxel reconstruction image that can accurately reflect the internal microstructure of the tissue is formed.

[0120] The reconstructed image is a three-dimensional visualization structure with spatial continuity and morphological integrity.

[0121] Figure 5 A block diagram of an electronic device suitable for implementing an image reconstruction method according to an embodiment of the present invention is shown.

[0122] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0123] like Figure 5 As shown, a computer electronic device 500 according to an embodiment of the present invention includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a ROM 502 (read-only memory) or a program loaded from a storage portion 508 into a RAM 503 (random access memory). The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0124] RAM 503 stores various programs and data required for the operation of electronic device 500. Processor 501, ROM 502, and RAM 503 are interconnected via bus 504. Processor 501 executes various operations of the method flow according to embodiments of the present invention by executing programs in ROM 502 and / or RAM 503. It should be noted that programs may also be stored in one or more memories other than ROM 502 and RAM 503. Processor 501 may also execute various operations of the method flow according to embodiments of the present invention by executing programs stored in one or more memories.

[0125] In embodiments of this application, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may also include one or more of the following components connected to the input / output (I / O) interface 505: an input section 506 including a keyboard, mouse, etc.; an output section 507 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the input / output (I / O) interface 505 as needed. A removable medium 511, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 510 as needed so that computer programs read from it can be installed into the storage section 508 as needed.

[0126] The embodiments of this application, and the method flow according to the embodiments of the present invention, can be implemented as a computer software program. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via communication section 509, and / or installed from removable medium 511. When the computer program is executed by processor 501, it performs the functions defined in the system of the embodiments of the present invention. In embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0127] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the image reconstruction method according to embodiments of the present invention.

[0128] In embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this invention, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0129] For example, in embodiments of this application, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0130] Embodiments of the present invention also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of the present invention. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the image reconstruction method provided in the embodiments of the present invention.

[0131] When the computer program is executed by the processor 501, it performs the functions defined in the system / apparatus of this invention. In embodiments of this application, the systems, apparatuses, modules, units, etc., described above can be implemented using computer program modules.

[0132] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 509, and / or installed from a removable medium 511. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0133] In embodiments of this application, program code for executing the computer programs provided in the embodiments of the present invention can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0134] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features described in the various embodiments of the present invention can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments of the present invention can be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All such combinations and / or pairings fall within the scope of this invention.

[0135] The embodiments of the present invention have been described above. However, these embodiments are merely illustrative and not intended to limit the scope of the invention. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of the invention, and all such substitutions and modifications should fall within the scope of the invention.

Claims

1. An image reconstruction method, characterized in that, The method includes: Acquire multiple pathological images representing the tumor region of the target object; Multiple pathological images are registered based on a preset registration mode to obtain multiple target registration images; The i-th target registration image is obtained, including: The pixel mapping relationship between the m-th target registration image and the i-th pathological image is extracted using a graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image, where 0 < m < i, and the 1-th target registration image is determined based on the 1-th pathological image. Using a sliding window, image blocks are cropped from the m-th initial registration image and the i-th pathological image to obtain the m-th initial registration image block and the i-th pathological image block with a mapping relationship. The m-th initial registration image is obtained by transforming the i-th pathological image based on the m-th initial transformation matrix. The m-th initial registered image block and the i-th pathological image block are non-rigidly registered using an optical flow estimation model to obtain the m-th reference registered image. The i-th target registration image is determined from the m reference registration images associated with the i-th pathological image; Sequence mapping is performed on multiple target registration images and multiple pathological images to obtain reconstructed images.

2. The method according to claim 1, characterized in that, The graph matching model includes an attention module, a graph construction module, and a graph matching module; The step of extracting the pixel mapping relationship between the m-th target registration image and the i-th pathological image using a graph matching model to obtain the m-th initial transformation matrix corresponding to the m-th target registration image includes: The m-th target registration image and the i-th pathological image are input into the attention module to obtain the first target enhancement feature and the first attention matrix corresponding to the m-th target registration image, and the second target enhancement feature and the second attention matrix corresponding to the i-th pathological image; The matching degree of the first target enhancement feature and the second target enhancement feature is detected to obtain the detection result; If the detection result indicates that the detection is passed, the graph construction module processes the first attention matrix and the second attention matrix to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image. The pixel mapping relationship between the first graph structure and the second graph structure is extracted using the graph matching module to obtain the m-th initial transformation matrix.

3. The method according to claim 2, characterized in that, The attention module includes self-attention units and cross-attention units; Specifically, the m-th target registration image and the i-th pathological image are input into the attention module to obtain a first attention matrix corresponding to the m-th target registration image and a second attention matrix corresponding to the i-th pathological image, including: The m-th target registration image and the i-th pathological image are respectively input into the self-attention unit to obtain the first initial enhancement feature corresponding to the m-th target registration image and the second initial enhancement feature corresponding to the i-th pathological image; The first initial enhanced feature and the m-th target registration image are fused to obtain the first initial fused feature; The second initial enhancement feature and the i-th pathological image are fused to obtain the second initial fusion feature; Based on the first initial fusion feature, a first query feature is determined; based on the second initial fusion feature, a first key feature and a first value feature are determined; the first query feature, the first key feature, and the first value feature are fused using the cross attention unit to obtain the first attention matrix corresponding to the m-th target registration image. The second query feature is determined based on the second initial fusion feature, the second key feature and the second value feature are determined based on the first initial fusion feature, and the second query feature, the second key feature and the second value feature are fused using the cross attention unit to obtain the second attention matrix corresponding to the i-th pathological image.

4. The method according to claim 2, characterized in that, The matching degree detection of the first target enhancement feature and the second target enhancement feature to obtain the detection result includes: The first target enhancement feature and the first initial fusion feature are fused to obtain the first target fusion feature; The second target enhancement feature and the second initial fusion feature are fused together to obtain the second target fusion feature; The first target fusion feature and the second target fusion feature are processed using an activation function to obtain a confidence score. The confidence score represents the reliability of the matching pixel pair formed by the pixels of the m-th target registration image and the i-th pathological image. When the number of target matching pixel pairs is greater than a preset threshold, a detection result indicating that the detection has passed is obtained, wherein the confidence score of the target matching pixel is greater than a preset confidence threshold. If the number of target matching pixel pairs is less than or equal to the preset number threshold, a detection result indicating that the detection failed is obtained.

5. The method according to claim 2, characterized in that, The process of using the graph construction module to process the first attention matrix and the second attention matrix to obtain a first graph structure corresponding to the m-th target registration image and a second graph structure corresponding to the i-th pathological image includes: Based on the first attention matrix and the transposed second attention matrix, a first cross attention matrix is ​​determined corresponding to the m-th target registration image. The first cross attention matrix characterizes the similarity between pixels in the m-th target registration image. A first graph structure is constructed based on the pixels in the first cross-attention matrix whose feature values ​​are greater than a preset similarity threshold; Based on the second attention matrix and the transposed first attention matrix, a second cross attention matrix corresponding to the i-th pathological image is determined. The second cross attention matrix characterizes the similarity between pixels in the i-th pathological image. A second graph structure is constructed based on the pixels in the second cross-attention matrix whose feature values ​​are greater than a preset similarity threshold.

6. The method according to claim 1, characterized in that, The optical flow estimation model includes optical flow estimation layers corresponding to each of the K-level resolutions; The step of performing non-rigid registration of the m-th initial registered image block and the i-th pathological image block pair using an optical flow estimation model to obtain the m-th reference registered image includes: The (k-1)th deformation field and the kth pathological image block are stitched together to obtain the kth stitched image block. The kth pathological image block is obtained by cropping the i-th pathological image corresponding to the k-th resolution using a sliding window. ; The kth optical flow estimation layer is used to perform non-rigid registration on the kth stitched image block and the kth matched image block with mapping relationship to obtain the kth deformation field. The kth matched image block is obtained by cropping the image block of the mth initial registered image corresponding to the kth resolution based on the sliding window. The k deformation field characterizes the pixel position correspondence between the mth initial registered image corresponding to the kth resolution and the ith pathological image. The i-th pathological image is transformed based on the target deformation field to obtain the m-th reference registration image. The target deformation field is obtained by fusing K deformation fields.

7. The method according to claim 1, characterized in that, Determining the i-th target registration image from the m reference registration images associated with the i-th pathological image includes: Calculate the similarity between the m-th reference registration image and the (m-1)-th target registration image to obtain the similarity corresponding to the m-th reference registration image; The reference registration image corresponding to the maximum similarity among m similarities is determined as the i-th target registration image.

8. The method according to claim 1, characterized in that, The method further includes: The initial pathological image was segmented using a threshold segmentation algorithm to obtain a background mask image and a non-background mask image; The non-background mask image is compared with the initial pathological image by performing a masking operation to obtain the non-background pathological image; The non-background pathological image is filtered to obtain the pathological image.

9. The method according to claim 1, characterized in that, The step of performing sequence mapping on multiple target registration images and multiple pathological images to obtain reconstructed images includes: Interpolate every two adjacent target registration images in the plurality of target registration images to obtain a plurality of interpolated images; A reconstructed image is obtained by sequentially mapping multiple interpolated images, multiple pathological images, and non-background mask images of each of the multiple pathological images.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 9.

Images