An endoscope video and pathological image matching system combined with deep learning

Abstract

The application discloses an endoscope video and pathological image matching system combined with deep learning, and relates to the technical field of medical image processing. Endoscope video frame images and pathological section images are projected to a shared latent space, and are constrained to satisfy a linear combination of topological deformation bases to obtain combination coefficients. Time sequence motion signatures of the endoscope video frame images are extracted, and cell nucleus topological graphs of the pathological section images are extracted. The time sequence motion signatures are applied to the cell nucleus topological graphs as boundary constraints, and stiffness information of the cell nucleus topological graphs is fed back to the time sequence motion signatures to obtain coupling features. A preliminary deformation field is generated according to the coupling features and the combination coefficients. The pathological section images are resampled according to the preliminary deformation field to generate virtual reconstruction graphs, and differences between the virtual reconstruction graphs and the endoscope video frame images are calculated. The combination coefficients are iteratively adjusted according to the differences until a termination condition is satisfied, and a target deformation field is output. According to the target deformation field, pathological region labels are mapped to the endoscope video frame images and are superimposed and displayed.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, specifically to a system for matching endoscopic videos and pathological images using deep learning. Background Technology

[0002] Endoscopic and pathological examinations are two complementary and important tools in the diagnosis and treatment of gastrointestinal tumors. Endoscopy can acquire real-time macroscopic dynamic videos of the gastrointestinal mucosa, while pathological examination provides the gold standard diagnostic information at the cellular level through biopsy sections. Precise spatial correlation between endoscopic videos and pathological images of the same patient has direct clinical guiding significance for early lesion localization, targeted biopsy, and postoperative assessment. However, cross-modal matching of endoscopic videos and pathological images faces significant technical challenges. The two differ greatly in their imaging mechanisms, resolution, and feature spaces, making it difficult to establish effective similarity measures using traditional manual feature-based registration methods. Furthermore, endoscopic videos contain complex non-rigid deformations such as respiration, heartbeat, and instrument manipulation, while pathological images are static slices; utilizing video temporal information to assist matching is a unique challenge in this scenario.

[0003] Existing technologies include deep learning-based medical image registration schemes. For example, CN113850852A discloses a multi-scale context-based endoscopic image registration method using a U-Net architecture. This method extracts multi-scale features through a hierarchical aggregation module and refines the feature maps using an adaptive feature scaling module, ultimately generating a parameterized registration field. This achieves deformable registration of endoscope image pairs within the same modality. However, this scheme only addresses static grayscale image pairs and does not address cross-modal scenarios between endoscopic videos and pathological images. Its network design and training strategy do not consider modal distribution differences. Its hierarchical aggregation module can only extract texture features within the same modality and cannot achieve semantic mapping between macroscopic mucosa and microscopic cellular structures; the adaptive feature scaling module also lacks cross-modal alignment capabilities. Furthermore, this scheme does not utilize video temporal information and relies only on single-frame images, which can easily lead to matching failures or localization errors in clinical applications due to insufficient information.

[0004] To address the aforementioned shortcomings, this invention proposes a deep learning-based system for matching endoscopic videos and pathological images. By constructing a joint learning framework that combines temporal coding and cross-modal feature alignment, it solves the problem of cross-scale and cross-modal spatial correlation between dynamic videos and static pathological slices, providing technical support for the precision diagnosis and treatment of gastrointestinal tumors. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based system for matching endoscopic videos and pathological images, in order to solve the problems mentioned in the background section.

[0006] To address the aforementioned technical problems, this invention provides the following technical solution: a deep learning-based endoscopic video and pathological image matching system, comprising:

[0007] The latent space alignment module is used to project endoscopic video frame images and pathological slide images into a shared latent space, and in the shared latent space, constrain the two sets of projected features to satisfy a linear combination of the same set of topological deformation bases, and output the combination coefficients.

[0008] The temporal coding module is used to extract the motion features of the endoscopic video frame images in the time dimension and output a temporal motion signature;

[0009] The cell topology encoding module is used to extract the structural features of the pathological slide image in the spatial dimension and output the cell nucleus topology map.

[0010] The coupling deformation module is used to apply the temporal motion signature as a boundary constraint to the cell nucleus topology graph, and to feed back the stiffness information of the cell nucleus topology graph to the temporal motion signature, and output coupling features;

[0011] The deformation field estimation module is used to perform spatial transformation calculations based on the coupling characteristics and the combination coefficients, and output a preliminary deformation field.

[0012] The closed-loop verification module is used to resample the pathological slide image based on the preliminary deformation field, generate a virtual reconstruction image, calculate the difference between the virtual reconstruction image and the endoscopic video frame image, and iteratively adjust the combination coefficients based on the difference until the iteration termination condition is met, at which point the target deformation field is output.

[0013] Preferably, the latent space alignment module maps the feature distribution of the endoscopic video frame image to the shared latent space through the first reversible flow unit, maps the feature distribution of the pathological slide image to the shared latent space through the second reversible flow unit, and calculates the linear regression coefficients of the two sets of projected features relative to the topological deformation basis through the basis constraint unit, and uses the linear regression coefficients as the combination coefficients.

[0014] Preferably, the temporal coding module extracts spatiotemporal features from multiple consecutive endoscopic video frames using a three-dimensional convolutional unit, calculates the dense optical flow field between adjacent frames using an optical flow calculation unit based on the extracted spatiotemporal features, and generates the temporal motion signature using a signature generation unit based on the temporal statistics of the dense optical flow field. The temporal motion signature includes the boundary value of tissue vibration amplitude and the principal component of vibration frequency.

[0015] Preferably, the cell topology encoding module identifies the cell nucleus position coordinates in the pathological slide image through the cell nucleus detection unit, and generates the cell nucleus topology map by performing triangulation based on the cell nucleus position coordinates through the topology map construction unit. The vertices of the cell nucleus topology map are the cell nucleus center points, and the edges of the cell nucleus topology map are the connecting lines between adjacent cell nuclei. The anisotropic stiffness coefficient is calculated by the stiffness calculation unit based on the displacement response of each vertex in the cell nucleus topology map under preset stress conditions.

[0016] Preferably, the coupling deformation module sets the boundary value of the tissue vibration amplitude in the temporal motion signature as the maximum displacement limit of each vertex when the cell nucleus topology is deformed by the boundary constraint unit, the anisotropic stiffness coefficient is used as a spatial attention weight by the stiffness feedback unit and weighted to the optical flow confidence of the corresponding spatial position in the temporal motion signature, and the feature fusion unit splices the boundary-constrained cell nucleus topology and the weighted temporal motion signature in the channel dimension to generate the coupling feature.

[0017] Preferably, the deformation field estimation module maps the coupling features and the combination coefficients into spatial transformation parameters through a spatial transformation network unit, and constructs the preliminary deformation field based on the spatial transformation parameters through a deformation field generation unit. The preliminary deformation field includes the displacement vector of each pixel in the pathological slide image to the coordinate space of the endoscopic video frame image.

[0018] Preferably, the closed-loop verification module performs bilinear interpolation resampling on the pathological slide image based on the preliminary deformation field through the resampling unit to generate the virtual reconstruction image, calculates the morphological gradient distance and structural similarity between the virtual reconstruction image and the endoscopic video frame image through the difference calculation unit to obtain the difference, and updates the combination coefficients and regenerates the preliminary deformation field only when the difference is greater than a preset threshold through the iterative adjustment unit.

[0019] Preferably, the iterative adjustment unit adjusts the step size of the combination coefficients along the gradient descent direction based on the gradient direction of the difference relative to the combination coefficients. After each update, the difference is recalculated until the difference is less than or equal to the preset threshold. Then, the current deformation field is output as the target deformation field. The maximum number of iterations is a preset value. When the maximum number of iterations is reached, the iteration is terminated and the current deformation field is output.

[0020] Preferably, it also includes a label mapping module, which transforms the pathological region labels marked on the pathological slide image to the coordinate space of the endoscopic video frame image according to the target deformation field to generate a superimposed label frame.

[0021] Preferably, it further includes a display output module, which overlays the superimposed label frame with the endoscopic video frame image; the endoscopic video frame image is a keyframe image in the dynamic video of gastrointestinal mucosal endoscopy, and the pathological slide image is a digital pathological slide corresponding to the biopsy point of the keyframe image.

[0022] This invention provides a deep learning-based system for matching endoscopic videos and pathological images. It offers the following advantages:

[0023] This system constructs a shared latent space to project endoscopic videos and pathological images bidirectionally, using linear combination coefficients of topological deformation bases to characterize cross-modal spatial mapping relationships. This design avoids the conventional path of directly calculating similarity in the original pixel space, thus eliminating the fundamental constraint of modal differences on matching accuracy. The bidirectional coupling mechanism between temporal motion signatures and cell nuclear topological maps allows macroscopic physiological vibration laws to constrain the deformation boundaries of microscopic structures, while the local rigidity information of microscopic structures can inversely calibrate noise interference in macroscopic motion estimation. This cross-scale information interaction method allows the matching process to be guided by both video temporal information and pathological structural information.

[0024] The closed-loop verification module achieves progressive refinement of the deformation field by updating only the combination coefficients without changing the network master parameters during the iteration process. This refinement process uses the morphological and structural consistency between the virtual reconstructed image and the original endoscopic frame as the evaluation criterion, so that the final output target deformation field can take into account both pixel-level spatial accuracy and region-level structural fidelity. The target deformation field output by the system can be used to map the diagnostic labels on the pathological slides to the corresponding spatial positions of the endoscopic video frames, providing spatial reference for lesion identification and localization during endoscopic operations. In the gastrointestinal endoscopic diagnosis and treatment scenario, the entire system can complete cross-modal spatial matching tasks without gold standard deformation field supervision. Attached Figure Description

[0025] Figure 1 This is a data flow diagram of the endoscopic video and pathological image matching system combining deep learning, as described in this invention.

[0026] Figure 2 This is a flowchart of the closed-loop iterative execution process of the present invention. Detailed Implementation

[0027] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0028] Example: Please refer to Figure 1 and Figure 2 This invention provides a deep learning-based system for matching endoscopic video and pathological images. The system's implementation begins with the data acquisition phase. The operator inputs a dynamic video stream of the digestive tract mucosa acquired by the endoscopic equipment into the system, along with digital pathological slides corresponding to biopsy sites in the video. Keyframe images are extracted from the video stream at preset sampling intervals, and the pathological slides are digitized by a high-resolution scanner and then input into the system. During system operation, the endoscopic video frame images and pathological slide images are first sent to a latent space alignment module. In this module, a first reversible flow unit performs multi-layer reversible transformation operations on the endoscopic video frame images, progressively converting the images from the original red-green-blue color space to the latent feature space, and then further mapping them to a shared latent space to obtain the first projection feature. A second reversible flow unit performs the same reversible transformation operation sequence on the pathological slide images, mapping them to the same shared latent space to obtain the second projection feature.

[0029] In the shared latent space, the basis constraint unit pre-stores a set of topological deformation bases. These base vectors are obtained in advance through the following method: collecting paired data of multiple sets of historical endoscopic video frame images and corresponding pathological slide images; for each pair of data, manually annotating key points or using an elastic registration algorithm to calculate the pixel-level displacement field from the pathological slide image space to the endoscopic video frame image space; expanding the pixel-level displacement field corresponding to each pair of data into a one-dimensional vector and arranging it into a deformation field matrix; performing principal component analysis on this matrix and selecting the first few principal component directions as the topological deformation bases; and calculating the linear regression coefficients of the first and second projection features relative to this set of topological deformation bases, averaging the two sets of coefficients and outputting the combined coefficients.

[0030] The temporal coding module receives multiple consecutive frames of endoscopic video images. The 3D convolution unit uses a 3D convolution kernel to perform convolution operations simultaneously in the spatial and temporal dimensions, extracting the spatial texture features of each frame and the temporal variation features between frames. Based on the extracted spatiotemporal features, the optical flow calculation unit uses a pyramid-layered refinement method to calculate the dense optical flow field between adjacent frames. This optical flow field records the positional offset of each pixel in the previous frame in the next frame. The signature generation unit performs statistical analysis on the optical flow field over time, statistically analyzing the offset variation patterns of each spatial position in multiple consecutive frames, extracting the boundary values ​​of tissue vibration amplitude and the principal components of vibration frequency, and integrating these two pieces of information into a temporal motion signature output.

[0031] The cell topology coding module receives pathological slide images. The cell nucleus detection unit preprocesses the images and uses a kernel segmentation algorithm to identify the coordinates of the center points of each cell nucleus. The topology graph construction unit uses the center points of each cell nucleus as vertices and employs a triangulation algorithm to connect adjacent vertices, forming a cell nucleus topology graph. Each edge in this graph represents the connection between two adjacent cell nuclei. The stiffness calculation unit applies a preset simulated micro-perturbation force to each vertex in the cell nucleus topology graph, calculates the displacement response of each vertex under this force condition, and calculates the anisotropic stiffness coefficient at each vertex based on the magnitude of the displacement response. This coefficient reflects the local stiffness of the tissue structure at that location.

[0032] The coupled deformation module receives a temporal motion signature and a cell nucleus topology map as two inputs. The boundary constraint unit reads the tissue vibration amplitude boundary value from the temporal motion signature and sets this boundary value as the maximum allowable displacement distance of each vertex in the cell nucleus topology map when performing subsequent deformation operations. That is, any deformation position where the displacement of any vertex exceeds this boundary value is judged as not conforming to physiological laws and is excluded. The stiffness feedback unit reads the anisotropic stiffness coefficient of each vertex in the cell nucleus topology map and uses this coefficient as a spatial attention weight, which is then weighted to the optical flow confidence at the same spatial position in the temporal motion signature. That is, the optical flow confidence weight corresponding to the region with higher local stiffness is increased, and the optical flow confidence weight corresponding to the region with lower local stiffness is decreased. The feature fusion unit concatenates the cell nucleus topology map after boundary constraint processing and the temporal motion signature after stiffness weighting processing along the channel dimension to form a coupled feature containing both spatial structure and temporal motion information.

[0033] The deformation field estimation module receives coupling features and combination coefficients. The spatial transformation network unit inputs the coupling features into a convolutional neural network, which, after processing through convolutional and fully connected layers, outputs spatial transformation parameters. The deformation field generation unit constructs a spatial transformation grid based on the spatial transformation parameters. Each node in the grid corresponds to a pixel position in the pathological slide image. The displacement of the grid is the displacement vector required to map that pixel to the coordinate space of the endoscopic video frame image. The set of displacement vectors of all pixels constitutes the preliminary deformation field.

[0034] The closed-loop verification module receives the preliminary deformation field and pathological slide images. The resampling unit performs bilinear interpolation resampling on the pathological slide images based on the displacement vectors of each pixel in the preliminary deformation field. This involves moving each pixel in the pathological slide image according to the direction and distance indicated by the displacement vector, filling any gaps through interpolation to generate a virtual reconstructed image. The difference calculation unit compares the virtual reconstructed image with the endoscopic video frame image, calculating the morphological gradient distance and structural similarity between them, and using the calculation results as the difference value. The iterative adjustment unit determines whether the difference value is greater than a preset threshold. If the difference value is not greater than the preset threshold, the iteration termination condition is met, and the current preliminary deformation field is directly output as the target deformation field. If the difference value is greater than the preset threshold, the iterative adjustment unit only performs an update operation on the combination coefficients. Specifically, it calculates the gradient direction of the difference value relative to the combination coefficients, adjusts the step size of the combination coefficients along the gradient descent direction, updates the combination coefficients, and then re-sends them to the deformation field estimation module to generate a new preliminary deformation field. The above resampling, difference calculation, and threshold judgment processes are repeated. A maximum number of iterations is preset. When the number of iterations reaches this maximum, the iteration terminates, and the current deformation field is output as the target deformation field. Throughout the iteration process, the network parameters of the latent space alignment module, temporal coding module, cellular topology coding module, and coupled deformation module remain unchanged; only the combination coefficients participate in the iterative update.

[0035] The first reversible flow unit receives an endoscopic video frame image as input, which is a single-channel matrix after grayscale processing. The first reversible flow unit internally contains multiple reversible transformation layers, each containing learnable affine coupling parameters. The input image sequentially passes through each reversible transformation layer. At each layer, the image feature matrix is ​​divided into two subsets along the channel dimension. One subset remains unchanged, while the other subset undergoes an affine transformation based on the statistics of the previous subset. The scaling and offset coefficients of this affine transformation are controlled by learnable parameters. Since each transformation layer is a bijective mapping, this unit can losslessly transform the endoscopic video frame image from the original pixel space to an intermediate feature space, and further map it to a shared latent space of a preset dimension to obtain the first projected feature vector. Throughout the mapping process, the inverse transformation methods of each reversible transformation layer are parsed and recorded, allowing the first projected feature vector to be accurately restored to the original image when needed.

[0036] The second reversible flow unit has the same structure as the first reversible flow unit, but the learnable parameters of the two units are independent. The second reversible flow unit receives a pathological slide image as input, which is also grayscaled and scaled to the same spatial resolution as the endoscopic video frame image. The pathological slide image is sequentially passed through each reversible transformation layer inside the second reversible flow unit, and mapped to a shared latent space with the same dimension as the first projected feature vector according to the same channel segmentation and affine transformation rules, to obtain the second projected feature vector.

[0037] The basis constraint unit internally stores a set of topological deformation bases. These basis vectors are obtained as follows: multiple registered endoscopic video frames and pathological slide image pairs are pre-collected; the true deformation field of each registration pair is calculated; the deformation fields are arranged as vectors to construct a deformation field matrix; principal component decomposition is performed on this matrix; and the first few principal component directions are taken as the topological deformation bases and stored in the basis constraint unit. During runtime, the basis constraint unit obtains the first and second projected eigenvectors, representing them as vectors of the same dimension in a shared latent space. Using the basis vectors of the topological deformation base as independent variables and the projected eigenvectors as dependent variables, the basis constraint unit solves for two sets of linear regression coefficients, minimizing the Euclidean distance between the weighted sum of the basis vectors and their corresponding projected eigenvectors. Finally, the basis constraint unit averages the first coefficient vector corresponding to the first projected eigenvector and the second coefficient vector corresponding to the second projected eigenvector, outputting this average as the combined coefficients. The combination coefficients, together with the topological deformation basis, define the relative positional relationship between endoscopic video frame features and pathological slide features in the shared latent space.

[0038] The 3D convolutional unit receives multiple consecutive frames of endoscopic video images as input, arranged in chronological order of acquisition. Internally, the 3D convolutional unit contains multiple layers of 3D convolutional kernels, each with preset dimensions in the spatial height, spatial width, and temporal depth directions. This unit combines the frames into a 3D tensor, where the temporal dimension corresponds to the frame sequence order. The 3D convolutional kernels simultaneously perform convolution operations on consecutive frames along the temporal dimension. In each convolutional operation, the kernel and the corresponding pixel values ​​in the spatiotemporal neighborhood of the 3D tensor are weighted and summed to generate a spatiotemporal feature map containing a mixture of spatial texture and temporal variation information. As the number of convolutional layers increases, subsequent 3D convolutional kernels continue convolution based on the spatiotemporal feature maps of previous layers, gradually expanding the effective receptive field, ultimately outputting an output tensor containing multi-level spatiotemporal abstract features.

[0039] The optical flow computation unit receives the spatiotemporal features output by the 3D convolutional unit as input. This unit employs a multi-layer pyramid structure for processing. First, the spatiotemporal features of two adjacent frames are input into each level of the pyramid. An initial motion estimate is calculated at the top level of the pyramid with a lower spatial resolution. This estimate is then passed down layer by layer to the bottom levels. In each layer, the motion estimate of the previous layer is incrementally corrected based on the spatiotemporal features of the current layer. The optical flow computation unit calculates the horizontal and vertical displacements between adjacent frames at each spatial location, forming a dense optical flow field. Each pixel position in this optical flow field corresponds to a two-dimensional displacement vector, which represents the direction and distance of the pixel's spatial position offset between the previous and next frames.

[0040] The signature generation unit receives the dense optical flow field sequence output by the optical flow calculation unit. This unit statistically analyzes the optical flow displacement vector at each spatial location over time, recording the trajectory of the displacement vector change at that location over multiple consecutive frame intervals. It extracts the vibration amplitude value from this trajectory, which reflects the physical oscillation range of the tissue at that spatial location during the acquisition process. Simultaneously, the signature generation unit performs a time-domain Fourier transform on the optical flow field sequence, converting the time-domain displacement signal to the frequency domain. In the frequency domain, it extracts the frequency component corresponding to the amplitude peak as the principal component of the vibration frequency at that location.

[0041] The signature generation unit combines the vibration amplitude values ​​at each spatial location into a vibration amplitude distribution map, and sets the maximum value at each location in the vibration amplitude distribution map as the boundary value of tissue vibration amplitude. Before setting the maximum value at each location in the vibration amplitude distribution map as the boundary value of tissue vibration amplitude, the signature generation unit first filters outliers from the vibration amplitude distribution map. Specifically, it calculates the interquartile range of the vibration amplitude values ​​at all locations in the vibration amplitude distribution map, and removes vibration amplitude values ​​that exceed the upper quartile plus 1.5 times the interquartile range, excluding them from the subsequent selection of the maximum value. For the removed locations, their vibration amplitude value is replaced by the average vibration amplitude of the adjacent locations.

[0042] The principal components of vibration frequency at each spatial location are combined to form a vibration frequency distribution map. The frequency value with the highest frequency in the vibration frequency distribution map is used as the global vibration frequency principal component. Finally, the boundary value of tissue vibration amplitude and the global vibration frequency principal component are integrated into a temporal motion signature output.

[0043] The cell nucleus detection unit receives a pathological slide image as input, which is a high-resolution digital image of a digitized pathological slide. The unit first performs color normalization on the image, adjusting regions with different staining intensities to a unified color space. Then, it uses a deep learning-based cell nucleus segmentation network to segment the cell nucleus regions in the image at the pixel level. This segmentation network outputs a binary mask image, where positions with values ​​of the first state correspond to cell nucleus regions, and positions with values ​​of the second state correspond to background regions. The cell nucleus detection unit performs connected component analysis on the binary mask image, treating each connected cell nucleus region as an independent cell nucleus. It calculates the centroid coordinates of each connected component and uses these centroid coordinates as the center point coordinates of the corresponding cell nucleus. Finally, it outputs a list of the horizontal and vertical coordinate values ​​of each cell nucleus center point in the image coordinate system.

[0044] The topology graph construction unit receives a list of coordinates of the center points of each cell nucleus from the cell nucleus detection unit as input. Using each center point as a vertex, this unit performs a constrained triangulation algorithm to partition the vertex set. During the partitioning process, the unit constructs an initial triangular mesh using all vertex coordinates. For each triangle in the mesh, it checks whether its circumcircle contains other vertices. If so, it adjusts the edge connections until all triangles' circumcircles no longer contain other vertices, thus forming a triangulation structure that satisfies the empty circumcircle property. After partitioning, the topology graph construction unit uses each edge in the triangular mesh as a connection line between adjacent cell nuclei. Vertices and connections together constitute the cell nucleus topology graph. This topology graph is an undirected graph structure, where each vertex records the spatial coordinates of its corresponding cell nucleus center point, and each edge records the connection relationship between two connected vertices and the spatial distance between them.

[0045] The stiffness calculation unit receives the cell kernel topology map output by the topology map construction unit as input. This unit applies simulated micro-perturbations to each vertex in the topology map, applying small displacement forces of equal magnitude along each edge connected to that vertex in the topology map, centered on the current vertex. The stiffness calculation unit records the actual displacement of each vertex after applying the perturbation force, obtained by comparing the difference in coordinates before and after the force application. Based on the ratio of the applied perturbation force amplitude to the measured displacement, the stiffness calculation unit calculates the local stiffness value of each vertex along each edge direction. The stiffness calculation unit combines the local stiffness values ​​measured at each vertex along different directions to form the anisotropic stiffness coefficient of that vertex. This coefficient is stored in matrix form, where diagonal elements represent the axial stiffness of the vertex along each coordinate axis, and off-diagonal elements represent the coupling stiffness between different directions. The stiffness calculation unit stores the anisotropic stiffness coefficients of each vertex in association with the cell kernel topology map, which is then used by the coupled deformation module to retrieve the stiffness coefficient at the corresponding position based on the vertex index.

[0046] The boundary constraint unit receives the temporal motion signature output by the temporal coding module and the cell nucleus topology map output by the cell topology coding module as two inputs. First, the boundary constraint unit parses the tissue vibration amplitude boundary value from the temporal motion signature; this boundary value is in scalar form. Then, the boundary constraint unit traverses each vertex in the cell nucleus topology map, reading the current spatial position of each vertex in the image coordinate system. For each vertex, the boundary constraint unit calculates the candidate displacement vector that the vertex will experience during subsequent deformation operations. This candidate displacement vector originates from the displacement sampling value of the deformation field generated in the previous iteration of the deformation field estimation module at the corresponding position. The boundary constraint unit calculates the magnitude of the candidate displacement vector and compares this magnitude with the tissue vibration amplitude boundary value. If the magnitude of the candidate displacement vector is not greater than the tissue vibration amplitude boundary value, the boundary constraint unit allows the vertex to maintain its current candidate displacement. If the magnitude of a candidate displacement vector exceeds the boundary value of tissue vibration amplitude, the boundary constraint unit performs a scaling operation on the candidate displacement vector of that vertex. Specifically, the candidate displacement vector is shortened in the same direction according to the proportional coefficient between its magnitude and the boundary value of tissue vibration amplitude, so that the magnitude of the shortened displacement vector is equal to the boundary value of tissue vibration amplitude. After the boundary constraint unit completes the displacement constraint on all vertices, it outputs a constrained topology graph after boundary limiting processing. The displacement of each vertex in this constrained topology graph is limited to the range allowed by the boundary value of tissue vibration amplitude.

[0047] The stiffness feedback unit receives the temporal motion signature output by the temporal coding module and the anisotropic stiffness coefficients output by the cell topology coding module as inputs, where each anisotropic stiffness coefficient corresponds one-to-one with a vertex of the cell nucleus topology graph. The stiffness feedback unit extracts the optical flow confidence score at each spatial location from the temporal motion signature; this optical flow confidence score records the reliability of the displacement estimation at each location during the optical flow calculation process.

[0048] The optical flow confidence score is acquired synchronously during the generation of the dense optical flow field by the optical flow calculation unit. The acquisition method is as follows: when the optical flow calculation unit performs motion estimation at each level of the pyramid, it records the brightness gradient magnitude and iteration residual at each spatial location. The ratio of the brightness gradient magnitude to the iteration residual is used as the basic confidence score for that location. At the same time, the consistency of motion estimation between each level of the pyramid at that location is recorded. If the difference in motion estimation between adjacent levels is greater than a preset threshold, the basic confidence score for that location is lowered; if it is less than the preset threshold, it remains unchanged. Finally, the confidence score after consistency verification is used as the output of the optical flow confidence score for that location.

[0049] For each spatial location, the stiffness feedback unit reads the anisotropic stiffness coefficients corresponding to that location. Since the anisotropic stiffness coefficients are stored in matrix form and contain multiple directional components, the stiffness feedback unit performs a scalar reduction operation on this matrix, extracting the sum of the main diagonal elements as a scalar representative stiffness value. The sum of the main diagonal elements represents the total axial stiffness at that location along all coordinate axes, reflecting the overall ability of that location to resist deformation. The stiffness feedback unit uses this representative stiffness value as a weight multiplier and multiplies it with the optical flow confidence at that location to obtain a weighted confidence score. In the multiplication operation, the optical flow confidence score is enhanced at locations with larger representative stiffness values ​​and weakened at locations with smaller representative stiffness values. After performing the weighted operation for each spatial location, the stiffness feedback unit outputs a stiffness-weighted temporal motion signature. The optical flow confidence scores at each location in this signature are modulated by the local stiffness information of the nucleus topology.

[0050] The feature fusion unit receives the constrained topology graph output by the boundary constraint unit and the weighted temporal motion signature output by the stiffness feedback unit as inputs. The constrained topology graph is graph-structured data, and the weighted temporal motion signature is tensor data with spatial and channel dimensions. The feature fusion unit first rasterizes the constrained topology graph, filling the displacement constraint information and position coordinate information carried by each vertex in the topology graph into a blank grid with the same resolution as the endoscopic video frame image, forming a spatial feature map. The feature fusion unit then concatenates the rasterized spatial feature map with the weighted temporal motion signature in the channel dimension. The resulting new tensor contains both spatial topology constraint channels and temporal motion constraint channels. This new tensor is output as a coupled feature to the deformation field estimation module.

[0051] The spatial transformation network unit receives the coupling features output by the coupling deformation module and the combination coefficients output by the latent space alignment module as two inputs. The coupling features are tensors with spatial and multi-channel dimensions, and the combination coefficients are in vector form. The spatial transformation network unit contains multiple convolutional layers. The coupling features are first downsampled through each convolutional layer, progressively compressing the spatial dimension and expanding the channel dimension, finally obtaining a one-dimensional feature vector after global average pooling. The spatial transformation network unit concatenates this one-dimensional feature vector with the combination coefficients along the feature dimension to obtain a joint feature vector. The joint feature vector is then input to multiple fully connected layers within the spatial transformation network unit, undergoing layer-by-layer nonlinear transformation to output spatial transformation parameters. These spatial transformation parameters include a horizontal scaling factor, a vertical scaling factor, a horizontal translation, a vertical translation, and a rotation angle, collectively forming an affine transformation description of the spatial coordinates of the pathological slide image. When the deformation field generation unit transforms the coordinates according to the above spatial transformation parameters, the execution order is as follows: first, perform rotation operation with the coordinate origin as the center, then perform scaling operation in the horizontal and vertical directions, and finally perform translation operation in the horizontal and vertical directions.

[0052] The deformation field generation unit receives spatial transformation parameters output by the spatial transformation network unit as input. First, it constructs a reference coordinate grid with the same spatial resolution as the pathological slide image. The coordinates of each node in this grid represent its row and column positions in the image coordinate system. The deformation field generation unit performs coordinate transformation operations on each node in the reference coordinate grid according to the spatial transformation parameters. The transformation method involves rotating, scaling, and translating the original row and column coordinates of each node sequentially to obtain the transformed new coordinate values. The deformation field generation unit calculates the difference between the transformed new coordinate values ​​and the original coordinate values ​​of each node, using this difference as the displacement vector at that node. The displacement vector includes row and column displacement components. The displacement vectors at each node collectively constitute a two-dimensional displacement field with the same spatial resolution as the pathological slide image. The deformation field generation unit outputs this displacement field as a preliminary deformation field. The displacement vector at each coordinate in the preliminary deformation field records the offset by which the pixel at that coordinate position in the pathological slide image needs to be moved to the target position in the coordinate space of the endoscopic video frame image. Since the reference coordinate grid covers the entire spatial range of the pathological slide image, the initial deformation field is a dense deformation field, with each pixel position corresponding to a displacement vector, and there are no holes or undefined areas.

[0053] The resampling unit receives the preliminary deformation field output from the deformation field estimation module and the original pathological slide image as two inputs. Each pixel position in the preliminary deformation field corresponds to a displacement vector, which includes row and column offsets. Using the pathological slide image as the source image, the resampling unit iterates through each pixel position in the source image. For the current pixel position, it reads the corresponding displacement vector from the preliminary deformation field and adds the row and column coordinates of the current pixel position to the displacement vector to obtain the target coordinate position. The components of the target coordinate position are usually non-integer values. The resampling unit takes four adjacent integer pixels around the target coordinate position and performs bilinear interpolation based on the pixel values ​​of these four integer pixels. This involves multiplying each of the four pixel values ​​by its corresponding weighting coefficient and then summing the results to obtain the interpolated pixel value at the target coordinate position. The resampling unit arranges all the interpolated pixel values ​​according to the target coordinate position to form a virtual reconstruction image. The spatial resolution of this virtual reconstruction image is the same as that of the pathological slide image and consistent with the spatial resolution of the endoscopic video frame image.

[0054] The difference calculation unit receives the virtual reconstructed image output by the resampling unit and the original endoscopic video frame image as input. The difference calculation unit performs morphological gradient extraction operation on the two images respectively. Specifically, it uses the Sobel operator to calculate the horizontal and vertical gradient values ​​at each pixel position. Based on the horizontal and vertical gradient values, it calculates the gradient magnitude and gradient direction at each pixel position to obtain the first gradient distribution of the virtual reconstructed image and the second gradient distribution of the endoscopic video frame image.

[0055] The difference calculation unit calculates the absolute value of the gradient magnitude difference between the first gradient distribution and the second gradient distribution at corresponding pixel positions, and takes the average of the sums of the absolute values ​​at all positions as the morphological gradient distance. Simultaneously, the difference calculation unit divides the virtual reconstructed image and the endoscopic video frame image into multiple local windows according to a preset window size. These local windows have overlapping areas, with the width of the overlapping area being half the window size. For each local window, the unit calculates the mean and variance of the pixels within the window, and calculates the structural similarity between the two images within that window based on the mean and variance. The window size is pre-configured according to the image resolution before system initialization and stored in the system configuration parameters.

[0056] The difference calculation unit averages the structural similarity of all local windows to obtain the global structural similarity. The unit then combines the morphological gradient distance and the global structural similarity into a single difference value. The combination method is as follows: the morphological gradient distance is multiplied by a first weighting coefficient to obtain a first weighted value; the global structural similarity is multiplied by a second weighting coefficient to obtain a second weighted value; and the first and second weighted values ​​are summed to obtain the difference value. Both the first and second weighting coefficients are preset positive values ​​and are pre-configured based on the statistical distribution of the training data before system initialization.

[0057] The iterative adjustment unit receives the difference value output by the difference calculation unit as input. Internally, the iterative adjustment unit stores preset thresholds and a maximum number of iterations. The preset thresholds are a first threshold and a second threshold, corresponding to the morphological gradient distance and global structural similarity, respectively. The iterative adjustment unit compares the morphological gradient distance with the first threshold and the global structural similarity with the second threshold. If the morphological gradient distance is not greater than the first threshold and the global structural similarity is not less than the second threshold, the iterative adjustment unit determines that the iteration termination condition is met, outputs the current preliminary deformation field as the target deformation field, and terminates subsequent operations. If the morphological gradient distance is greater than the first threshold or the global structural similarity is less than the second threshold, the iterative adjustment unit sends an update trigger signal to the latent space alignment module. This triggers the latent space alignment module to readjust the combination coefficients and output the updated combination coefficients, which are then sent to the deformation field estimation module to generate a new preliminary deformation field. The iterative adjustment unit counts the number of iterations. When the number of iterations reaches the maximum number of iterations, even if the iteration termination condition has not yet been met, the iterative adjustment unit terminates the iteration and outputs the current preliminary deformation field as the target deformation field.

[0058] When the iterative adjustment unit determines that the current difference value is greater than a preset threshold, it initiates the update process for the combination coefficients. The update method is as follows: The iterative adjustment unit first triggers the gradient calculation process for the difference value relative to the combination coefficients. This gradient calculation is implemented through an automatic differentiation mechanism. Specifically, it uses each element in the combination coefficients as the independent variable and the difference value as the dependent variable, calculates the partial derivatives of the dependent variable with respect to each independent variable, and arranges all partial derivatives in the same order as the combination coefficients to form a gradient vector. Each component of this gradient vector indicates the trend of the difference value as it increases along the direction of each combination coefficient element. A positive component value indicates that adding that element will lead to an increase in the difference value, and a negative component value indicates that adding that element will lead to a decrease in the difference value.

[0059] The iterative adjustment unit determines the update direction and magnitude of the combination coefficients based on the gradient vector. Specifically, the update method is as follows: subtract the product of the gradient component value and the step size parameter from the current value of each element in the combination coefficients to obtain the updated values ​​of each element in the combination coefficients. The step size parameter is a pre-set positive scalar value used to control the distance moved along the gradient descent direction in a single update.

[0060] The step size parameter is set during system initialization. It can be set as a fixed constant, pre-configured and stored in the system configuration parameters based on the resolution of the input image. Alternatively, an adaptive method can be used, where the step size parameter decreases with each iteration according to a preset decay coefficient. The system configuration parameters also include a lower limit for the step size parameter. When the step size parameter decays below this lower limit during iterations, decay stops, and this lower limit is used as the step size parameter for subsequent iterations.

[0061] After performing an update operation, the iterative adjustment unit re-sends the updated combination coefficients to the deformation field estimation module. The deformation field estimation module then regenerates a new preliminary deformation field based on these updated combination coefficients and the current coupling characteristics. This new preliminary deformation field is then sequentially processed by the resampling unit's interpolation reconstruction and the difference calculation unit's difference evaluation to obtain new difference values. The iterative adjustment unit again determines the relationship between the new difference value and a preset threshold. If the threshold condition is still not met, the process of gradient calculation, combination coefficient update, and deformation field regeneration is repeated.

[0062] The iterative adjustment unit internally sets a maximum number of iterations, which is a preset positive integer value. Each time the update process begins, the iterative adjustment unit increments the iteration count and compares the current count with the maximum number of iterations. If the current count is less than the maximum number of iterations, the iterative adjustment unit allows the update process to continue. If the current count reaches the maximum number of iterations, the iterative adjustment unit forcibly terminates the iteration process, stops gradient calculation and combination coefficient update operations, and directly outputs the preliminary deformation field corresponding to the current round as the target deformation field.

[0063] Throughout the entire iteration process, the network parameters in the latent space alignment module, except for the combination coefficients, remain unchanged. The inference parameters of the temporal coding module, cell topology coding module, and coupled deformation module also remain unchanged. Only the combination coefficients are updated in each iteration. After each update, the regenerated deformation field gradually approaches the previous round in the direction of reducing the difference between the virtual reconstructed image and the endoscopic video frame image, until the threshold condition is met or the number of iterations is exhausted.

[0064] The label mapping module receives the target deformation field output by the closed-loop verification module and pre-annotated pathological region labels on the pathological slide image as input. The pathological region labels are label mask images with the same spatial resolution as the pathological slide image. Pixel positions belonging to the pathological region in the label mask image are assigned a first label value, and pixel positions not belonging to the pathological region are assigned a second label value. The label mapping module first analyzes the displacement vector at each pixel position in the target deformation field to obtain the complete displacement mapping relationship of each pixel position from the coordinate space of the pathological slide image to the coordinate space of the endoscopic video frame image.

[0065] The label mapping module iterates through each pixel position in the label mask image. For the current pixel position, it reads the corresponding displacement vector from the target deformation field and adds the row and column coordinates of the pixel position to the displacement vector to obtain the target coordinate position. The label mapping module checks whether the target coordinate position is within the spatial boundary of the endoscopic video frame image. If the target coordinate position is outside the boundary, the label information of that pixel is discarded; if the target coordinate position is within the boundary, the label value of that pixel is retained. For the retained label value, the label mapping module uses nearest neighbor interpolation to assign the label value to the target coordinate position. That is, it selects the nearest integer pixel from the four adjacent integer pixels of the target coordinate position as the landing point and records the label value in the overlay label frame corresponding to the landing point. After performing the above transformation and assignment operations on all pixel positions, the label mapping module outputs the overlay label frame. The spatial resolution of the overlay label frame is consistent with that of the endoscopic video frame image, and the label value at each pixel position indicates whether a pathological region exists in the original pathological slide image corresponding to that position and what type of pathological region exists.

[0066] The display output module receives the overlaid label frame and the original endoscopic video frame image from the label mapping module as two inputs, where the overlaid label frame and the endoscopic video frame image have the same spatial resolution. The display output module parses the pixel positions in the overlaid label frame, reading the label value at each position. The label value includes a first label value and a second label value, where the first label value corresponds to the pathological area and the second label value corresponds to the non-pathological area. The display output module assigns a display color to each pixel position based on the label value. Specifically, positions with the first label value are assigned a preset warning color value, and positions with the second label value are assigned a transparent color value, thereby generating a color overlay layer.

[0067] The display output module performs pixel-level overlay of the color overlay layer and the endoscopic video frame image. The overlay method involves covering the original pixel values ​​of the corresponding positions in the endoscopic video frame image with opaque areas in the color overlay layer, while retaining the original pixel values ​​of the corresponding positions in the endoscopic video frame image with transparent areas in the color overlay layer, resulting in a fused display frame. The display output module outputs the fused display frame to a display device connected to the system, allowing operators to observe the spatial distribution of pathological areas on the mucosal surface in real time during endoscopic procedures.

[0068] The system receives endoscopic video frame images from dynamic video streams of the digestive tract mucosa acquired by the endoscopic equipment. Operators select frames corresponding to specific time points from the dynamic video stream via the system's interactive interface as keyframe images. These keyframe images record the real-time image of the biopsy forceps reaching the mucosal surface. Simultaneously, the system receives pathological slide images corresponding to the same biopsy site as the keyframe image. These pathological slide images are obtained from digital pathological slides obtained after the biopsy tissue has been fixed, embedded, sectioned, stained, and digitally scanned. Before operation, the system determines the spatial correspondence between the keyframe images and pathological slide images through preoperative planning or intraoperative navigation positioning. Two images of the same biopsy site in different imaging modalities are associated as input data pairs. During system operation, the input data is processed sequentially through the aforementioned modules, with the output of one module serving as the input for the next. Finally, the label mapping module and display output module generate and present a superimposed display of the pathological area and the endoscopic video frame images.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A system for matching endoscopic videos and pathological images using deep learning, characterized in that, include: The latent space alignment module is used to project endoscopic video frame images and pathological slide images into a shared latent space, and in the shared latent space, constrain the two sets of projected features to satisfy a linear combination of the same set of topological deformation bases, and output the combination coefficients. The temporal coding module is used to extract the motion features of the endoscopic video frame images in the time dimension and output a temporal motion signature; The cell topology encoding module is used to extract the structural features of the pathological slide image in the spatial dimension and output the cell nucleus topology map. The coupling deformation module is used to apply the temporal motion signature as a boundary constraint to the cell nucleus topology graph, and to feed back the stiffness information of the cell nucleus topology graph to the temporal motion signature, and output coupling features; The deformation field estimation module is used to perform spatial transformation calculations based on the coupling characteristics and the combination coefficients, and output a preliminary deformation field. The closed-loop verification module is used to resample the pathological slide image based on the preliminary deformation field, generate a virtual reconstruction image, calculate the difference between the virtual reconstruction image and the endoscopic video frame image, and iteratively adjust the combination coefficients based on the difference until the iteration termination condition is met, at which point the target deformation field is output.

2. The endoscopic video and pathological image matching system combining deep learning according to claim 1, characterized in that: The latent space alignment module maps the feature distribution of the endoscopic video frame image to the shared latent space through the first reversible flow unit, maps the feature distribution of the pathological slide image to the shared latent space through the second reversible flow unit, and calculates the linear regression coefficients of the two sets of projected features relative to the topological deformation basis through the basis constraint unit, and uses the linear regression coefficients as the combination coefficients.

3. The endoscopic video and pathological image matching system combining deep learning according to claim 2, characterized in that: The temporal coding module extracts spatiotemporal features from multiple consecutive endoscopic video frames using a three-dimensional convolutional unit. Based on the extracted spatiotemporal features, the optical flow calculation unit calculates the dense optical flow field between adjacent frames. Based on the temporal statistics of the dense optical flow field, the signature generation unit generates the temporal motion signature, which includes the boundary value of tissue vibration amplitude and the principal component of vibration frequency.

4. The endoscopic video and pathological image matching system combining deep learning according to claim 3, characterized in that: The cell topology encoding module identifies the cell nucleus position coordinates in the pathological slide image through the cell nucleus detection unit, and generates the cell nucleus topology map by triangulation based on the cell nucleus position coordinates through the topology map construction unit. The vertices of the cell nucleus topology map are the cell nucleus center points, and the edges of the cell nucleus topology map are the connecting lines between adjacent cell nuclei. The anisotropic stiffness coefficient is calculated by the stiffness calculation unit based on the displacement response of each vertex in the cell nucleus topology map under preset stress conditions.

5. The endoscopic video and pathological image matching system combining deep learning according to claim 4, characterized in that: The coupling deformation module sets the boundary value of the tissue vibration amplitude in the temporal motion signature as the maximum displacement limit of each vertex when the cell nucleus topology is deformed by the boundary constraint unit, the anisotropic stiffness coefficient is used as a spatial attention weight and weighted to the optical flow confidence of the corresponding spatial position in the temporal motion signature by the stiffness feedback unit, and the boundary-constrained cell nucleus topology and the weighted temporal motion signature are spliced ​​in the channel dimension by the feature fusion unit to generate the coupling feature.

6. The endoscopic video and pathological image matching system combining deep learning according to claim 5, characterized in that: The deformation field estimation module maps the coupling features and the combination coefficients into spatial transformation parameters through a spatial transformation network unit. The deformation field generation unit constructs the preliminary deformation field based on the spatial transformation parameters. The preliminary deformation field includes the displacement vector of each pixel in the pathological slide image to the coordinate space of the endoscopic video frame image.

7. The endoscopic video and pathological image matching system combining deep learning according to claim 6, characterized in that: The closed-loop verification module uses a resampling unit to perform bilinear interpolation resampling on the pathological slide image based on the preliminary deformation field to generate a virtual reconstruction image. The difference calculation unit calculates the morphological gradient distance and structural similarity between the virtual reconstruction image and the endoscopic video frame image to obtain the difference. The iterative adjustment unit updates the combination coefficients and regenerates the preliminary deformation field when the difference is greater than a preset threshold.

8. The endoscopic video and pathological image matching system combining deep learning according to claim 7, characterized in that: The iterative adjustment unit adjusts the step size of the combination coefficients along the gradient descent direction based on the gradient direction of the difference relative to the combination coefficients. After each update, the difference is recalculated until the difference is less than or equal to the preset threshold. Then, the current deformation field is output as the target deformation field. The maximum number of iterations is a preset value. When the maximum number of iterations is reached, the iteration is terminated and the current deformation field is output.

9. The endoscopic video and pathological image matching system combining deep learning according to claim 8, characterized in that: It also includes a label mapping module, which transforms the pathological region labels marked on the pathological slide image to the coordinate space of the endoscopic video frame image based on the target deformation field to generate superimposed label frames.

10. The endoscopic video and pathological image matching system combining deep learning according to claim 9, characterized in that: It also includes a display output module, which overlays the superimposed label frame with the endoscopic video frame image; the endoscopic video frame image is a key frame image in the dynamic video of gastrointestinal mucosal endoscopy, and the pathological slide image is a digital pathological slide corresponding to the biopsy point of the key frame image.

Images