Multimodal medical image analysis system, medium and device for gastrointestinal disorder identification

Abstract

The present application belongs to the technical field of medical image processing, and provides a multi-modal medical image analysis system, medium and equipment for gastrointestinal disease recognition, which extracts multi-scale texture features and structure features of a gastrointestinal endoscope color image and a gastrointestinal endoscope OCT depth image respectively, fuses to obtain a cross-modal feature sequence; learns spatial correspondence of different modal data on blood vessel morphology and gastrointestinal wall interlayer structure based on the feature sequence after adding position coding; obtains a multi-scale global context feature sequence and a local detail feature sequence based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, fuses to obtain a fused multi-modal feature; obtains an adjusted dense deformation field based on the fused multi-modal feature; performs coarse registration and fine registration based on the adjusted dense deformation field, and obtains registered gastrointestinal endoscope color images and gastrointestinal endoscope OCT depth images. The application has strong anti-interference ability and high data stability.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of medical image processing, and particularly relates to a multi-modal medical image analysis system for gastrointestinal disease identification, a medium and an apparatus. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] In the biomedical detection field, analyzing blood oxygen saturation by means of gastrointestinal mucosa images is a key means for early detection of gastrointestinal and systemic diseases. As an important digestive and absorptive tissue of the human body, the blood oxygen saturation of the blood vessels of the gastrointestinal mucosa, such as arterial blood oxygen saturation and venous blood oxygen saturation, can directly reflect the local metabolic state and the systemic circulatory function. By analyzing the blood oxygen level of the blood vessels of the gastrointestinal mucosa, doctors can indirectly assess gastrointestinal diseases such as gastritis, gastric ulcer, colorectal cancer, inflammatory bowel disease, and gastrointestinal polyps, as well as systemic diseases such as hypertension, coronary heart disease, stroke risk, chronic kidney disease, and Alzheimer's disease.

[0004] Currently, the commonly used imaging techniques mainly include three types: gastrointestinal endoscopic color imaging, gastrointestinal endoscopic optical coherence tomography (OCT), and laser Doppler flowmetry (LDF). Gastrointestinal endoscopic color imaging uses different wavelength light sources (such as 570 nm and 605 nm) to analyze the color difference of blood vessels to calculate blood oxygen saturation; OCT combined with blood flow imaging (OCT-A) assesses the blood oxygen state of microvessels; and LDF measures the correlation between blood flow velocity and blood oxygen concentration. However, these traditional methods calculate blood oxygen saturation based on the spectral absorption difference between pigment hemoglobin and deoxyhemoglobin, and have many defects:

[0005] Limited imaging quality: the complex internal environment of the gastrointestinal tract (such as mucus interference and intestinal peristalsis) leads to insufficient resolution and contrast of the gastrointestinal mucosa images, which in turn seriously affects the accuracy of the data;

[0006] Significant environmental interference: external factors such as endoscopic light source intensity and mucosal surface reflection can easily cause deviations in spectral data, making the detection results unstable;

[0007] Poor data stability: a large number of samples need to be collected manually for verification, and the entire process is complex and time-consuming, making it difficult to meet the real-time detection requirements. SUMMARY

[0008] In order to solve at least one technical problem in the background art, the present application provides a multi-modal medical image analysis system, medium and device for gastrointestinal disease recognition, which has strong anti-interference ability, high data stability and can meet the real-time gastrointestinal lesion positioning requirements.

[0009] In order to achieve the above-mentioned purpose, the present application adopts the following technical solutions:

[0010] The first aspect of the present application provides a multi-modal medical image analysis system for gastrointestinal disease recognition, comprising the following steps:

[0011] The modal feature encoding and position perception module is used to extract multi-scale texture features and structure features of the gastrointestinal endoscope color image and the gastrointestinal endoscope OCT depth image respectively, and fuse the extracted multi-scale texture features and structure features to obtain a cross-modal feature sequence.

[0012] The dynamic feature alignment module is used to align the features between the modalities based on the feature sequence after adding the position encoding, combined with the cross attention layer, by sharing the query key-value matrix, to learn the spatial correspondence of different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure.

[0013] The feature extraction module is used to extract a multi-scale global context feature sequence and a local detail feature sequence based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, and fuse the multi-scale and local detail features to obtain a fused multi-modal feature.

[0014] The dynamic deformation field generation and calibration module is used to convert the fused multi-modal feature into a displacement field, then up-sample the displacement field to the original image resolution, generate a dense deformation field, introduce a physiological signal, and adjust the dense deformation field based on the physiological signal to obtain an adjusted dense deformation field.

[0015] The multi-stage registration module is used to perform coarse registration and fine registration based on the adjusted dense deformation field, to obtain the registered gastrointestinal endoscope color image and gastrointestinal endoscope OCT depth image.

[0016] Further, in the modal feature encoding and position perception module, the cross-modal blood vessel feature is extracted based on the obtained gastrointestinal endoscope color image and gastrointestinal endoscope OCT depth image, which includes inputting the gastrointestinal endoscope color image and the gastrointestinal endoscope OCT depth image into corresponding CNN encoders respectively, extracting multi-scale texture features and structure features, and fusing the extracted multi-scale texture features and structure features to obtain a cross-modal feature sequence.

[0017] Further, in the modal feature encoding and position-aware module, based on the encoded feature sequence, the spatial correspondence of different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure is learned, including combining a cross-attention layer, taking the gastrointestinal endoscopy image tokens as a query, and the gastrointestinal endoscopy OCT image tokens as a key value, calculating an attention weight, and combining the attention weight to learn the spatial correspondence of different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure.

[0018] Further, in the feature extraction module, a multi-head global attention mechanism is combined, each head captures global context information of different scales to obtain a multi-scale global context feature sequence, and the multi-scale global context feature sequence and the interlayer structure tokens of the gastrointestinal endoscopy OCT depth image are combined to extract a local detail feature sequence by combining a local variability attention mechanism.

[0019] Further, in the feature extraction module, each head captures global context information of different scales, specifically including: head 1 and head 2 capture large blood vessel trunks, head 3 and head 4 focus on gastrointestinal lesion edges, and head 5 and head 6 capture microvessel details.

[0020] Further, in the feature extraction module, when the global context information and the local detail feature are fused to obtain the fused multi-modal feature, a pyramid feature fusion strategy is adopted, the global blood vessel distribution feature and the local microvessel texture feature are superimposed at the bottom layer, the global gastrointestinal specific reference region relative position feature and the local gastrointestinal lesion edge gradient feature are fused at the middle layer, and the global spectral feature and the local blood vessel bifurcation detail feature are combined at the high layer.

[0021] Further, in the dynamic deformation field generation and calibration module, the dynamic adjustment of the dense deformation field parameters is represented as:

[0022] ,

[0023] wherein, is the adjusted dense deformation field parameter, is the unadjusted dense deformation field parameter, is a physiological signal influence weight coefficient, is an activation value of the encoded physiological signal of the gated recurrent unit, and S is an output state vector.

[0024] Further, the system further comprises a gastrointestinal lesion positioning module, inputting the registered gastrointestinal endoscopy color image into a backbone network of a YOLOv5 model, extracting two-dimensional texture features of the gastrointestinal lesion, performing depth slice feature extraction on the registered gastrointestinal endoscopy OCT depth image, obtaining three-dimensional structure features of the gastrointestinal lesion region, inputting the two-dimensional texture features of the gastrointestinal lesion and the three-dimensional structure features of the gastrointestinal lesion region into a detection head of the YOLOv5 after fusion, and outputting a gastrointestinal lesion candidate box and a confidence, and determining a gastrointestinal lesion center coordinate and a range according to the gastrointestinal lesion candidate box.

[0025] The second aspect of the application provides a computer readable storage medium.

[0026] A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements:

[0027] Respectively extracting multi-scale texture features and structure features of the gastrointestinal endoscopy color image and the gastrointestinal endoscopy OCT depth image, and fusing the extracted multi-scale texture features and structure features to obtain a cross-modal feature sequence;

[0028] Based on the feature sequence after adding the position coding, combining a cross-attention layer, aligning the features between the modalities through a shared query key-value matrix, and learning the spatial correspondence of the different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure;

[0029] Based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, extracting a multi-scale global context feature sequence and a local detail feature sequence, and fusing the multi-scale and local detail features to obtain a fused multi-modal feature;

[0030] Converting the fused multi-modal feature into a displacement field, upsampling the displacement field to the original image resolution, generating a dense deformation field, introducing a physiological signal, adjusting the dense deformation field based on the physiological signal, and obtaining an adjusted dense deformation field;

[0031] Performing coarse registration and fine registration based on the adjusted dense deformation field, and obtaining the registered gastrointestinal endoscopy color image and the gastrointestinal endoscopy OCT depth image.

[0032] The third aspect of the application provides a computer device.

[0033] A computer device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the following when executing the program:

[0034] Respectively extracting multi-scale texture features and structure features of the gastrointestinal endoscopy color image and the gastrointestinal endoscopy OCT depth image, and fusing the extracted multi-scale texture features and structure features to obtain a cross-modal feature sequence;

[0035] Based on the feature sequence after adding the position code, combined with the cross attention layer, the modal interlayer structure features of different modal data on the blood vessel shape and the gastrointestinal wall are learned by aligning the modal interlayer structure features through the shared query key-value matrix;

[0036] Based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, multi-scale global context feature sequences and local detail feature sequences are extracted, and the multi-scale and local detail features are fused to obtain fused multi-modal features.

[0037] The fused multi-modal features are converted into a displacement field, and then the displacement field is upsampled to the original image resolution to generate a dense deformation field, a physiological signal is introduced, and the dense deformation field is adjusted based on the physiological signal to obtain an adjusted dense deformation field.

[0038] Based on the adjusted dense deformation field, coarse registration and fine registration are performed to obtain registered gastrointestinal endoscopic color images and gastrointestinal endoscopic OCT depth images.

[0039] Compared with the prior art, the beneficial effects of the present application are:

[0040] 1、The present application uses self-attention mechanism to model the global correlation of the feature sequence of multi-modal images, calculates the attention weight of any two pixel features (such as the feature vectors of key structures such as blood vessel branch nodes, gastrointestinal lesion edge contours, and microvessel textures) in the feature sequence, accurately quantifies the correlation strength of different regions in spatial position and structural morphology, and thus includes global structural features such as the continuity of the main stem and branches of the blood vessels and the distribution relationship between the gastrointestinal lesions and the surrounding blood vessels in the feature learning category, breaks through the visual limitations of traditional local feature extraction methods, establishes a feature dependency relationship at the whole image scale, and improves the accuracy of gastrointestinal lesion positioning.

[0041] 2、The present application adopts a multi-subspace learning strategy of multi-head attention for special areas (such as gastrointestinal mucosa reflection, shadow generated by blood vessel intersection, and low-contrast microvessel area) in the gastrointestinal mucosa image that are easily disturbed: different attention subspaces focus on local detail features (such as the edge texture of microvessels and the boundary gradient of gastrointestinal lesions) and global context features (such as the overall distribution trend of the blood vessel network and the relative position relationship between the gastrointestinal lesions and the gastrointestinal specific reference area), and through complementary fusion of multi-subspace features, effectively weaken the interference of local noise (such as pixel value anomaly caused by reflection and feature blur caused by shadow) on global registration, and ensure that key structural features can still be accurately identified and strengthened in complex backgrounds.

[0042] The advantages of the additional aspects of the present application will be partially given in the following description, partially will become apparent from the following description, or will be understood by the practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0043] The accompanying drawings, which constitute a part of this specification, are incorporated herein to provide further

[0044] Figure 1 is a block diagram of a multi-modality medical image analysis system for gastrointestinal disease identification provided by embodiments of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described with reference to the drawings and embodiments.

[0046] It should be noted that the following detailed description is exemplary in nature, and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0047] It is also to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting, as the scope of the application will be limited only by the appended claims. Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. Although any methods and materials similar or equivalent to those described herein can be used in the practice or testing of the present application, the preferred methods and materials are described. Unless otherwise indicated, the materials used in the examples are commercially available.

[0048] In view of the complex blood vessel distribution in the gastrointestinal mucosa image, the gastrointestinal lesion shape and the structural characteristics easily affected by intestinal peristalsis, the application fully utilizes the advantages of the self-attention mechanism of the Transformer to extract the global features of the image, so as to significantly improve the robustness of registration. First, the self-attention mechanism is used to model the global correlation of the feature sequence of the multi-modal image. By calculating the attention weight of any two pixel features (such as the feature vectors of key structures such as blood vessel branch nodes, gastrointestinal lesion edge contours and microvessel textures) in the feature sequence, the correlation strength of different regions in the spatial position and structural form is accurately quantified. Therefore, the continuity of the blood vessel trunk and branches, the distribution relationship between the gastrointestinal lesion and the surrounding blood vessels and other global structural features are included in the feature learning category, breaking through the visual limitations of traditional local feature extraction methods and establishing a feature dependency relationship at the whole image scale. At the same time, for the special areas in the gastrointestinal mucosa image that are easily disturbed (such as the reflection of the gastrointestinal mucosa, the shadow generated by the intersection of blood vessels and the low-contrast microvessel area), a multi-subspace learning strategy of multi-head attention is adopted: different attention subspaces focus on local detail features (such as the edge texture of microvessels and the boundary gradient of gastrointestinal lesions) and global context features (such as the overall distribution trend of the blood vessel network and the relative position relationship between the gastrointestinal lesion and the gastrointestinal specific reference area). Through the complementary fusion of multi-subspace features, the interference of local noise (such as abnormal pixel values caused by reflection and feature blurring caused by shadow) on global registration is effectively weakened, ensuring that key structural features can still be accurately identified and strengthened in complex backgrounds.

[0049] Embodiment one

[0050] As shown in Figure 1 , the embodiment provides a multi-modal medical image analysis system for gastrointestinal disease identification, including the following steps:

[0051] An image acquisition module is used to acquire gastrointestinal endoscopic color images and gastrointestinal endoscopic OCT depth images.

[0052] In the image acquisition module, a multi-spectral gastrointestinal endoscope is used to acquire color images and grayscale images, covering wavelength bands of 550 nm, 600 nm, etc. A gastrointestinal endoscopic optical coherence tomography image acquisition module is introduced to acquire the depth information of the gastrointestinal wall.

[0053] In this embodiment, when acquiring the gastrointestinal endoscopic color image, the endoscope parameters are set to high resolution (such as 4K or higher) and short exposure time (such as 1 / 1000 second) to ensure that the image is clear, complete and can accurately reflect the spectral information of the gastrointestinal mucosa.

[0054] An image preprocessing module is used to denoise and enhance the contrast of the acquired gastrointestinal endoscopic color images and gastrointestinal endoscopic OCT depth images.

[0055] In the image preprocessing module, the median filtering algorithm is used to remove mucus interference noise in the image, and the histogram equalization method is used to enhance the contrast of the image. For gastrointestinal endoscopy OCT images, wavelet transform is used to denoise and preserve the image details (such as the interlayer structure of the gastrointestinal wall and microvascular texture).

[0056] The cross-modal feature encoding and position-aware module is used to input the color image of the gastrointestinal endoscope and the depth image of the gastrointestinal endoscope OCT to the corresponding CNN encoder, respectively, to extract multi-scale texture features and structural features, and to fuse the extracted multi-scale texture features and structural features to obtain a cross-modal feature sequence.

[0057] In this embodiment, the CNN encoder adopts a lightweight design, such as MobileNetV3, which reduces the computational load while maintaining the feature resolution (512×512). The extracted multi-scale texture features include features such as blood vessel bifurcation, gastrointestinal lesion edges, and microvascular textures, while the structural features include the thickness of the gastrointestinal wall layers. The extracted multi-scale texture features and structural features are converted into sequence tokens through linear projection, with each token corresponding to a 16×16 pixel region, forming a feature sequence of dimension N×D, where N is the number of tokens and D is the dimension of the feature vector. In this embodiment, D=512.

[0058] The positional encoding addition module is used to add positional encoding to cross-modal feature sequences to obtain feature sequences with added positional encoding.

[0059] In this embodiment, a hybrid positional coding strategy is adopted. Sinusoidal positional codes are added to the tokens of the gastrointestinal endoscopy images to capture global spatial relationships, and learnable positional codes are added to the tokens of the gastrointestinal endoscopy OCT images to adapt to nonlinear changes in the depth direction. The specific hybrid positional coding strategy is expressed as follows:

[0060] ,

[0061] in, Indicates the position code at the 1st i The first feature token, the first j The values ​​in each dimension are used to assign spatial location information to the features of gastrointestinal endoscopy images or gastrointestinal endoscopy OCT images, helping the model to perceive the spatial distribution relationship of the features; This is a learned location embedding function designed for gastrointestinal endoscopic OCT images. The input is the index i of a feature token, and the output is the learned location encoding vector corresponding to that token. Since gastrointestinal endoscopic OCT is a tomographic imaging technique of the gastrointestinal wall (with complex inter-layer structures), "learned encoding" can more flexibly capture the locational correlations of its features. i Indexed by tokenj is a dimension index.

[0062] a dynamic feature alignment module, configured to align the features of different modalities based on the positionally encoded feature sequence, by combining a cross-attention layer, learning the spatial correspondence of different modalities in the blood vessel morphology and the interlayer structure of the gastrointestinal tract wall through a shared query key-value matrix.

[0063] In this embodiment, a cross-attention layer is added after the encoder, taking the gastrointestinal endoscopy image tokens as the query Q and the gastrointestinal endoscopy OCT image tokens as the key-value (K-V) to calculate the attention weight:

[0064]

[0065] After obtaining the attention weight, the spatial correspondence of different modalities in the blood vessel morphology and the interlayer structure of the gastrointestinal tract wall is learned in the following way: the gastrointestinal endoscopy image tokens (Q) capture two-dimensional structural features such as blood vessel bifurcation and gastrointestinal lesion edge, and the gastrointestinal endoscopy OCT image tokens (K-V) provide three-dimensional structural features such as the interlayer thickness of the gastrointestinal tract wall and the depth of the blood vessels; the attention weight will assign higher weights to the features in K that match the structure of Q (such as the gastrointestinal endoscopy blood vessel bifurcation point corresponding to the specific interlayer boundary in OCT), and realize the spatial correlation mapping of inter-modal features through the weighted sum of V; at the same time, by minimizing the inter-modal feature difference loss (such as L1 loss) through backpropagation, the parameters of the Q-K-V matrix are iteratively optimized, so that the aligned feature sequence achieves the optimal in the blood vessel morphology continuity (such as the trunk-branch connection) and the interlayer structure consistency (such as the depth of the mucosa and muscle layer corresponding to the gastrointestinal lesion area).

[0066] a feature extraction module, configured to obtain a multi-scale global context feature sequence based on the aligned cross-modal blood vessel features, by combining a multi-head global attention mechanism, each head capturing different scale global context information, and obtain a local detail feature sequence by combining the multi-scale global context feature sequence and the interlayer structure tokens of the gastrointestinal endoscopy OCT depth image, and combining a local variability attention mechanism, and obtain a fused multi-modal feature by fusing the multi-scale and local detail features; the feature extraction module specifically includes a global information extraction module, a local information extraction module, and a feature fusion module.

[0067] ​Wherein, the global information extraction module is used to adopt a multi-head global attention encoder, each head independently captures different scale global context; input: two modal aligned key structure feature sequence (dimension NxD, N is the number of tokens, D=512, including gastrointestinal mucosa blood vessel texture, gastrointestinal endoscopic OCT interlayer structure and other alignment features); output: multi-scale global context feature sequence (dimension remains NxD, each token integrates the structure correlation information of the corresponding scale in the whole graph, such as head 1-2 output global distribution characteristics of large blood vessel trunk, head 3-4 output global contour characteristics of gastrointestinal lesion edge, head 5-6 output global network characteristics of microvessel). Specifically, head 1 and head 2 capture large blood vessel trunks, head 3 and head 4 focus on gastrointestinal lesion edges, and head 5 and head 6 capture microvessel details; in this embodiment, the receptive field of head 1 and head 2 is 64x64, the receptive field of head 3 and head 4 is 32x32, and the receptive field of head 5 and head 6 is 16x16; after obtaining the global features, the residual connection and layer normalization (LayerNorm) are used to alleviate the gradient disappearance and ensure the effective transmission of the global features.

[0068] The present application is aimed at the special areas in the gastrointestinal mucosa image which are easily disturbed (such as gastrointestinal mucosa reflection, shadow generated by blood vessel intersection, low-contrast microvessel area, etc.), and a multi-subspace learning strategy of multi-head attention is adopted: different attention subspaces focus on local detail features (such as edge texture of microvessel and boundary gradient of gastrointestinal lesion) and global context features (such as overall distribution trend of blood vessel network and relative position relationship between gastrointestinal lesion and gastrointestinal specific reference area), through complementary fusion of multi-subspace features, the interference of local noise (such as abnormal pixel value caused by reflection and feature blur caused by shadow) on global registration is effectively weakened, and the key structure features can still be accurately identified and strengthened in the complex background.

[0069] Wherein, the local information extraction module is used to introduce deformable attention in the Transformer decoder, input: multi-scale global context feature sequence output by the global information extraction module (as query token, focusing on the local area needing to be refined, such as blood vessel bifurcation and low-contrast microvessel area), interlayer structure tokens of the original gastrointestinal endoscopic OCT image (as reference feature library); output: sub-pixel level aligned local detail feature sequence (dimension NxD, each query token corresponds to the weighted features of 8 reference points, highlighting the fine correlation of local structure, such as OCT interlayer boundary matching features at the blood vessel bifurcation). For example, for the blood vessel bifurcation area in the gastrointestinal mucosa image, the model automatically samples the interlayer boundary tokens of the surrounding gastrointestinal endoscopic OCT image, realizing sub-pixel level alignment. The sampling formula is:

[0070] ,

[0071] wherein, is a predicted offset, is a weight, is a number of reference points, in the present embodiment = 8, is a feature vector corresponding to the final sampling position in the reference feature library;

[0072] The feature fusion module is configured to employ a pyramid feature fusion strategy to associate and fuse the multi-scale global features output by the global attention encoder and the local detail features output by the local deformation perception module to obtain a fused feature vector.

[0073] In the present embodiment, the pyramid feature fusion strategy includes associating and fusing the multi-scale global features output by the global attention encoder and the local detail features output by the local deformation perception module through cross-layer skip connection (SkipConnection): the bottom layer (4x down-sampling) superimposes the global blood vessel distribution features and the local micro blood vessel texture features to strengthen the integrity of the blood vessel network; the middle layer (2x down-sampling) fuses the global gastrointestinal lesion and the gastrointestinal specific reference region relative position features and the local gastrointestinal lesion edge gradient features to improve the structure positioning accuracy; the high layer (original image) combines the global spectral features and the local blood vessel bifurcation detail features to optimize the microstructure recognition; element-level addition and channel attention weighting (assigning higher weights to key structure feature channels) are employed in the fusion process to ensure the effective association of global-local features.

[0074] The dynamic deformation field generation and calibration module is configured to convert the fused multi-modal features into a displacement field, up-sample the displacement field to the resolution of the original image, generate a dense deformation field, introduce a physiological signal, adjust the dense deformation field based on the physiological signal, and obtain an adjusted dense deformation field.

[0075] In the present embodiment, in the process of the dense deformation field, the attention weights output by the decoder are converted into a displacement field (Δx, Δy) through a 2-layer fully connected network, and each token corresponds to a two-dimensional displacement vector. The displacement field is up-sampled to the resolution of the original image (512x512) through bilinear interpolation to form a dense deformation field , which is expressed as:

[0076] ,

[0077] The dense deformation field generated by the formula is used for geometric correction of the gastrointestinal endoscopic OCT image: is the offset of each pixel (x, y) on the image in the horizontal and vertical directions; the pixel position is adjusted through pixel resampling (such as bilinear interpolation), and the structural misalignment of the image and the gastrointestinal endoscopic image caused by intestinal peristalsis (such as ± 15° deformation, local stretching) is corrected, and corrected image data is provided for the subsequent registration step of gastrointestinal lesion positioning. The deformation field directly acts on the gastrointestinal endoscopic OCT image, and corrects the local stretching caused by intestinal peristalsis (such as ± 15° deformation).

[0078] In this embodiment, physiological signals are introduced, and the dense deformation field is adjusted based on the physiological signals to obtain an adjusted dense deformation field, which includes:

[0079] The physiological signals such as heart rate (HR) and blood pressure (BP) are introduced as auxiliary inputs, which are encoded into state vectors s through a gated recurrent unit (GRU), and the deformation field parameters θ are dynamically adjusted through the following formula:

[0080] , ,

[0081] Wherein, is the dense deformation field parameter before adjustment (containing the , offset matrix of all pixels); is the physiological signal influence weight coefficient (empirical value is set to 0.1-0.3, which controls the adjustment amplitude of the physiological signal on the deformation field, and avoids excessive correction); is the activation value of the physiological signal after being encoded by the gated recurrent unit (GRU), and S is the output state vector (fusing the time sequence features of heart rate and blood pressure), and the ReLU function ensures that the activation value is non-negative, avoiding reverse adjustment; is the dense deformation field parameter after adjustment (fusing physiological signal feedback, such as appropriately reducing the local deformation amplitude when the heart rate is too fast, and improving the stability of the deformation field).

[0082] The multi-stage registration module is used for coarse registration and fine registration based on the adjusted dense deformation field, and the registered gastrointestinal endoscopic color image and gastrointestinal endoscopic OCT depth image are obtained.

[0083] In this embodiment, in the coarse registration stage: the encoder is fixed (to avoid bottom feature offset), and only the parameters are optimized; the optimizer (learning rate set to 1e-4) is used, and the mean square error (MSE) is used as the loss function (to calculate the pixel value difference between the corrected gastrointestinal endoscopic OCT image and the gastrointestinal endoscopic image); 200 iterations are performed until MSE<0.05, and global structure alignment (such as matching of large blood vessel trunks and approximate areas of gastrointestinal lesions) is realized.

[0084] Fine registration stage: unfreeze the CNN encoder (allow fine-tuning of the underlying features), introduce structural similarity (SSIM) loss (measure the structural similarity of the image, weight 0.6) and edge-preserving loss (Edge-Preserving Loss, weight 0.4, avoid edge blur by edge gradient penalty); use AdamW optimizer (learning rate set to 5e-5, weight decay 1e-5), focus on optimizing the alignment accuracy of the bifurcation points of blood vessels (through key point matching loss) and the interlayer boundary of gastrointestinal endoscopic OCT (through interlayer distance loss); iterate 50-100 rounds until SSIM>0.92, achieve accurate alignment of fine structures such as gastrointestinal lesion edges and microvessels.

[0085] Optic disc positioning module for fusing the registered gastrointestinal endoscopic color image and gastrointestinal endoscopic OCT depth image to obtain gastrointestinal lesion positioning results;

[0086] In this embodiment, the registered gastrointestinal endoscopic color image is input into the backbone network (CSPDarknet) of the YOLOv5 model to extract two-dimensional texture features of the gastrointestinal tract, including surface morphological features such as gray difference and edge profile between the gastrointestinal lesion and the surrounding tissue.

[0087] Depth slice feature extraction is performed on the registered gastrointestinal endoscopic OCT depth image to obtain three-dimensional structural features of the gastrointestinal lesion region, such as gastrointestinal lesion infiltration depth and thickness distribution of each layer of the gastrointestinal tract wall; after fusing the two-dimensional texture features of the gastrointestinal lesion and the three-dimensional structural features of the gastrointestinal lesion region, the detection head of YOLOv5 is input to output gastrointestinal lesion candidate boxes and confidence; the gastrointestinal lesion center coordinates and range are determined according to the gastrointestinal lesion candidate boxes; in this embodiment, the non-maximum suppression algorithm (NMS threshold is set to 0.5 in this embodiment) is used to remove duplicate detection boxes, and the shape (irregular or approximately circular) and location prior information (usually located in a specific section of the gastrointestinal tract) of the gastrointestinal lesion are combined for verification and correction, and finally the accurate gastrointestinal lesion center coordinates and range are output.

[0088] Specifically, when fusing the two-dimensional texture features of the gastrointestinal lesion and the three-dimensional structural features of the gastrointestinal lesion region, a convolution attention mechanism can be used to fuse the above two types of features, dynamically allocate weights to enhance the role of depth features, for example, when distinguishing between gastrointestinal lesions and adjacent normal tissues (such as inflammatory areas and blood vessel dense areas), preferentially use the three-dimensional structural features of the gastrointestinal endoscopic OCT to improve the discrimination; the final output gastrointestinal lesion positioning result is used to define the region of interest (ROI): to delineate the key area for subsequent gastrointestinal mucosa blood vessel segmentation and spectral analysis, reduce the interference of irrelevant background areas in the gastrointestinal mucosa image, and improve the efficiency and accuracy of subsequent analysis.

[0089] a multi-task learning module configured to learn a gastrointestinal mucosal blood vessel segmentation result and a blood oxygen saturation based on the gastrointestinal lesion positioning result; in this embodiment, the multi-task learning strengthens the synergy of blood vessel structure recognition (assisting gastrointestinal lesion positioning) and blood oxygen calculation (assisting disease assessment), and the specific content is as follows: the gastrointestinal mucosal blood vessel segmentation and the blood oxygen saturation prediction are combined into a multi-task learning problem, and a double-tower architecture of "shared encoder + task-specific decoder" is designed to realize collaborative optimization: an improved UNet encoder is used as a shared feature extraction layer, which includes 5 down-sampling stages, and multi-scale features are extracted through residual blocks and skip connections: shallow features capture details such as blood vessel edges, middle features depict blood vessel morphology and spectral characteristics, and deep features aggregate global blood vessel distribution information, which serve both tasks to avoid redundant extraction.

[0090] The blood vessel segmentation task outputs a binary blood vessel mask through the up-sampling branch, and the blood oxygen prediction task outputs the arterial / venous blood oxygen values through the fully connected branch; the cross-entropy loss function is used to optimize the blood vessel segmentation task (the weighted cross-entropy is used to solve the class imbalance, and the blood vessel pixel weight is set to 5 and the background is 1), and the mean square error loss function is used to optimize the blood oxygen saturation prediction task (the arterial and venous blood oxygen prediction errors are calculated respectively). The total loss function of multi-task learning is:

[0091] ,

[0092] wherein, is the cross-entropy loss of blood vessel segmentation, is the mean square error loss of blood oxygen saturation prediction, and are weight coefficients.

[0093] The blood oxygen saturation learning task is based on a deep learning spectral analysis model, which directly learns the mapping relationship of blood oxygen saturation from the multispectral image, rather than relying only on the traditional optical density ratio and Beer-Lambert law.

[0094] Specifically, a convolutional neural network (CNN) or a Transformer network is used for end-to-end blood oxygen saturation prediction, and the model input is the gray value of the multispectral image, and the output is the arterial and venous blood oxygen saturation value.

[0095] The formula of blood oxygen saturation is:

[0096] wherein, is the optical density ratio, and are empirical parameters.

[0097] ​Further, a dynamic calibration mechanism is introduced to adjust the blood oxygen saturation calculation model according to individual differences and real-time physiological states of patients.

[0098] In combination with physiological parameters such as heart rate and blood pressure, the empirical parameters are adjusted in real time and to make the calculation results more consistent with the actual blood oxygen saturation values; the calculated optical density ratio is filtered using methods such as median filtering or Gaussian filtering to remove noise interference. At the same time, a tissue scattering model is established to correct the optical density measurement deviation caused by gastrointestinal mucosa tissue scattering.

[0099] In combination with physiological parameters such as heart rate (HR), blood pressure (BP), and respiratory rate (RR), the empirical parameters are dynamically adjusted a and b using the following formula:

[0100] ,

[0101] ,

[0102] The dynamically calibrated blood oxygen saturation is ,

[0103] wherein, is the current measured heart rate (Heart Rate), i.e., the number of times the heart beats per minute, is the reference heart rate, which refers to the heart rate of the human body in the "calm, basal metabolism" state (such as resting heart rate), serving as a reference benchmark for physiological state, is the current measured systolic blood pressure (Systolic Blood Pressure), which is the pressure in the artery when the heart contracts, is the reference systolic blood pressure, which is the reference value of the systolic pressure of the human body in the basal state, is the current measured diastolic blood pressure (Diastolic Blood Pressure), which is the pressure in the artery when the heart relaxes, is the reference diastolic blood pressure, which is the reference value of the diastolic pressure of the human body in the basal state, a and b are the empirical parameters before dynamic adjustment, and are the empirical parameters after dynamic adjustment, is the heart rate-related adjustment coefficient, is the blood pressure-related adjustment coefficient. This dynamic adjustment mechanism can significantly improve the accuracy of blood oxygen saturation calculation and make it closer to the true value.

[0104] Example Two

[0105] ​The embodiment provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the following steps:

[0106] Multi-scale texture features and structure features of the gastrointestinal endoscope color image and the gastrointestinal endoscope OCT depth image are extracted respectively, and the extracted multi-scale texture features and structure features are fused to obtain a cross-modal feature sequence;

[0107] Based on the feature sequence after adding the position encoding, the cross attention layer is combined, the inter-modal features are aligned through a shared query key-value matrix, and the spatial correspondence of different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure is learned;

[0108] Based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, multi-scale global context feature sequences and local detail feature sequences are extracted, and the multi-scale and local detail features are fused to obtain fused multi-modal features;

[0109] The fused multi-modal features are converted into a displacement field, the displacement field is up-sampled to the original image resolution, a dense deformation field is generated, a physiological signal is introduced, the dense deformation field is adjusted based on the physiological signal, and an adjusted dense deformation field is obtained.

[0110] Based on the adjusted dense deformation field, coarse registration and fine registration are performed, and the registered gastrointestinal endoscope color image and the registered gastrointestinal endoscope OCT depth image are obtained.

[0111] Embodiment three

[0112] The embodiment provides a computer device, which comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, and the processor implements the following steps when executing the program:

[0113] Multi-scale texture features and structure features of the gastrointestinal endoscope color image and the gastrointestinal endoscope OCT depth image are extracted respectively, and the extracted multi-scale texture features and structure features are fused to obtain a cross-modal feature sequence;

[0114] Based on the feature sequence after adding the position encoding, the cross attention layer is combined, the inter-modal features are aligned through a shared query key-value matrix, and the spatial correspondence of different modal data on the blood vessel morphology and the gastrointestinal wall interlayer structure is learned;

[0115] Based on the aligned cross-modal blood vessel and gastrointestinal wall interlayer structure features, multi-scale global context feature sequences and local detail feature sequences are extracted, and the multi-scale and local detail features are fused to obtain fused multi-modal features;

[0116] The fused multi-modal features are converted into a displacement field, and the displacement field is up-sampled to the resolution of the original image to generate a dense deformation field, a physiological signal is introduced, and the dense deformation field is adjusted based on the physiological signal to obtain an adjusted dense deformation field;

[0117] Based on the adjusted dense deformation field, coarse registration and fine registration are performed to obtain a registered gastrointestinal endoscope color image and a gastrointestinal endoscope OCT depth image.

[0118] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage, etc.) containing computer-usable program code.

[0119] The present application is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus generate a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0120] These computer program instructions can also be stored in a computer-readable memory that can direct the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including instruction means, which implement the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0121] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable data processing apparatus to produce a computer-implemented process, so that the instructions executed on the computer or other programmable data processing apparatus provide a means for implementing the functions specified in the flowcharts and / or block diagrams. Figure 1 one or more flows and / or blocks Figure 1 means for performing the functions specified in the flowchart

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by a computer program instructing relevant hardware, and the program can be stored in a computer readable storage medium. When the program is executed, the program can include the processes of the above-mentioned embodiment methods. The storage medium can be a magnetic disc, an optical disc, a read-only memory (ROM), a random access memory (RAM), or the like.

[0123] The above only describes the preferred embodiments of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal medical image analysis system for identifying gastrointestinal diseases, characterized in that, include: The modal feature encoding and position awareness module is used to extract multi-scale texture and structural features from color images and OCT depth images of gastrointestinal endoscopy, respectively, and to fuse the extracted multi-scale texture and structural features to obtain a cross-modal feature sequence. The dynamic feature alignment module is used to align features between modalities based on feature sequences with added positional encoding, combined with a cross-attention layer, by sharing a query key-value matrix, and to learn the spatial correspondence between different modalities in vascular morphology and gastrointestinal wall layer structure. The feature extraction module is used to extract multi-scale global context feature sequences and local detail feature sequences based on aligned cross-modal vascular and gastrointestinal wall interlayer structural features, and fuse multi-scale and local detail features to obtain fused multi-modal features; The dynamic deformation field generation and calibration module is used to convert the fused multimodal features into a displacement field, then upsample the displacement field to the original image resolution to generate a dense deformation field, introduce physiological signals, and adjust the dense deformation field based on the physiological signals to obtain the adjusted dense deformation field. The multi-stage registration module is used for coarse and fine registration based on the adjusted dense deformation field to obtain the registered color images and OCT depth images of the gastrointestinal endoscope.

2. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, In the modal feature encoding and position awareness module, the extraction of cross-modal vascular features based on the acquired gastrointestinal endoscopy color image and gastrointestinal endoscopy OCT depth image includes inputting the gastrointestinal endoscopy color image and gastrointestinal endoscopy OCT depth image into the corresponding CNN encoder respectively, extracting multi-scale texture features and structural features, and fusing the extracted multi-scale texture features and structural features to obtain a cross-modal feature sequence.

3. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, In the modal feature encoding and location awareness module, based on the added encoded feature sequences, the spatial correspondence of different modal data in terms of vascular morphology and gastrointestinal wall structure is learned. This includes combining a cross-attention layer, using gastrointestinal endoscopy image tokens as queries and gastrointestinal endoscopy OCT image tokens as keys, calculating attention weights, and combining the attention weights to learn the spatial correspondence of different modal data in terms of vascular morphology and gastrointestinal wall structure.

4. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, In the feature extraction module, a multi-head global attention mechanism is used to capture global context information at different scales for each head, resulting in a multi-scale global context feature sequence. This multi-scale global context feature sequence is then combined with interlayer structure tokens from gastrointestinal endoscopy OCT depth images and a local variability attention mechanism to extract local detail feature sequences.

5. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 4, characterized in that, In the feature extraction module, each head captures global context information at different scales, specifically: heads 1 and 2 capture the main trunks of large blood vessels, heads 3 and 4 focus on the edges of gastrointestinal lesions, and heads 5 and 6 capture microvascular details.

6. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, In the feature extraction module, when fusing global contextual information and local detail features to obtain fused multimodal features, a pyramid feature fusion strategy is adopted. The bottom layer superimposes global vascular distribution features with local microvascular texture features, the middle layer fuses global gastrointestinal specific reference region relative position features with local gastrointestinal lesion edge gradient features, and the top layer combines global spectral features with local vascular bifurcation detail features.

7. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, In the dynamic deformation field generation and calibration module, the dynamic adjustment of dense deformation field parameters is expressed as follows: ， in, For the adjusted dense deformation field parameters, The parameters of the dense deformation field before adjustment. The weighting coefficients are influenced by physiological signals. S is the activation value of the gated recurrent unit after encoding the physiological signal, and S is the output state vector.

8. The multimodal medical image analysis system for gastrointestinal disease identification as described in claim 1, characterized in that, The system also includes a gastrointestinal lesion localization module, which inputs the registered gastrointestinal endoscopy color image into the backbone network of the YOLOv5 model, extracts the two-dimensional texture features of the gastrointestinal lesion, performs depth slice feature extraction on the registered gastrointestinal endoscopy OCT depth image, obtains the three-dimensional structural features of the gastrointestinal lesion region, and inputs the two-dimensional texture features and the three-dimensional structural features of the gastrointestinal lesion region into the YOLOv5 detection head, outputs the candidate box and confidence score of the gastrointestinal lesion, and determines the center coordinates and range of the gastrointestinal lesion based on the candidate box.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it performs the following steps: Multi-scale texture and structural features were extracted from color images and OCT depth images of gastrointestinal endoscopy, and the extracted multi-scale texture and structural features were fused to obtain a cross-modal feature sequence. Based on the feature sequences with added position encoding, combined with the cross-attention layer, the spatial correspondence between different modal data in terms of vascular morphology and gastrointestinal wall layer structure is learned by aligning intermodal features through a shared query key-value matrix; Based on the aligned cross-modal vascular and gastrointestinal wall interlayer structural features, multi-scale global context feature sequences and local detail feature sequences are extracted, and multi-scale and local detail features are fused to obtain fused multi-modal features; The fused multimodal features are converted into a displacement field, and then the displacement field is upsampled to the original image resolution to generate a dense deformation field. Physiological signals are introduced, and the dense deformation field is adjusted based on the physiological signals to obtain the adjusted dense deformation field. Coarse and fine registration were performed based on the adjusted dense deformation field to obtain the registered color images and OCT depth images of the gastrointestinal endoscope.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it performs the following steps: Multi-scale texture and structural features were extracted from color images and OCT depth images of gastrointestinal endoscopy, and the extracted multi-scale texture and structural features were fused to obtain a cross-modal feature sequence. Based on the feature sequences with added position encoding, combined with the cross-attention layer, the spatial correspondence between different modal data in terms of vascular morphology and gastrointestinal wall layer structure is learned by aligning intermodal features through a shared query key-value matrix; Based on the aligned cross-modal vascular and gastrointestinal wall interlayer structural features, multi-scale global context feature sequences and local detail feature sequences are extracted, and multi-scale and local detail features are fused to obtain fused multi-modal features; The fused multimodal features are converted into a displacement field, and then the displacement field is upsampled to the original image resolution to generate a dense deformation field. Physiological signals are introduced, and the dense deformation field is adjusted based on the physiological signals to obtain the adjusted dense deformation field. Coarse and fine registration were performed based on the adjusted dense deformation field to obtain the registered color images and OCT depth images of the gastrointestinal endoscope.

Images