Multi-modal multi-view endoscope image segmentation method, device, equipment and medium

Abstract

This application discloses a method, apparatus, device, and medium for multimodal and multi-view endoscopic image segmentation, relating to the field of artificial intelligence technology. The method includes: acquiring narrowband light images and visible light images; extracting modal features and viewpoint features respectively; fusing modal features to obtain multimodal fusion features; fusing viewpoint features to obtain multi-view fusion features; completing, registering, and fusing the multimodal fusion features and multi-view fusion features to obtain joint features; and segmenting the corresponding regions according to the modal features, viewpoint features, and joint features respectively. This application solves the problem of spatial misregistration between the modalities of narrowband light images and visible light images through spatial registration and feature fusion, improving the segmentation accuracy of multimodal images; the adaptive feature learning mechanism enhances adaptability to multi-view scenes, effectively addressing the impact of viewpoint changes on the segmentation results; and the combination of a deep learning multimodal and multi-view segmentation network effectively improves the detection accuracy of lesion regions.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to multimodal and multi-view endoscopic image segmentation methods, devices, equipment and media. Background Technology

[0002] Endoscopic image segmentation aims to accurately label lesion areas in an automated manner. However, due to the different modalities (such as narrowband binarization (NBI) and visible light lithography (WLI)) and perspectives involved in endoscopic lens acquisition, segmentation tasks face numerous challenges. Currently, narrowband binarization (NBI) and visible light lithography (WLI) modalities are widely used in medical endoscopy. NBI images enhance tissue details, especially vascular structures, while WLI images provide a more comprehensive field of view and background information. However, due to differences in acquisition equipment and variations in lens perspective, these two modalities often suffer from spatial misregistration, preventing traditional segmentation methods from effectively utilizing the complementary information of the two modalities and thus affecting segmentation accuracy. Furthermore, endoscopic images may undergo complex geometric deformations during acquisition due to changes in lens angle, lighting conditions, and tissue surface morphology, making it difficult for existing segmentation methods to address these issues. Summary of the Invention

[0003] The main objective of this application is to propose a multimodal, multi-view endoscopic image segmentation method, apparatus, device, and medium to improve the segmentation accuracy of endoscopic images.

[0004] To achieve the above objectives, one aspect of this application proposes a multimodal, multi-view endoscopic image segmentation method, the method comprising the following steps:

[0005] Acquire narrowband and visible light images of the lesion area taken by an endoscope from different perspectives;

[0006] Modal features are extracted from the narrowband light image and the visible light image respectively, and then the modal features are fused to obtain multimodal fusion features;

[0007] Spatial registration is performed on the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively, and then the viewpoint features are fused to obtain multi-viewpoint fusion features;

[0008] The multimodal fusion features and the multi-view fusion features are completed, registered, and fused to obtain joint features;

[0009] The deep learning-based segmentation network obtains a first segmented region based on the modal features and the joint features; the deep learning-based segmentation network obtains a second segmented region based on the viewpoint features and the joint features.

[0010] In some embodiments, the extraction of modal features from the narrowband light image and the visible light image respectively includes the following steps:

[0011] A feature extraction network is used to extract the texture of the lesion region in the narrowband light image and the visible light image as local detail features;

[0012] The feature extraction network is used to extract the shape of the lesion region in the narrowband light image and the visible light image as global context features;

[0013] The feature extraction network is used to extract the edges of the lesion region in the narrowband light image and the visible light image as multi-scale features;

[0014] The local detail features, the global context features, and the multi-scale features are used as the modal features.

[0015] In some embodiments, fusing the modal features to obtain multimodal fusion features includes the following steps:

[0016] The attention weights of the local detail features, the global context features, and the multi-scale features are calculated using a cross-attention mechanism.

[0017] The local detail features, global context features, and multi-scale features are weighted and fused according to the corresponding attention weights to obtain the multimodal fusion features; wherein, the multimodal fusion features include vascular detail information of the narrowband light image and overall structural information of the visible light image.

[0018] In some embodiments, the step of spatially registering the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively includes the following steps:

[0019] The spatial transformation relationship between the narrowband light image and the visible light image is calculated using a deep learning-based deformation field model or geometric transformation model.

[0020] Based on the spatial transformation relationship, the narrowband light image and the visible light image are mapped to a unified spatial coordinate system, and then the spatially transformed registered image features are output as the viewpoint features.

[0021] In some embodiments, fusing the viewpoint features to obtain multi-viewpoint fused features includes the following steps:

[0022] The viewpoint features are jointly modeled by three-dimensional position encoding or viewpoint self-attention mechanism to eliminate viewpoint differences, and then the multi-viewpoint fused features are output.

[0023] In some embodiments, the step of completing, registering, and fusing the multimodal fusion features and the multiview fusion features to obtain joint features includes the following steps:

[0024] For missing regions in the multimodal fusion features or the multiview fusion features, dynamic completion is performed using the complete information of the multiview fusion features or the multimodal fusion features through saliency analysis and nonlocal attention mechanisms;

[0025] A deep learning-based deformation field model is used to spatially register the multimodal fusion features and the multi-view fusion features;

[0026] The multimodal fusion features that have undergone dynamic completion and spatial registration, and the multi-view fusion features, are fused together using feature weighting and cross-attention mechanisms to obtain the joint features.

[0027] In some embodiments, the method further includes the following steps:

[0028] The segmentation loss function is calculated based on the pixel-level annotation data of the lesion region, the first segmentation region, and the second segmentation region; wherein the segmentation loss function includes cross-entropy loss, Dice loss, and boundary smoothing loss;

[0029] The parameters of the multimodal, multi-view endoscopic image segmentation model are optimized using the AdamW optimizer based on the segmentation loss function and the prediction loss; wherein, the multimodal, multi-view endoscopic image segmentation model is used to segment the narrowband light image and the visible light image to obtain the first segmentation region and the second segmentation region;

[0030] The optimized parameters are updated to the multimodal, multi-view endoscopic image segmentation model using the backpropagation algorithm.

[0031] Set a dynamically decaying learning rate to accelerate the convergence of the multimodal, multi-view endoscopic image segmentation model;

[0032] The performance metrics of the validation set are used to determine whether the multimodal, multi-view endoscopic image segmentation model meets the convergence condition; wherein, the convergence condition includes the rate of change of the segmentation loss function being less than a first preset threshold, or the segmentation accuracy of the first segmentation region and the second segmentation region reaching a second preset threshold.

[0033] If the multimodal multi-view endoscopic image segmentation model satisfies the convergence condition, then the trained multimodal multi-view endoscopic image segmentation model is saved.

[0034] If the multimodal, multi-view endoscopic image segmentation model does not meet the convergence condition, then return to the step of optimizing the parameters of the multimodal, multi-view endoscopic image segmentation model using the AdamW optimizer based on the segmentation loss function and prediction loss, until the convergence condition is met.

[0035] To achieve the above objectives, another aspect of this application provides a multimodal, multi-view endoscopic image segmentation apparatus, the apparatus comprising:

[0036] The image acquisition unit is used to acquire narrowband light images and visible light images of the lesion area taken by the endoscope from different perspectives;

[0037] The modal feature processing unit is used to extract modal features from the narrowband light image and the visible light image respectively, and then fuse the modal features to obtain multimodal fusion features;

[0038] A viewpoint feature processing unit is used to spatially register the narrowband light image and the visible light image, and then extract the viewpoint features of the narrowband light image and the visible light image respectively, and then fuse the viewpoint features to obtain multi-viewpoint fusion features;

[0039] The feature union unit is used to complete, register, and fuse the multimodal fusion features and the multiview fusion features to obtain the joint features;

[0040] An image segmentation unit is used to segment a first segmented region based on the modality features and the joint features by a deep learning-based segmentation network; and to segment a second segmented region based on the viewpoint features and the joint features by the deep learning-based segmentation network.

[0041] To achieve the above objectives, another aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal multi-view endoscopic image segmentation method.

[0042] To achieve the above objectives, another aspect of the embodiments of this application proposes a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal, multi-view endoscopic image segmentation method.

[0043] The embodiments of this application include at least the following beneficial effects:

[0044] This application can acquire narrowband and visible light images of lesion areas captured by an endoscope from different perspectives; extract modal features from the narrowband and visible light images respectively, and then fuse the modal features to obtain multimodal fusion features; perform spatial registration on the narrowband and visible light images, and then extract the viewpoint features from the narrowband and visible light images respectively, and then fuse the viewpoint features to obtain multi-view fusion features; complete, register, and fuse the multimodal fusion features and multi-view fusion features to obtain joint features; a deep learning-based segmentation network segments the first segmented region based on the modal features and joint features; a deep learning-based segmentation network segments the second segmented region based on the viewpoint features and joint features. This application solves the problem of spatial misregistration between the modalities of narrowband and visible light images through spatial registration and feature fusion, improving the segmentation accuracy of multimodal images; the adaptive feature learning mechanism of this application enhances adaptability to multi-view scenes, effectively addressing the impact of viewpoint changes on the segmentation results; combined with a deep learning-based multimodal multi-view segmentation network, it effectively improves the detection accuracy of lesion areas in endoscopic images. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating the multimodal, multi-view endoscopic image segmentation method provided in this application embodiment;

[0047] Figure 2 An example flowchart of a multimodal, multi-view endoscopic image segmentation method provided in this application embodiment;

[0048] Figure 3 An example scene diagram of dual-modal dual-view endoscopic image segmentation provided in this application embodiment;

[0049] Figure 4 This is a schematic diagram of the structure of the multimodal, multi-view endoscopic image segmentation device provided in the embodiments of this application;

[0050] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit it. In the following description, when referring to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this application; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this application as detailed in the appended claims.

[0052] It is understood that the terms “first,” “second,” etc., used in this application may be used herein to describe various concepts, but unless otherwise stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the words “if,” “when,” or “in response to a determination” as used herein may be interpreted as “when…” or “when…” or “in response to a determination.”

[0053] As used in this application, the terms "at least one", "multiple", "each", "any", etc., "at least one" includes one, two or more, "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the multiples.

[0054] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0055] To address the inaccuracy of endoscopic image segmentation, existing research has attempted to improve the segmentation performance of multimodal endoscopic images through image registration and feature fusion techniques. For example, some methods achieve modal registration through geometric transformations and jointly model the features of multimodal images using simple fusion strategies (such as feature stitching or weighted summation). However, these methods often neglect the deep complementary information between modalities, failing to fully leverage the synergistic effect of the two modalities in the segmentation task. Furthermore, the feature distributions of multi-view endoscopic images vary significantly, and existing methods struggle to effectively handle the problems caused by viewpoint changes, resulting in poor generalization ability of the model in cross-view scenarios. To address the shortcomings of existing technologies, some embodiments of this application propose a deep learning-based multimodal multi-view endoscopic image segmentation method. This method combines spatial registration, feature fusion, and adaptive feature learning techniques to address the spatial misregistration between NBI and WLI images and the inconsistent feature distributions of multi-view images. Specifically, some embodiments of this application use a cross-attention mechanism to perform weighted fusion of multimodal features, fully mining the complementary information of the two modalities, thereby enhancing the detection capability of lesion areas. Simultaneously, spatial registration of multi-view images is achieved using deep learning deformation field models or geometric transformation models, resolving the feature inconsistency problem caused by viewpoint changes. Furthermore, an adaptive feature learning mechanism effectively handles differences in feature distribution by dynamically adjusting network parameters, enhancing the generalization ability of the segmentation model. This application comprehensively solves the core challenges in multimodal and multi-view endoscopic image segmentation through multiple technologies, including multimodal fusion, multi-view spatial registration, and adaptive feature learning, providing efficient and reliable technical support for lesion detection and demonstrating broad application prospects.

[0056] This application provides a method, apparatus, device, and medium for multimodal and multi-view endoscopic image segmentation. The technical solution includes: acquiring narrowband and visible light images of a lesion region captured by an endoscope from different viewpoints; extracting modal features from the narrowband and visible light images respectively, and then fusing the modal features to obtain multimodal fusion features; spatially registering the narrowband and visible light images, and then extracting the viewpoint features from the narrowband and visible light images respectively, and then fusing the viewpoint features to obtain multi-view fusion features; completing, registering, and fusing the multimodal and multi-view fusion features to obtain joint features; segmenting a first segmented region using a deep learning-based segmentation network based on the modal features and joint features; and segmenting a second segmented region using a deep learning-based segmentation network based on the viewpoint features and joint features. This application solves the problem of spatial misregistration between narrowband light images and visible light images by using spatial registration and feature fusion, thereby improving the segmentation accuracy of multimodal images. The adaptive feature learning mechanism of this application enhances the adaptability to multi-view scenes and effectively addresses the impact of viewpoint changes on the segmentation results. Combined with a deep learning multimodal and multi-view segmentation network, it effectively improves the detection accuracy of lesion regions in endoscopic images.

[0057] This application provides a method, apparatus, device, and medium for multimodal, multi-view endoscopic image segmentation, relating to the field of artificial intelligence technology. The multimodal, multi-view endoscopic image segmentation method, apparatus, device, and medium provided in this application can be applied to terminals, servers, or software running on terminals or servers. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, or vehicle terminal, but is not limited to these; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application implementing knowledge extraction methods, but is not limited to the above forms.

[0058] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0059] Reference Figure 1 This application provides a multimodal, multi-view endoscopic image segmentation method, which may include, but is not limited to, steps S100 to S140, as follows:

[0060] S100: Acquire narrowband and visible light images of the lesion area taken by the endoscope from different perspectives.

[0061] S110: Extract the modal features of the narrowband light image and the visible light image respectively, and then fuse the modal features to obtain multimodal fusion features.

[0062] Further, the extraction of modal features of the narrowband light image and the visible light image in S110 may include the following steps S111 to S113:

[0063] S111: A feature extraction network is used to extract the texture of the lesion region in the narrowband light image and the visible light image as local detail features;

[0064] S112: The feature extraction network is used to extract the shape of the lesion region in the narrowband light image and the visible light image as global context features;

[0065] S113: The feature extraction network is used to extract the edges of the lesion region in the narrowband light image and the visible light image as multi-scale features;

[0066] The local detail features, the global context features, and the multi-scale features are used as the modal features.

[0067] More specifically, the fusion of the modal features in S110 to obtain multimodal fusion features includes the following steps S114-S115:

[0068] S114: Calculate the attention weights of the local detail features, the global context features, and the multi-scale features using a cross-attention mechanism;

[0069] S115: The local detail features, the global context features, and the multi-scale features are weighted and fused according to the corresponding attention weights to obtain the multimodal fusion features; wherein, the multimodal fusion features include the vascular detail information of the narrowband light image and the overall structural information of the visible light image.

[0070] S120: Spatial registration is performed on the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively, and then the viewpoint features are fused to obtain multi-viewpoint fusion features.

[0071] Further, in S120, spatial registration of the narrowband light image and the visible light image, followed by extraction of the viewing angle features of the narrowband light image and the visible light image respectively, includes the following steps S121 to S122:

[0072] S121: Calculate the spatial transformation relationship between the narrowband light image and the visible light image using a deep learning-based deformation field model or geometric transformation model;

[0073] S122: Based on the spatial transformation relationship, the narrowband light image and the visible light image are mapped to a unified spatial coordinate system, and then the spatially transformed registered image features are output as the viewpoint features.

[0074] Further, the fusion of the viewpoint features in S120 to obtain multi-view fused features includes the following step S123:

[0075] S123: Jointly model the viewpoint features by using 3D position encoding or viewpoint self-attention mechanism to eliminate viewpoint differences, and then output the multi-viewpoint fusion features.

[0076] S130: Complete, register and fuse the multimodal fusion features and the multi-view fusion features to obtain joint features.

[0077] Furthermore, S130 may include the following steps S131 to S133:

[0078] S131: For missing regions in the multimodal fusion features or the multi-view fusion features, dynamic completion is performed using the complete information of the multi-view fusion features or the multimodal fusion features through saliency analysis and nonlocal attention mechanism;

[0079] S132: Spatial registration of the multimodal fusion features and the multi-view fusion features is performed using a deep learning deformation field model;

[0080] S133: The multimodal fusion features that have undergone dynamic completion and spatial registration and the multi-view fusion are fused through feature weighting and cross-attention mechanisms to obtain the joint features.

[0081] S140: The deep learning-based segmentation network segments the region into a first segmented region based on the modal features and the joint features; the deep learning-based segmentation network segments the region into a second segmented region based on the viewpoint features and the joint features.

[0082] In some optional implementations, this application can utilize a multimodal, multi-view endoscopic image segmentation model to perform steps S110–S140, that is, inputting narrowband light images and visible light images into the multimodal, multi-view endoscopic image segmentation model, and using the multimodal, multi-view endoscopic image segmentation model to output a first segmented region and a second segmented region. To improve segmentation accuracy, embodiments of this application may further include a step of training the multimodal, multi-view endoscopic image segmentation model, specifically including the following steps S151–S157:

[0083] S151: Calculate a segmentation loss function based on the pixel-level annotation data of the lesion region, the first segmentation region, and the second segmentation region; wherein, the segmentation loss function includes cross-entropy loss, Dice loss, and boundary smoothing loss;

[0084] S152: The parameters of the multimodal multi-view endoscopic image segmentation model are optimized using the AdamW optimizer based on the segmentation loss function and the prediction loss; wherein, the multimodal multi-view endoscopic image segmentation model is used to segment the narrowband light image and the visible light image to obtain the first segmentation region and the second segmentation region;

[0085] S153: Update the optimized parameters to the multimodal, multi-view endoscopic image segmentation model using the backpropagation algorithm;

[0086] S154: Set a dynamically decaying learning rate to accelerate the convergence of the multimodal, multi-view endoscopic image segmentation model;

[0087] S155: Determine whether the multimodal, multi-view endoscopic image segmentation model meets the convergence condition by using the performance metrics of the validation set; wherein, the convergence condition includes the rate of change of the segmentation loss function being less than a first preset threshold, or the segmentation accuracy of the first segmentation region and the second segmentation region reaching a second preset threshold.

[0088] S156: If the multimodal multi-view endoscopic image segmentation model satisfies the convergence condition, then save the trained multimodal multi-view endoscopic image segmentation model.

[0089] S157: If the multimodal multi-view endoscopic image segmentation model does not meet the convergence condition, then return to the step of optimizing the parameters of the multimodal multi-view endoscopic image segmentation model using the AdamW optimizer based on the segmentation loss function and prediction loss, until the convergence condition is met.

[0090] The following section will provide a detailed introduction and explanation of the solutions in the embodiments of this application, using specific application examples.

[0091] This embodiment relates to the fields of artificial intelligence and medical image processing, specifically to a deep learning-based multimodal and multi-view segmentation method for endoscopic images. This method is primarily applied to the automatic segmentation of endoscopic images, particularly for spatial registration, feature fusion, and automatic detection and localization of lesion regions in narrow-band (NBI) and visible-light (WLI) modal images.

[0092] This embodiment aims to address the problems of spatial misregistration and significant differences in multi-view feature distribution between narrowband (NBI) and visible light (WLI) endoscopic images, as well as the insufficient robustness of existing segmentation methods in complex scenes. It proposes a deep learning-based multimodal, multi-view endoscopic image segmentation method. This method combines spatial registration, feature fusion, and adaptive feature learning techniques to achieve high-precision segmentation of multimodal, multi-view endoscopic images, solving the core challenge of existing methods in lesion region detection.

[0093] Reference Figure 2 , Figure 2 This is an example flowchart of a multimodal, multi-view endoscopic image segmentation method provided in this embodiment. Figure 3 This is an example scene diagram of dual-modal dual-view endoscopic image segmentation in this embodiment.

[0094] It is understood that the multimodal and multi-view endoscopic image segmentation model trained in this embodiment simultaneously inputs images from two modalities and perspectives, and outputs lesion segmentation maps corresponding to the two images after inference by the deep network model.

[0095] For example, the technical solution of this embodiment includes the following steps:

[0096] Step 1: Input narrowband light (NBI) and visible light (WLI) dual-modal multi-view images, and preprocess the input images, including image size normalization, noise removal, contrast enhancement and color space conversion, to ensure image quality and consistency.

[0097] Specifically, the input images are dual-modal, dual-view images captured by an endoscopic (or endoscope) device at the lesion site within the human body. This is because two modal light sources (visible white light WLI and narrowband light NBI) are used during endoscopic procedures. These two light source environments cannot coexist because simultaneous imaging with both light sources would cause the light environments to merge due to the narrowband light being part of the visible spectrum. The spatial positions of the two images are also mismatched, partly due to changes in the handheld endoscope position and partly due to changes in the lesion location caused by the movement of human muscle tissue.

[0098] Furthermore, by combining a convolutional neural network (CNN) with a geometric transformation module, spatial registration between NBI and WLI images is achieved, eliminating spatial inconsistencies caused by changes in the viewing angle of the imaging device.

[0099] Step 2: Use a feature extraction network (such as a convolutional neural network CNN or a Transformer network) to extract features from the NBI and WLI modalities respectively. The extracted features include local detail features, global context features and multi-scale features, which capture information such as texture, shape and edge of the lesion area respectively.

[0100] Specifically, by implementing a CNN-based feature extraction module, local detail features such as texture, color, edges, and shape from images at different viewpoints are extracted. A cross-attention mechanism is used to jointly model the channel features and spatial features between modalities, thereby improving the efficiency of the segmentation network in utilizing complementary information between modalities.

[0101] The feature extraction module in step 2 can abstract low-dimensional features into high-dimensional contextual information encoding through deep neural network inference, and obtain multi-scale features through deep multi-level feature extraction.

[0102] Step 3: Use the cross-attention mechanism to perform weighted fusion of features extracted from NBI and WLI images. In the cross-attention mechanism, through adaptive learning, the attention weights of different modal features are calculated to highlight the salient features of the lesion area. The fused features can simultaneously retain the vascular detail information of the NBI image and the overall structural information of the WLI image, thereby improving the segmentation performance.

[0103] Specifically, in step 3, the attention mechanism adaptively fuses low-dimensional detail features and high-dimensional context features extracted from different modalities through a cross-fusion mechanism, complementing the information from the two modalities and improving the segmentation performance of the model.

[0104] Step 4: Use the spatial registration module to register the NBI and WLI images to solve the spatial inconsistency problem caused by the change of viewpoint between the two modal images. During the spatial registration process, the spatial transformation relationship between the two modalities is calculated by using a deformation field model or geometric transformation model based on deep learning. The NBI and WLI images are mapped to a unified spatial coordinate system, and the registered image features after spatial transformation are output.

[0105] Specifically, in step 4, the image registration module based on CNN and fully connected network realizes the inference of spatial transformation relationship between images from different perspectives and the dimensionality reduction of spatial transformation features. By mapping the local targets of two images, images of different perspective modalities are mapped to a unified spatial coordinate system, and the spatial transformation image features are calculated by inference through registration three-dimensional position encoding.

[0106] In step 4, in response to the differences in feature distribution of endoscopic images from different perspectives, the adaptive feature learning mechanism enhances the model's ability to handle inconsistent feature distributions by dynamically adjusting network parameters.

[0107] Step 5: The multi-view feature fusion module adaptively fuses image features from different viewpoints. By introducing three-dimensional position encoding or viewpoint self-attention mechanism, it jointly models multi-view features, effectively eliminating the influence of viewpoint differences on feature representation, and outputting a fused feature representation that is robust to viewpoint changes.

[0108] Specifically, the multi-view feature fusion module described in step 5 adaptively fuses the image features obtained after transforming images from different viewpoints, introduces three-dimensional position encoding into the attention fusion mechanism, jointly models the multi-view features and the features of the image itself, minimizes the difference features caused by the difference in viewpoints, and outputs the viewpoint fusion feature representation.

[0109] Step 6: Input the fused features from multimodal and multi-viewpoint models into the adaptive learning module. Further optimize the feature representation through feature completion, registration, and deep fusion. For any missing regions in the features, dynamic completion is performed using complete information from another modality or viewpoint through saliency analysis and non-local attention mechanisms. Simultaneously, a deep learning deformation field model is used to spatially register the multimodal and multi-viewpoint features, resolving spatial inconsistency issues. Finally, deep fusion is achieved through feature weighting and cross-attention mechanisms, generating a joint feature representation that possesses both spatial consistency and rich semantic expression, providing high-quality input for the segmentation task.

[0110] Specifically, the adaptive learning module in step 6 adaptively learns the fused features based on prior knowledge of endoscopic image recognition, and dynamically adjusts the prior feature parameters through additional optimization strategies. The feature selection mechanism can automatically filter highly relevant features.

[0111] Step 7: Input the feature layer optimized by adaptive learning, and predict the lesion region for each of the two input images. The segmentation network adopts a deep learning-based segmentation architecture (such as UNet, DeepLab or Transformer-based segmentation network) to accurately segment the lesion region in the image and output the lesion segmentation result for each image.

[0112] Specifically, step 7 can predict the extent of lesions from two perspectives using the feature layers obtained in the previous step, and the segmentation network uses a CNN-based deep network to perform pixel-level segmentation of the image.

[0113] Step 7 involves designing a comprehensive loss function, including segmentation loss, bimodal image supervision loss, and feature consistency loss, to guide the model's parameter updates and ensure the segmentation accuracy of the lesion region.

[0114] Step 8: Based on the pixel-level labeled data of the lesion area and the segmentation prediction results, calculate the segmentation loss function, which includes cross-entropy loss, Dice loss and boundary smoothing loss; optimize the network parameters using the AdamW optimizer and update the network weights through the backpropagation algorithm; set a learning rate decay mechanism to dynamically adjust the learning rate to accelerate model convergence.

[0115] Specifically, step 8 uses various loss functions for endoscopic imaging lesion tasks to dynamically optimize the model parameters.

[0116] Step 9: Determine whether the model has converged by using the performance metrics of the validation set. The convergence conditions include the segmentation loss function no longer decreasing significantly or the segmentation accuracy reaching a preset threshold. If the model converges, save the trained segmentation model for subsequent use. If it has not converged, return to continue optimizing the network parameters until the model meets the convergence conditions.

[0117] In summary, this embodiment provides a deep learning-based multimodal and multi-view segmentation method for endoscopic images, aiming to address the spatial misregistration between narrowband (NBI) and visible light (WLI) modal images, as well as the significant differences in feature distribution across multiple viewpoints, thereby improving the accuracy of endoscopic image segmentation. Endoscopic image segmentation has significant application value in medical image analysis, especially in the automatic detection and localization of lesion areas. However, due to the changing viewpoints of endoscopic lenses and the spatial differences in multimodal imaging, traditional segmentation methods cannot effectively combine multimodal and multi-view information, affecting segmentation performance. To address these challenges, this embodiment proposes a segmentation framework for multimodal and multi-view endoscopic images. First, multimodal feature fusion of NBI and WLI images is achieved through a deep learning algorithm. The feature fusion module utilizes a cross-attention mechanism to fully exploit the complementary information of the two modalities, preserving the microscopic vascular details in the NBI image while incorporating the global tissue structure information of the WLI image, significantly improving the overall performance of the segmentation task. Second, this embodiment designs a multi-view spatial feature registration and fusion technique. By employing deep learning-based deformation field models or geometric transformation models, spatial alignment of image features from different perspectives is achieved, resolving the spatial inconsistency issue caused by changes in lens perspective. Furthermore, 3D positional encoding and a viewpoint self-attention mechanism are used to adaptively fuse features from different perspectives, generating a joint feature representation robust to viewpoint changes. In addition, this embodiment proposes a strategy for the completion and fusion of multimodal and multi-viewpoint features. Addressing the issue of incomplete lesion region information due to viewpoint occlusion or incomplete modal information, complete features from another perspective or modality are used to complete the missing information. Through saliency analysis and non-local attention mechanisms, dynamic completion between multimodal and multi-viewpoint features is achieved, further improving the integrity of the lesion region. Subsequently, a feature weighting mechanism and a deep fusion network are used to jointly model the completed multimodal and multi-viewpoint features, generating a high-quality segmentation feature representation. The innovation of this embodiment lies in solving the core challenges in endoscopic image segmentation through multimodal fusion, multi-viewpoint spatial feature registration and fusion, and dynamic completion and deep fusion of multimodal and multi-viewpoint features. This method demonstrates high accuracy and robustness in endoscopic image segmentation tasks, providing strong technical support for lesion detection and showing broad application prospects.

[0118] This embodiment can also provide more specific implementation methods, as follows:

[0119] This embodiment proposes a multimodal, multi-view endoscopic image segmentation method to address the problems of spatial misregistration of multimodal images, inconsistent feature distribution across multiple views, and insufficient lesion detection accuracy in existing endoscopic image segmentation methods. To this end, this embodiment designs a deep learning-based multimodal, multi-view segmentation framework that fully utilizes the complementary information of multimodal images and adaptively optimizes for the distribution characteristics of multi-view images, thereby improving segmentation accuracy and robustness. This embodiment includes the following steps:

[0120] Step 1: Input narrowband light (NBI) and visible light (WLI) dual-modal multi-view images, and preprocess the input images, including image size normalization, noise removal, contrast enhancement and color space conversion, to ensure image quality and consistency.

[0121] Step 2: Use a feature extraction network (such as a convolutional neural network CNN or a Transformer network) to extract features from the NBI and WLI modalities respectively. The extracted features include local detail features, global context features and multi-scale features, which capture information such as texture, shape and edge of the lesion area respectively.

[0122] Step 3: Use the cross-attention mechanism to perform weighted fusion of features extracted from NBI and WLI images. In the cross-attention mechanism, through adaptive learning, the attention weights of different modal features are calculated to highlight the salient features of the lesion area. The fused features can simultaneously retain the vascular detail information of the NBI image and the overall structural information of the WLI image, thereby improving the segmentation performance.

[0123] Step 4: Use the spatial registration module to register the NBI and WLI images to solve the spatial inconsistency problem caused by the change of viewpoint between the two modal images. During the spatial registration process, the spatial transformation relationship between the two modalities is calculated by using a deformation field model or geometric transformation model based on deep learning. The NBI and WLI images are mapped to a unified spatial coordinate system, and the registered image features after spatial transformation are output.

[0124] Step 5: The multi-view feature fusion module adaptively fuses image features from different viewpoints. By introducing three-dimensional position encoding or viewpoint self-attention mechanism, it jointly models multi-view features, effectively eliminating the influence of viewpoint differences on feature representation, and outputting a fused feature representation that is robust to viewpoint changes.

[0125] Step 6: Input the fused features from multimodal and multi-viewpoint models into the adaptive learning module. Further optimize the feature representation through feature completion, registration, and deep fusion. For any missing regions in the features, dynamic completion is performed using complete information from another modality or viewpoint through saliency analysis and non-local attention mechanisms. Simultaneously, a deep learning deformation field model is used to spatially register the multimodal and multi-viewpoint features, resolving spatial inconsistency issues. Finally, deep fusion is achieved through feature weighting and cross-attention mechanisms, generating a joint feature representation that possesses both spatial consistency and rich semantic expression, providing high-quality input for the segmentation task.

[0126] Step 7: Input the feature layer optimized by adaptive learning, and predict the lesion region for each of the two input images. The segmentation network adopts a deep learning-based segmentation architecture (such as UNet, DeepLab or Transformer-based segmentation network) to accurately segment the lesion region in the image and output the lesion segmentation result for each image.

[0127] Step 8: Based on the pixel-level labeled data of the lesion area and the segmentation prediction results, calculate the segmentation loss function, which includes cross-entropy loss, Dice loss and boundary smoothing loss; optimize the network parameters using the AdamW optimizer and update the network weights through the backpropagation algorithm; set a learning rate decay mechanism to dynamically adjust the learning rate to accelerate model convergence.

[0128] Step 9: Determine whether the model has converged by using the performance metrics of the validation set. The convergence conditions include the segmentation loss function no longer decreasing significantly or the segmentation accuracy reaching a preset threshold. If the model converges, save the trained segmentation model for subsequent use. If it has not converged, return to continue optimizing the network parameters until the model meets the convergence conditions.

[0129] The dual-modal, dual-view endoscopic image lesion region segmentation method designed in this embodiment can still be referred to. Figure 2 In this embodiment, the method first uses paired dual-modal, dual-view endoscopic images during the training phase. These images are taken under different lighting conditions, depicting the same lesion area. To improve the generalization performance of the neural network, this method performs the same type and parameters of image enhancement on both images during the input phase, such as rotation, cropping, flipping, and adding noise.

[0130] In step 2, the multimodal image feature extraction module first performs CNN-based feature extraction on the image. Both modules have the same network structure but do not share parameters; this is to extract image features from different imaging modalities to accommodate the differences in characteristics between NBI and WLI images. The extracted features include low-level image features such as edges and textures, as well as high-level features such as region shape and semantic information. During WLI modal feature extraction, spatial features are fused into the variables of the next layer as the convolution operation is reasoned. During NBI modal feature extraction, vascular features highly correlated with lesion features are mapped to the variables of the next layer during channel feature extraction.

[0131] Step 3, the feature fusion process, aims to combine the complementary information of the NBI and WLI modalities to generate more discriminative joint features. First, a 1×1 convolution is used to unify the channels of the features from both modalities, ensuring consistent feature dimensions. Next, a cross-attention mechanism is used to capture the correlation between the two modalities, enhancing the important features of each modality through attention weights and highlighting key regions. Subsequently, the updated modal features are concatenated along the channel dimension and further fused through convolutional layers to extract meaningful contextual information from the joint features. The final fused features fully combine the detailed information of the NBI modality (such as blood vessel and lesion contrast) and the global background information of the WLI modality (such as tissue shape and structure), providing high-quality input for subsequent segmentation tasks.

[0132] In step 4, spatial feature extraction employs multi-scale convolution operations. Small convolutional kernels capture local fine texture information (such as edges and small lesions), while large convolutional kernels extract global structural information (such as the overall shape and distribution of lesions), ensuring both detail preservation and understanding of the overall layout. To further enhance the representation of key regions, a spatial attention mechanism is introduced. Global pooling extracts global information of salient regions, generating spatial weight maps that focus attention on salient locations related to lesion areas or blood vessels, while effectively suppressing background interference. Furthermore, a spatial transformation network is used to spatially adjust the feature maps for each viewpoint. Affine transformations (such as translation, rotation, and scaling) are learned to correct feature positions, reducing the interference of viewpoint changes on feature extraction and making the features more standardized. To capture the correlation between distant spatial locations, a nonlocal module is used to model spatial context relationships. By calculating the correlation between any two spatial locations, the semantic association between the lesion area and surrounding tissues is enhanced. Finally, a dynamic weight allocation mechanism generates independent weight maps for the spatial features of each viewpoint, highlighting the recognition importance of the same region in different viewpoints. Simultaneously, by combining viewpoint feature differences, the extracted spatial features are guaranteed to have high expressive power. These processing steps together generate spatial feature maps with detailed precision and global understanding, laying a solid foundation for subsequent perspective fusion and lesion segmentation.

[0133] In step 6, during the completion, registration, and final fusion of bimodal and biview fusion features, the problem of incomplete lesion region information caused by viewpoint differences is first addressed through a completion mechanism. For lesion regions that may be incomplete in a single viewpoint due to occlusion or viewpoint changes, more complete features from another viewpoint are used for compensation. Through saliency analysis and region difference detection, the missing locations of lesion regions are identified, and nonlocal attention mechanisms or feature interpolation methods are used to map lesion features from another viewpoint to the current viewpoint, dynamically filling in feature gaps. Simultaneously, bimodal fusion features (such as microscopic details in NBI or global semantics in WLI) are used to assist in completion, further refining the representation of the lesion region. After completion, to eliminate spatial inconsistencies between bimodal and biview features, spatial registration techniques are employed. The lesion region is aligned through keypoint detection and affine transformation, and a consistency loss function is used to optimize the spatial consistency and semantic expression of the features. Based on this, the completed and registered features are further integrated through channel splicing and a fusion network. An attention weighting mechanism is used to highlight key features of the lesion region while suppressing redundant information. Ultimately, the fused feature map possesses both the detail advantage of dual-modality and the completion capability of dual-viewpoint, ensuring not only the integrity of the lesion region but also achieving spatial consistency and semantic expression unity, providing high-quality input features for subsequent segmentation tasks.

[0134] In step 8, during the training of the segmentation network, a comprehensive segmentation loss function is first calculated using pixel-level labeled data of the lesion region and the prediction results. This loss function consists of cross-entropy loss, Dice loss, and boundary smoothing loss. Cross-entropy loss focuses on the accuracy of global pixel classification, Dice loss focuses on the overlap of lesion regions, solving the segmentation problem of small target regions, and boundary smoothing loss optimizes the edge quality of the segmentation results by constraining the gradient difference between the predicted and labeled boundaries. These three losses are weighted and summed to form a comprehensive loss function, which optimizes global classification performance, region overlap, and boundary detail performance, respectively. Next, the AdamW optimizer is used to optimize the network parameters. This optimizer combines momentum update and weight decay mechanisms, which not only enables fast convergence but also effectively prevents overfitting through regularization. Through the backpropagation algorithm, the network weights are updated layer by layer according to the gradient calculated by the segmentation loss function, so that the model parameters continuously approach the optimal solution. In addition, to further improve training efficiency and stability, a learning rate decay mechanism is adopted to dynamically adjust the learning rate according to the training progress. A high learning rate is initially set to accelerate convergence, and then the learning rate is gradually reduced in the later stages through strategies such as exponential decay, cosine annealing, or validation set performance monitoring to avoid model oscillations in the later stages of training. This comprehensive training strategy ensures that the model can achieve a balance between global accuracy, local region overlap, and boundary detail optimization, while improving the accuracy and robustness of the segmentation results.

[0135] In summary, the beneficial effects of this embodiment include:

[0136] This embodiment proposes a multimodal, multi-view endoscopic image segmentation method. This method addresses the spatial misregistration problem between the NBI and WLI modalities through spatial registration and feature fusion techniques, thereby improving the segmentation accuracy of multimodal images. The proposed adaptive feature learning mechanism enhances the model's adaptability to multi-view scenes, effectively addressing the impact of viewpoint changes on the segmentation results. This embodiment combines a deep learning-based multimodal, multi-view segmentation framework to effectively improve the detection accuracy of lesion regions in endoscopic images.

[0137] Reference Figure 4 This application also provides a multimodal, multi-view endoscopic image segmentation device, which can implement the above-mentioned multimodal, multi-view endoscopic image segmentation method. The device includes:

[0138] The image acquisition unit is used to acquire narrowband light images and visible light images of the lesion area taken by the endoscope from different perspectives;

[0139] The modal feature processing unit is used to extract modal features from the narrowband light image and the visible light image respectively, and then fuse the modal features to obtain multimodal fusion features;

[0140] A viewpoint feature processing unit is used to spatially register the narrowband light image and the visible light image, and then extract the viewpoint features of the narrowband light image and the visible light image respectively, and then fuse the viewpoint features to obtain multi-viewpoint fusion features;

[0141] The feature union unit is used to complete, register, and fuse the multimodal fusion features and the multiview fusion features to obtain the joint features;

[0142] An image segmentation unit is used to segment a first segmented region based on the modality features and the joint features by a deep learning-based segmentation network; and to segment a second segmented region based on the viewpoint features and the joint features by the deep learning-based segmentation network.

[0143] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0144] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described multimodal, multi-view endoscopic image segmentation method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0145] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] Please see Figure 5 , Figure 5 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0147] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0148] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 using the multimodal multi-view endoscopic image segmentation method of the embodiments of this application.

[0149] The input / output interface 503 is used to implement information input and output;

[0150] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0151] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0152] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0153] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described multimodal, multi-view endoscopic image segmentation method.

[0154] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0155] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0156] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0157] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0158] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0160] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0161] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0162] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0163] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0164] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0165] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A multimodal, multi-view endoscopic image segmentation method, characterized in that, The method includes the following steps: Acquire narrowband and visible light images of the lesion area taken by an endoscope from different perspectives; Modal features are extracted from the narrowband light image and the visible light image respectively, and then the modal features are fused to obtain multimodal fusion features; Spatial registration is performed on the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively, and then the viewpoint features are fused to obtain multi-viewpoint fusion features; The multimodal fusion features and the multi-view fusion features are completed, registered, and fused to obtain joint features; The deep learning-based segmentation network obtains a first segmented region based on the modal features and the joint features; the deep learning-based segmentation network obtains a second segmented region based on the viewpoint features and the joint features. The step of spatially registering the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively includes the following steps: The spatial transformation relationship between the narrowband light image and the visible light image is calculated using a deep learning-based deformation field model or geometric transformation model. Based on the spatial transformation relationship, the narrowband light image and the visible light image are mapped to a unified spatial coordinate system, and then the spatially transformed registered image features are output as the viewpoint features. The process of fusing the viewpoint features to obtain multi-view fused features includes the following steps: The viewpoint features are jointly modeled by three-dimensional position encoding or viewpoint self-attention mechanism to eliminate viewpoint differences and then output the multi-viewpoint fusion features. The process of completing, registering, and fusing the multimodal fusion features and the multiview fusion features to obtain joint features includes the following steps: For missing regions in the multimodal fusion features or the multiview fusion features, dynamic completion is performed using the complete information of the multiview fusion features or the multimodal fusion features through saliency analysis and nonlocal attention mechanisms; A deep learning-based deformation field model is used to spatially register the multimodal fusion features and the multi-view fusion features; The multimodal fusion features that have undergone dynamic completion and spatial registration, and the multi-view fusion features, are fused together using feature weighting and cross-attention mechanisms to obtain the joint features.

2. The multimodal, multi-view endoscopic image segmentation method according to claim 1, characterized in that, The extraction of modal features from the narrowband light image and the visible light image, respectively, includes the following steps: A feature extraction network is used to extract the texture of the lesion region in the narrowband light image and the visible light image as local detail features; The feature extraction network is used to extract the shape of the lesion region in the narrowband light image and the visible light image as global context features; The feature extraction network is used to extract the edges of the lesion region in the narrowband light image and the visible light image as multi-scale features; The local detail features, the global context features, and the multi-scale features are used as the modal features.

3. The multimodal, multi-view endoscopic image segmentation method according to claim 2, characterized in that, The process of fusing the modal features to obtain multimodal fusion features includes the following steps: The attention weights of the local detail features, the global context features, and the multi-scale features are calculated using a cross-attention mechanism. The local detail features, global context features, and multi-scale features are weighted and fused according to the corresponding attention weights to obtain the multimodal fusion features; wherein, the multimodal fusion features include vascular detail information of the narrowband light image and overall structural information of the visible light image.

4. The multimodal, multi-view endoscopic image segmentation method according to any one of claims 1 to 3, characterized in that, The method further includes the following steps: The segmentation loss function is calculated based on the pixel-level annotation data of the lesion region, the first segmentation region, and the second segmentation region; wherein the segmentation loss function includes cross-entropy loss, Dice loss, and boundary smoothing loss; The parameters of the multimodal, multi-view endoscopic image segmentation model are optimized using the AdamW optimizer based on the segmentation loss function and the prediction loss; wherein, the multimodal, multi-view endoscopic image segmentation model is used to segment the narrowband light image and the visible light image to obtain the first segmentation region and the second segmentation region; The optimized parameters are updated to the multimodal, multi-view endoscopic image segmentation model using the backpropagation algorithm. Set a dynamically decaying learning rate to accelerate the convergence of the multimodal, multi-view endoscopic image segmentation model; The performance metrics of the validation set are used to determine whether the multimodal, multi-view endoscopic image segmentation model meets the convergence condition; wherein, the convergence condition includes the rate of change of the segmentation loss function being less than a first preset threshold, or the segmentation accuracy of the first segmentation region and the second segmentation region reaching a second preset threshold. If the multimodal multi-view endoscopic image segmentation model satisfies the convergence condition, then the trained multimodal multi-view endoscopic image segmentation model is saved. If the multimodal, multi-view endoscopic image segmentation model does not meet the convergence condition, then return to the step of optimizing the parameters of the multimodal, multi-view endoscopic image segmentation model using the AdamW optimizer based on the segmentation loss function and prediction loss, until the convergence condition is met.

5. A multimodal, multi-view endoscopic image segmentation device, characterized in that, The device includes: The image acquisition unit is used to acquire narrowband light images and visible light images of the lesion area taken by the endoscope from different perspectives; A modal feature processing unit is used to extract modal features from the narrowband light image and the visible light image respectively, and then fuse the modal features to obtain multimodal fusion features; A viewpoint feature processing unit is used to spatially register the narrowband light image and the visible light image, and then extract the viewpoint features of the narrowband light image and the visible light image respectively, and then fuse the viewpoint features to obtain multi-viewpoint fusion features; The feature union unit is used to complete, register and fuse the multimodal fusion features and the multiview fusion features to obtain the joint features; An image segmentation unit is used to segment a first segmented region based on the modal features and the joint features by a deep learning-based segmentation network; and to segment a second segmented region based on the viewpoint features and the joint features by the deep learning-based segmentation network. in: The step of spatially registering the narrowband light image and the visible light image to extract the viewpoint features of the narrowband light image and the visible light image respectively includes the following steps: The spatial transformation relationship between the narrowband light image and the visible light image is calculated using a deep learning-based deformation field model or geometric transformation model. Based on the spatial transformation relationship, the narrowband light image and the visible light image are mapped to a unified spatial coordinate system, and then the spatially transformed registered image features are output as the viewpoint features. The process of fusing the viewpoint features to obtain multi-view fused features includes the following steps: The viewpoint features are jointly modeled by three-dimensional position encoding or viewpoint self-attention mechanism to eliminate viewpoint differences and then output the multi-viewpoint fusion features. The process of completing, registering, and fusing the multimodal fusion features and the multiview fusion features to obtain joint features includes the following steps: For missing regions in the multimodal fusion features or the multiview fusion features, dynamic completion is performed using the complete information of the multiview fusion features or the multimodal fusion features through saliency analysis and nonlocal attention mechanisms; A deep learning-based deformation field model is used to spatially register the multimodal fusion features and the multi-view fusion features; The multimodal fusion features that have undergone dynamic completion and spatial registration, and the multi-view fusion features, are fused together using feature weighting and cross-attention mechanisms to obtain the joint features.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the multimodal, multi-view endoscopic image segmentation method as described in any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal, multi-view endoscopic image segmentation method as described in any one of claims 1 to 4.

Images