Cross-modal endoscopic polyp collaborative detection method based on comparative learning

Abstract

The invention relates to a cross-modal endoscopic polyp collaborative detection method based on comparative learning, and belongs to the technical field of medical image analysis, and the method comprises the following steps: S1, obtaining a capsule endoscopic image, carrying out the preprocessing, and constructing a training set; s2, constructing a semantic contrast learning cross-modal detection network, and training the semantic contrast learning cross-modal detection network by using a training set; and S3, acquiring a capsule endoscopy image, preprocessing the capsule endoscopy image, inputting the preprocessed capsule endoscopy image into the trained semantic contrast learning cross-modal detection network, and performing target detection on polyps in the image. According to the invention, the workload of film reading is effectively reduced, the risk of missing detection is reduced, and a more reliable and more universal artificial intelligence auxiliary solution is provided for early screening of digestive tract diseases.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of medical image analysis, and relates to a cross-modal endoscopic polyp collaborative detection method based on contrast learning. BACKGROUND

[0002] With the breakthrough development of deep learning technology, computer-aided diagnosis systems have made significant progress in the field of colorectal polyp detection. Early research mainly focuses on single-modality endoscopic image analysis, such as white light endoscopy and narrow-band imaging, and achieves good detection performance on static image datasets by improving target detection frameworks such as Faster R-CNN and YOLO. However, with the popularization of new devices such as capsule endoscopy, traditional methods expose two major defects: first, the imaging characteristics of different endoscopic devices differ significantly, leading to a sharp decline in cross-domain detection performance; second, existing algorithms do not effectively utilize the complementary features between different imaging modalities.

[0003] The massive amount of capsule endoscopic images brings a huge burden to clinicians in reading the images, and the reading level of doctors varies, which can easily lead to missed diagnosis, limiting the clinical application of capsule endoscopy. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a cross-modal endoscopic polyp collaborative detection method based on contrast learning.

[0005] To achieve the above purpose, the present application provides the following technical scheme:

[0006] A cross-modal endoscopic polyp collaborative detection method based on contrast learning, comprising the following steps:

[0007] S1: Obtain ordinary endoscopic and capsule endoscopic images, pre-process them, and construct a training set;

[0008] S2: Construct a semantic contrast learning cross-modal detection network, and train the semantic contrast learning cross-modal detection network using the training set;

[0009] S3: Obtain ordinary endoscopic and capsule endoscopic images, pre-process them, and input them into the trained semantic contrast learning cross-modal detection network to detect polyps in the images.

[0010] Further, the pre-processing in step S1 includes gray scale transformation, dynamic noise reduction, and normalization of the endoscopic images; the details of the polyp blood vessel texture are preserved during gray scale transformation; the polyp edge sharpness is preserved during dynamic noise reduction; and the images are scaled to a uniform size after normalization.

[0011] Further, the semantic contrast learning cross-modal detection network structure comprises: a superpixel segmentation and mask generation module, a dual-flow improved MAE encoder, a cross-modal cross-attention module, and a multi-scale feature decoding and reconstruction module.

[0012] The pixel segmentation and mask generation module segments the input image by superpixel segmentation to obtain a segmented image M seg The obtained mask image M seg A probability heat map is generated, 10% mask rate is set in the polyp region, and 60% mask rate is set in the background region, as follows:

[0013] M dynamic =α·M seg +(1-α)·M random ,α∈[0.7,0.9]

[0014] Wherein, M seg is a binary mask output by the segmentation model, M random represents a randomly generated mask for introducing randomness, and alpha is a dynamic balance coefficient.

[0015] The dual-flow improved MAE encoder is used to extract features.

[0016] The cross-modal cross-attention module is used to realize fine-grained alignment at the feature level.

[0017]

[0018] Wherein, Q represents the target domain feature, K and V represent the source domain feature, d k represents the dimension of K, and the specific calculation is as follows:

[0019] Q=W q z tgt

[0020] K=W k z src

[0021] V=W v z src

[0022] Wherein, W q , W k and W v are weight matrices of Q, K and V respectively, z src and z tgt represent the feature vectors extracted from the source domain and the target domain.

[0023] The multi-scale feature decoding and reconstruction module is used to perform pixel-level classification on the aligned features to finally realize polyp target instance segmentation output.

[0024] Further, in training the semantic contrast learning cross-modal detection network, the images of the source domain and the target domain are respectively input into the superpixel segmentation and mask generation module to obtain a lesion-aware mask M dynamic The obtained mask M dynamic is input into a double-flow improved MAE encoder to extract features, the features extracted by the encoder are input into a cross-modal cross-attention module for feature alignment, then the aligned features are pixel-level classified, and finally the polyp target instance segmentation is output; the image of the source domain is a normal endoscopic image, and the image of the target domain is a capsule endoscopic image.

[0025] Further, the training steps of the semantic contrast learning cross-modal detection network are as follows:

[0026] S21: Superpixel segmentation is performed on the input source domain and target domain images, first converting from RGB space to CIE-Lab color space, and constructing a five-dimensional feature vector V = [L, a, b, x, y] combined with pixel coordinates, wherein [L, a, b] are color components, and [x, y] are spatial coordinates;

[0027] S22: The image is divided into K regular grids with a spacing S, and the cluster center is adjusted to the position with the minimum gradient in the 3x3 neighborhood of each grid center;

[0028] S23: For each pixel, calculate the distance D with all seed points in the adjacent 2Sx2S region, and define the comprehensive distance D of pixel i and cluster center k as:

[0029]

[0030] wherein d c is the color distance; d s is the spatial distance; m is the compactness parameter, which controls the weight of color and space; is the initial step size of superpixel, N is the total number of pixels, and K is the number of superpixels; d c and d s are calculated as follows:

[0031]

[0032] wherein L i , a i and b i represent the color values of the i-th pixel; L k , a k and b k represent the color values of the cluster center k; x i and y i represent the position coordinates of the i-th pixel in the image; x k and yk denotes the position coordinates of the cluster center k in the image;

[0033] S24: Calculate the pixel distance in the 2Sx2S region around each cluster center, reduce the time complexity from O(NK) to O(N), K is the number of superpixels, assign the pixels to the nearest cluster center to form the superpixel region, and finally obtain the segmented mask image M seg ;

[0034] S25: Generate a probability heat map from the obtained mask image, set a 10% mask rate in the polyp region and a 60% mask rate in the background region, as follows:

[0035] M dynamic = α·M seg +(1-α)·M random ,α∈[0.7,0.9]

[0036] where M seg is the binary mask output by the segmentation model, M random represents a randomly generated mask used to introduce randomness, and α is a dynamic balance coefficient;

[0037] S26: Calculate the loss function and use the random descent method for back propagation to optimize the parameters of the model.

[0038] Further, the loss function comprises:

[0039] 1) Lesion-aware weighted reconstruction loss

[0040]

[0041] where λ1, λ2 are weights used to enhance the reconstruction accuracy of the lesion area; is the output image of the network model; ⊙ represents Hadamard product, i.e., element multiplication; ‖‖2 represents L2 norm (Euclidean norm) used to calculate the reconstruction error of the lesion area; ‖‖1 represents L1 norm used to calculate the reconstruction error of the non-lesion area;

[0042] 2) Constraint the latent distribution loss of the decoder trained by the source domain and the target domain through KL divergence D KL

[0043]

[0044] where p source (z) is the latent distribution of the source domain data after passing through the decoder; p target (z) is the latent distribution of the target domain data after passing through the decoder; ​

[0045] 3) For different modal image pairs (x src ,x tgt ) of the same polyp, the features (z src ,z tgt ) of the shared encoder are extracted, and the contrast loss

[0046]

[0047] Wherein, sim(z src ,z tgt ) represents the similarity between the source domain feature z src and the target domain feature z tgt , and the cosine similarity is used for calculation; τ is a temperature parameter; is a negative sample feature; K represents the number of negative samples;

[0048] 4) Attribute-level contrast learning is constructed, and the atlas constraint loss

[0049]

[0050] Wherein, and respectively represent the feature vectors of the i th sample in the source domain and the target domain, represent the square of the Euclidean distance; γ is a balance coefficient, D JS is the Jensen-Shannon divergence, p src (s) and p tgt (s) respectively represent the latent distribution of the source domain sample and the target domain sample;

[0051] 5) The above loss functions are combined at multiple levels, and the total loss function is:

[0052]

[0053] Wherein, λ recon , λ cont , λ kg , λ align are the weights of the loss functions.

[0054] The beneficial effects of the present application are that: the present application realizes feature complementation through a detection framework with domain robustness, combined with an unsupervised segmentation method to construct a multi-level optimization framework, and realizes cross-modal integration of multi-source data; effectively reduces the workload of reading films, and reduces the risk of missed detection, provides a more reliable and more universal artificial intelligence auxiliary solution for early screening of digestive diseases.

[0055] Additional advantages, objects, and features of the application will be apparent to those skilled in the art upon examination of the following specification. It is intended that the application not be limited by the disclosed implementation, but that it be construed as including all such modifications, equivalents, and alternatives falling within the scope of the invention. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to make the objects, technical solutions and advantages of the present application clearer, the preferred embodiments of the present application will be described in detail below with reference to the drawings, in which:

[0057] Figure 1 Flow chart of the cross-modal endoscopic polyp collaborative detection method based on contrastive learning. DETAILED DESCRIPTION

[0058] The other advantages and effects of the present application can be easily understood by those skilled in the art from the content disclosed in the specification. The present application can also be implemented or applied by different specific embodiments, and the details in the specification can be modified or changed in different views and applications without departing from the spirit of the present application. It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the diagrams, not the number, shape and size of the components in actual implementation. The shapes, numbers and proportions of the components in actual implementation can be changed arbitrarily, and the layout pattern of the components can be more complex.

[0060] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the present application, however, it is obvious to those skilled in the art that the embodiments of the present application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams instead of details, to avoid making the embodiments of the present application difficult to understand.

[0061] Embodiment 1:

[0062] As Figure 1As shown, the present application provides a cross-modal endoscopic polyp collaborative detection method based on contrast learning, which combines an unsupervised segmentation method to construct a multi-level optimization framework and realizes feature complementation through cross-modal integration of multi-source data. The present application constructs basic representation ability through self-supervised pre-training, narrows the domain difference through adversarial learning, and then strengthens cross-modal consistency through contrast learning, finally realizes high-precision polyp detection in unlabeled capsule endoscopy images.

[0063] The present application first obtains binary masks for the source domain (ordinary endoscope) and the target domain (capsule endoscope) respectively using an unsupervised segmentation method, performs reconstruction training on the masks, constructs a shared feature extractor, improves the clinical relevance of self-supervised learning through a lesion area key retention mechanism, constructs a cross-modal positive sample pair, minimizes the cross-modal feature distance of the same polyp, constructs a polyp attribute knowledge graph (size, shape, etc.) to constrain the semantic space, and realizes cross-modal alignment from the pixel level to the semantic level. When the detection model is trained, the collected images are input into the trained detection model in the actual application process, and finally high-precision automatic detection of polyps in different endoscopes is realized.

[0064] The method specifically includes the following steps:

[0065] Step one: Obtain ordinary endoscope and capsule endoscope images, and perform image preprocessing such as gray scale transformation, dynamic noise reduction, and normalization; specifically including: performing gray scale transformation on the images to retain the detail information of polyp blood vessel texture; performing noise reduction processing on the gray scale transformed images to retain the edge sharpness of polyps; performing normalization processing on the filtered images, and scaling to an image size of 600*600;

[0066] Step two: parameter training of the designed network model, optimization using an adaptive learning rate optimization algorithm; during offline training, the preprocessed images are input into the detection network to be trained, first for the data of the source domain (ordinary endoscope) and the target domain (capsule endoscope), the images of the source domain and the target domain are respectively input into the dual-flow improved MAE architecture network for pre-training, and the specific training steps are as follows:

[0067] Perform superpixel segmentation on the input images, first convert from RGB space to CIE-Lab color space, and construct a five-dimensional feature vector V=[L,a,b,x,y] combining pixel coordinates, where [L,a,b] are color components, and [x,y] are spatial coordinates.

[0068] Divide the image into K regular grids with a spacing of S, and adjust the seed point to the position with the smallest gradient in the 3*3 neighborhood at the center of each grid to avoid noise or edge interference.

[0069] For each pixel, calculate its distance D with all seed points in the adjacent 2Sx2S region, define the integrated distance D of pixel i and cluster center k as:

[0070]

[0071] where d c is the color distance; d s is the spatial distance; m is the compactness parameter, which controls the weight of color and space; (N is the total number of pixels, K is the number of superpixels) is the initial step of superpixel; the specific calculation is as follows:

[0072]

[0073] where L i , a i and b i represent the color value of the i-th pixel; L k , a k and b k represent the color value of the cluster center k; x i and y i represent the position coordinates of the i-th pixel in the image; x k and y k represent the position coordinates of the cluster center k in the image.

[0074] Calculate the distance of each pixel in the 2Sx2S region around the cluster center, reduce the time complexity from O(NK) to O(N) (K is the number of superpixels), assign the pixels to the nearest cluster center, form the superpixel region, and finally get the segmented mask image M seg .

[0075] Generate the probability heat map from the obtained mask image, set 10% mask rate in the polyp region and 60% mask rate in the background region, as follows:

[0076] M dynamic = a M seg + (1-a) M random , a e [0.7, 0.9]

[0077] where M seg is the binary mask output by the segmentation model, M random represents a randomly generated mask, which is used to introduce randomness, and a is a dynamic balance coefficient.

[0078] For the pre-trained network model, introduce the lesion perception weighted reconstruction loss for training:

[0079]

[0080] Wherein, λ1=1.0, λ2=0.3, to enhance the reconstruction accuracy of the lesion area; Output image for network model; ⊙ represents Hadamard product, i.e., element-wise product; ‖‖2 represents L2 norm (Euclidean norm), used to calculate reconstruction error in lesion areas; ‖‖1 represents L1 norm, used to calculate reconstruction error in non-lesion areas.

[0081] Through KL divergence D KL The latent distribution of the decoder trained in the source and target domains is constrained by the following constraint rules:

[0082]

[0083] Where, p source (z) is the latent distribution of the source domain data after the decoder; p target (z) is the potential distribution of the target domain data after the decoder.

[0084] Different modal image pairs (x) of the same polyp src ,x tgt ), extract the features (z) of the shared encoder src ,z tgt The contrast loss design is as follows:

[0085]

[0086] Where, sim(z) src ,z tgt ) represents the source domain feature z src and target domain features z tgt The similarity between them is calculated using cosine similarity; τ = 0.07 is the temperature parameter; Features of negative samples (different polyps or background); K represents the number of negative samples.

[0087] Introducing a cross-modal attention module to achieve fine-grained alignment at the feature level:

[0088]

[0089] Where Q represents the target domain feature, K and V represent the source domain features, and d k This represents the dimension of K. The specific calculation is as follows:

[0090] Q = W q z tgt

[0091] K = W k z src

[0092] V = W v z src

[0093] wherein W q , W k and W v are the weight matrices of Q, K and V respectively, z src and z tgt denote the feature vectors extracted from the source domain and the target domain.

[0094] The attribute-level contrastive learning is constructed, and the atlas constraint loss is designed as follows:

[0095]

[0096] wherein, and denote the feature vectors of the i-th sample in the source domain and the target domain respectively, denotes the square of the Euclidean distance; γ = 0.5 is a balance coefficient, D JS is the Jensen-Shannon divergence, p src (s) and p tgt (s) denote the latent distribution of the samples in the source domain and the target domain respectively.

[0097] A multi-level joint training strategy is performed, and the total loss function is designed as follows:

[0098]

[0099] wherein λ recon = 1.0, λ cont = 0.5, λ kg = 0.3, and λ align = 0.2.

[0100] The total loss calculated will be back propagated to optimize the parameters of the model using the random descent method.

[0101] Step three: input the preprocessed ordinary endoscopic images and capsule endoscopic images into the trained detection network, and complete the polyp segmentation of different endoscopic devices.

[0102] Embodiment 2:

[0103] An electronic device includes a memory and a processor;

[0104] The memory is configured to store a computer program;

[0105] The processor is configured to implement the method of embodiment 1 when executing the computer program.

[0106] Embodiment 3:

[0107] A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the method of embodiment 1.

[0108] Embodiment 4:

[0109] A computer program product comprising a computer program which, when executed by a processor, implements the method of embodiment 1.

[0110] In the above embodiments, references in the specification to "the embodiment" indicate that a particular feature, structure, or characteristic described is included in at least some embodiments, but not necessarily all embodiments. The various appearances of "the embodiment" are not necessarily all referring to the same embodiment.

[0111] In the above embodiments, although the present application has been described with reference to particular embodiments, many alternatives, modifications, and variations will be apparent to those skilled in the art from the preceding description. For example, other storage structures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed. The embodiments of the present application are intended to embrace all such alternatives, modifications, and variations as can fall within the scope of the appended claims.

[0112] The computer readable storage medium in the embodiments can be understood by those skilled in the art that all or part of the steps of the methods described above can be completed by a hardware comprised in a computer program. The computer program described above can be stored in a computer readable storage medium. The program is executed by a processor to perform the steps of the methods described above. The storage medium described above includes various types of memories, such as a ROM, a RAM, a magnetic disk, or an optical disk, and so on.

[0113] The electronic terminal provided in the embodiments includes a processor, a memory, a transceiver, and a communication interface. The memory and the communication interface are connected with the processor and the transceiver and complete communication between each other. The memory is configured to store a computer program. The communication interface is configured to communicate. The processor and the transceiver are configured to run the computer program, so that the electronic terminal performs each step of the method.

[0114] In the embodiments, the memory can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory.

[0115] The processor described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0116] The present application can be applied in numerous general or special computing system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, etc.

[0117] The present application can be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present application can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules can be located in local and remote computer storage media including memory storage devices.

[0118] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the present application, and all of them should be covered in the scope of the claims of the present application.

Claims

1. A cross-modality endoscopic polyp co-detection method based on contrastive learning, characterized in that: The method comprises the following steps: S1: obtaining common endoscopic and capsule endoscopic images, preprocessing, and constructing a training set; S2: constructing a semantic contrast learning cross-modal detection network, and training the semantic contrast learning cross-modal detection network using the training set; S3: obtaining common endoscopic and capsule endoscopic images, preprocessing, and inputting the images into the trained semantic contrast learning cross-modal detection network to detect polyps in the images.

2. The contrastive learning based cross-modality endoscopic polyp co-detection method of claim 1, wherein: The preprocessing in step S1 comprises grayscale conversion, dynamic noise reduction, and normalization of the endoscopic images; the details of polyp blood vessel texture are retained during grayscale conversion; the edge sharpness of the polyps is retained during dynamic noise reduction; and the images are scaled to a uniform size after normalization.

3. The contrastive learning based cross-modality endoscopic polyp co-detection method of claim 1, wherein: The structure of the semantic contrast learning cross-modal detection network comprises a superpixel segmentation and mask generation module, a dual-stream improved MAE encoder, a cross-modal cross-attention module, and a multi-scale feature decoding and reconstruction module; The pixel segmentation and mask generation module utilizes superpixel segmentation to segment the input image to obtain a segmented image M seg The obtained mask image M seg Generate a probability heat map, set 10% mask rate in the polyp area and 60% mask rate in the background area, as follows: M dynamic = a - M seg + (1 - a) - M random , a e [0.7, 0.9] Wherein, M seg is a binary mask output by the segmentation model, M random represents a randomly generated mask for introducing randomness, and a is a dynamic balance coefficient. The dual-stream improved MAE encoder is used to extract features; The cross-modal cross-attention module is used to realize feature-level fine-grained alignment: where Q represents the target domain feature, K and V represent the source domain feature, d k represents the dimension of K, and is specifically calculated as follows: Q = W q z tgt K = W k z src V = W v z src where W q , W k and W v are weight matrices of Q, K and V, respectively, z src and z tgt represent the feature vectors extracted from the source domain and the target domain; The multi-scale feature decoding and reconstruction module is used to perform pixel-level classification on the aligned features and finally realize polyp target instance segmentation output.

4. The contrastive learning based cross-modality endoscopic polyp co-detection method of claim 3, wherein: In training the semantic contrast learning cross-modal detection network, the images of the source domain and the target domain are respectively input into the superpixel segmentation and mask generation module to obtain a lesion-aware mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M dynamic The obtained mask map M <000 5. The contrastive learning based cross-modality endoscopic polyp co-detection method of claim 4, wherein: The training steps of the semantic contrast learning cross-modal detection network are as follows: S21: performing superpixel segmentation on the input source domain and target domain images, first converting from RGB space to CIE-Lab color space, and combining pixel coordinates to construct a five-dimensional feature vector V=[L,a,b,x,y], wherein [L,a,b] are color components, and [x,y] are spatial coordinates; S22: dividing the image into K regular grids with a spacing of S, and adjusting the cluster center to the position with the minimum gradient in the 3*3 neighborhood of each grid center; S23: for each pixel, calculating the distance D of the pixel with all seed points in the adjacent 2S*2S region, and defining the comprehensive distance D of pixel i and cluster center k as: where d c is the color distance; d s is the spatial distance; m is the compactness parameter, controlling the weight of color and space; is the initial step size of superpixels, N is the total number of pixels, and K is the number of superpixels; d c and d s are calculated as follows: wherein L i , a i and b i represent the color values of the i-th pixel; L k , a k and b k represent the color values of the cluster center k; x i and y i represent the position coordinates of the i-th pixel in the image; x k and y k represent the position coordinates of the cluster center k in the image; S24: Calculate the distance of pixels in the 2Sx2S region around each cluster center, reduce the time complexity from O(NK) to O(N), K is the number of superpixels, assign the pixels to the nearest cluster center, form the superpixel region, and finally get the segmented mask image M seg ; S25: generating a probability heat map from the obtained mask image, setting a 10% mask rate in the polyp area, and setting a 60% mask rate in the background area, as follows: M dynamic = a - M seg + (1 - a) - M random , a e [0.7, 0.9] Wherein, M seg is a binary mask output by the segmentation model, M random represents a randomly generated mask for introducing randomness, and a is a dynamic balance coefficient. S26: calculating a loss function, using a random descent method for back propagation, and optimizing the parameters of the model.

6. The contrastive learning based cross-modality endoscopic polyp co-detection method of claim 5, wherein: The loss function comprises: 1) Lesion-aware weighted reconstruction loss Wherein, λ1, λ2 are weights for strengthening the reconstruction accuracy of the lesion area; to output an image for the network model; ⊙ represents Hadamard product, that is, element multiplication; ‖‖2 represents L2 norm (Euclidean norm) for calculating the reconstruction error of the lesion area; ‖‖1 represents L1 norm for calculating the reconstruction error of the non-lesion area; 2) by KL divergence D KL constraining the latent distribution loss of the decoder trained on source and target domains where p source (z) is the latent distribution of the source domain data after the decoder; p target (z) is the latent distribution of the target domain data after the decoder; 3) for different modal image pairs (x src ,x tgt ) of the same polyp, extract the shared encoder features (z src ,z tgt ), construct the contrastive loss wherein sim(z src ,z tgt ) represents the similarity between the source domain feature z src and the target domain feature z tgt , and is calculated using cosine similarity; and τ is a temperature parameter. is a negative sample feature; and K represents the number of negative samples. 4) Construct attribute-level contrastive learning, atlas-constrained loss wherein, and denote the feature vector of the i-th sample in the source and target domains, respectively, denotes the squared Euclidean distance; γ is a balancing coefficient, D JS is the Jensen-Shannon divergence, p src (s) and p tgt (s) denote the latent distribution of the source and target domain samples, respectively; 5) multi-level joint of the above loss functions, and a total loss function is: where λ recon ,λ cont ,λ kg ,λ align are the weights of each loss function.

7. An electronic device, comprising: comprising a memory and a processor; The memory is configured to store a computer program; The processor is configured to implement the cross-modal endoscopic polyp collaborative detection method based on contrast learning according to any one of claims 1-6 when executing the computer program.

8. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by the processor, the cross-modal endoscopic polyp collaborative detection method based on contrast learning according to any one of claims 1-6 is implemented.

9. A computer program product, characterised in that: The computer program is executed by the processor to implement the cross-modal endoscopic polyp collaborative detection method based on contrast learning according to any one of claims 1-6.

Images