A method and device for detecting colon polyps based on endoscopic images

Abstract

This invention belongs to the technical field of medical auxiliary equipment, and in particular, relates to a method and apparatus for detecting colon polyps based on endoscopic images. The method comprises the following steps: S1, multi-scale feature extraction based on dilated convolution and a ResNet50 residual network; S2, feature enhancement using frame-assisted contrast learning; S3, prediction based on a CenterNet network; and S4, self-guided bounding box matching during the testing phase. By deploying this efficient detection method on a colonoscopy detection device, the present invention enables real-time and accurate localization of colonoscopic polyps during colonoscopy procedures, improving the efficiency of colonoscopic polyp examination and treatment, reducing the risks associated with multiple examinations and delayed treatment for patients, and providing a strong guarantee for the early diagnosis and treatment of intestinal diseases.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary equipment, and in particular to a method and device for detecting colon polyps based on endoscopic images. Background Art

[0002] Colorectal cancer is a common cancer. According to the latest cancer statistics released by the International Agency for Cancer, colorectal cancer ranks third and second among all cancers in terms of new cases and deaths, respectively. There were 1.9 million new cases and 935,000 deaths from colorectal cancer, respectively. These data demonstrate that colorectal cancer poses a serious threat to public health, making its prevention and diagnosis crucial. Studies have found that over 80% of colorectal cancers originate from polyps. Every 1% improvement in the early detection and removal of polyps reduces the conversion rate to malignant cancer by 3%. Therefore, identifying and removing malignant polyps is crucial for reducing colorectal cancer mortality. Colonoscopy is considered the gold standard for early polyp screening. Doctors use an endoscope to observe the patient's intestines, allowing for timely analysis and treatment of any polyps detected. However, its effectiveness is affected by various factors, including the doctor's skill level, the patient's preoperative preparation, the visibility of colorectal polyps, and the high rate of missed diagnosis of early colorectal cancer. Therefore, some researchers have proposed using computer-assisted diagnosis technology for colorectal polyp detection.

[0003] However, existing polyp detection methods are mainly designed based on the experience of previous natural video object detection research, ignoring the inherent uniqueness of colonoscopy motion patterns. Therefore, this study rethinks the task of video polyp detection and summarizes three core challenges in colonoscopy videos: first, there are substantial differences in examination quality due to different operating doctors and other reasons, and the problem of missed lesions still exists; second, the video jitter caused by camera movement during colonoscopy will cause image blur and prone to mirror reflection problems; third, endoscopic images usually contain strong boundary tissue structures such as colon folds, blood vessels, and bubbles, resulting in a high false positive rate for colon polyp detection. In summary, there are still huge difficulties and challenges in accurately detecting colonoscopic polyps, and existing related technologies still need further exploration in practical applications. Summary of the Invention

[0004] (1) Technical problems solved

[0005] In response to the deficiencies of the existing technology, the present invention provides a method and device for detecting colon polyps based on endoscopic images, which solves the problem of missed or false polyp detection caused by the difficulty in accurately identifying polyps during colorectal polyp removal surgery.

[0006] (2) Technical solution

[0007] In order to achieve the above-mentioned purpose, the invention specifically adopts the following technical solutions:

[0008] A method for detecting colon polyps based on endoscopic imaging comprises the following steps:

[0009] S1. Multi-scale feature extraction based on dilated convolution and resnet50 residual network, as follows:

[0010] The features of small objects are enhanced from two aspects. First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information.

[0011] Secondly, from the perspective of receptive field expansion, atrous convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use atrous convolution. The multi-branch convolution structure includes four branches, of which the first branch is a residual structure, which performs 1×1 convolution and 3×3 standard convolution on the input feature map; the second and third branches are atrous convolution branches, which first perform 1×1 convolution on the input feature map to adjust the number of channels, and then perform 1×3 and 3×1 standard convolutions in sequence, and finally extract features through 3×3 atrous convolution with an expansion rate of 5; the fourth branch performs 1×1 convolution for feature fusion and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations with convolution kernel sizes of 1×3, 3×1 and 3×3, respectively. Atrous convolution layers are added to the two middle branches so that the extracted feature maps can retain more context information. The mathematical expressions are as follows:

[0012] (1);

[0013] (2);

[0014] (3);

[0015] (4);

[0016] in, and Represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3 respectively; Represents a dilated convolution operation with a dilation rate of 5; Represents the connection operation of the feature map; ⊕ represents the element-by-element addition operation of the feature map; F is the input feature map; and Represents the output feature maps of the first three branches after standard convolution and dilated convolution; Y is the output feature map of the feature extraction module FEM, and FEM is a feature extraction unit composed of the multi-branch convolution structure and the residual structure;

[0017] Compared with RFB-s, FEM has a lighter structure and enables the model to learn richer local context features through multi-branch dilated convolution, thereby enhancing the ability to express polyp characteristics;

[0018] S2. Feature enhancement combined with frame-assisted contrastive learning, as follows:

[0019] In colonoscopy videos, some hidden polyps are very similar to the intestinal wall in color and texture. This phenomenon is particularly noticeable when encountering interference such as lighting changes or obvious shadows. Therefore, an advanced training strategy is needed to distinguish this homogeneity. Inspired by recent research on supervised contrastive learning, contrastive learning is guided by ground truth boxes. Specifically, to address problems such as lighting and shadows, adjacent frame alignment is proposed.

[0020] First, the foreground feature is the in-frame feature of the colonoscopy polyp detection frame, which is the original anchor feature Fa. The attention map is calculated through the channel attention mechanism, and the attention map is combined with Fa through a jump connection to obtain the enhanced anchor feature. The formula is as follows:

[0021] (5);

[0022] in, It is the weight that is adaptively adjusted according to the similarity metric and is determined through neural network training. It is the original anchor feature, which can be understood as the in-frame feature of the colonoscopy polyp detection frame. It is the result of normalizing the adjacent frames.

[0023] Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows:

[0024] (6);

[0025] Use 3×3 deformable convolution to transform the dynamic field of the background Incorporating enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows:

[0026] (7);

[0027] The above formula is to add the obtained dynamic field to the anchor feature to achieve feature alignment. For the enhanced anchor feature and reference features , using the inter-frame difference method to calculate the dynamic field of the reference frame and the target frame, given a batch of And the corresponding binary image ;

[0028] in, Represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. represents the number of channels, Represents the height of the feature map, Represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask;

[0029] First, the anchors and reference frames are concatenated at the batch level as and , in order to utilize cross-frame information; then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ;

[0030] Then, for each foreground channel pattern, i.e., “query”, another different foreground feature is randomly selected as the “positive sample”, and all background features in the same batch are used as “negative samples”; finally, the one-step contrast loss is calculated by InfoNCE:

[0031] (8);

[0032] Among them, this formula is the specific form of InfoNCE (a contrastive loss function). In contrastive learning, the model learns the similarities and differences between samples. middle, is the query feature, which is the pattern of each foreground channel. is the positive sample feature, Represents the dot product of query features and positive sample features, which is used to measure the similarity between them. The function exponentially scales this similarity score. is the temperature hyperparameter, which is used to adjust the sharpness of the distribution, The smaller the value, the sharper the distribution and the stricter the distinction between positive and negative samples. In the above example, all negative sample features ( Belong to the set , is the embedding set of negative samples) and query features The dot product results are exponentially scaled and summed. The denominator is used to normalize the similarity scores of positive and negative samples.

[0033] This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrastive loss:

[0034] (9);

[0035] The above calculation yields the loss for a single sample. Repeating this process and averaging it yields the final loss. By combining adjacent frames with self-supervised comparative learning, cross-frame information is effectively leveraged to enhance feature learning, solving the problem of distinguishing background and foreground in colonoscopy polyp detection.

[0036] S3, prediction based on the centernet network, as follows:

[0037] Use a trained centernet network target detection model to infer a set of test images, denoted as ; For each image , and get a set of bounding boxes ;

[0038] in, , since the number of bounding boxes generated for each image during inference may be different, Representing an image The number of bounding boxes generated;

[0039] in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty;

[0040] S4, self-guided bounding box matching in the test phase, as follows:

[0041] The label assignment of low-confidence bounding boxes is affected by high-confidence bounding boxes. In the subsequent matching process, the feature similarity between the bounding boxes needs to be compared;

[0042] Therefore, for each bounding box Introducing additional feature information , is obtained by Crop out the bounding box The image block within the image block, the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula:

[0043] (10);

[0044] It can be understood that through the training of the neural network, the weight network is initialized and the network predicts a series of detection frames. The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, recorded as and , set the confidence threshold and ;

[0045] For categories Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, similarly, for the category Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, found HF and The bounding box will be used as the initial confirmation and Class bounding box; then, from HF and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching. To achieve more efficient and effective matching performance, the classic K-means algorithm is used. All bounding boxes that have not been added to the set of confirmed bounding boxes are considered candidate bounding boxes.

[0046] In order to ensure the reliability of subsequent multiple rounds of EMD matching and bounding box propagation, it is necessary to calculate the distance constraint for each category. Bounding box, calculate the bounding box and other HF The shortest Euclidean distance between bounding boxes, for all HF After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ;

[0047] Similarly, this method is effective for HF The same process is performed on the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, and as a constraint;

[0048] Specifically, the multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. This is done to ensure that the bounding boxes added in the early stages are more likely to be correctly assigned labels. In the second stage, the constraints are relaxed to allow the bounding boxes to propagate across a wider range of candidate boxes.

[0049] In each round of propagation, assuming there are a candidate bounding boxes and m confirmed bounding boxes, among these confirmed bounding boxes, t belong to category c1 and the remaining m−t belong to category c2. The input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box;

[0050] Specifically, Including t confirmed c1 class bounding boxes and m−t confirmed c2 class bounding boxes, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is obtained. and The matching traffic between Indicates that its potential value is 0 or 1;

[0051] The matching optimization goal is:

[0052] (11);

[0053] The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the detection box to the final result. The constraints are: each confirmed bounding box matches at most one candidate bounding box, each candidate bounding box matches at most one confirmed bounding box, and a candidate bounding box can only match at most one confirmed bounding box. When matching traffic , represents the candidate bounding box Successfully and confirmed bounding box match;

[0054] (12);

[0055] (13);

[0056] (14);

[0057] (15);

[0058] In the first stage of propagation, for each The associated confirmed c1 class bounding box (where ),turn up Make ,if Less than distance constraint , the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ), then update the confirmed bounding box set To the new state, the newly added c1 and c2 class bounding boxes are added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching;

[0059] In the first stage of propagation, for the bounding box associated with the confirmed bounding box of c1, the found of , calculate its Euclidean shortest distance is ;

[0060] The second phase of propagation starts when no new bounding boxes have been added to the confirmed sets of classes c1 and c2 in the previous round of propagation, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, that is, any matching pair found in EMD matching will be used to add the candidate bounding box to the confirmed set of bounding boxes;

[0061] Specifically, for each The associated confirmed c1 class bounding box (where ), this method finds , making , and then the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where );

[0062] The termination condition of multi-round bounding box propagation is when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the set of confirmed bounding boxes are reported as the final detection results, covering all bounding boxes of categories c1 and c2.

[0063] A colon polyp detection device based on endoscopic images, wherein the colon polyp detection device performs colon polyp detection using a colon polyp detection method based on endoscopic images;

[0064] Deploying the above polyp detection algorithm to a colonoscopy polyp detection device involves several key steps.

[0065] First, in terms of hardware integration, high-performance embedded chips or GPUs are used to accelerate the computing process and ensure timely feedback of results in real-time detection scenarios. For self-guided bounding box matching during the testing phase, the device needs to have precise image positioning and annotation capabilities. When a possible polyp area is detected, the bounding box can be accurately drawn and its position and size can be continuously adjusted through a self-guided mechanism to closely match the actual shape of the polyp. This requires the device's image display and processing systems to have high resolution and low latency, so that doctors can clearly observe the test results in real time.

[0066] In addition, a lot of testing and verification work was carried out, including the use of a model simulating the intestinal environment to verify the detection accuracy of the device. After rigorous testing and optimization, this vision-based colonoscopic polyp detection method can be deployed on the device and truly applied in clinical practice, providing doctors with accurate and reliable polyp detection assistance, thereby improving the efficiency of colonoscopic polyp detection and treatment and protecting the health of patients.

[0067] (3) Beneficial effects

[0068] Compared with the prior art, the present invention provides a method and device for detecting colon polyps based on endoscopic imaging, which has the following beneficial effects:

[0069] The present invention, by deploying the efficient detection method on a colonoscopy detection device, can achieve real-time and accurate positioning of colonoscopic polyps during the colonoscopy operation, thereby improving the efficiency of colonoscopic polyp examination and treatment, reducing the risks of patients due to multiple examinations and delayed treatment, and providing a strong guarantee for the early diagnosis and treatment of intestinal diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 Schematic diagram of the multi-scale feature extraction block of the dilated convolution in the present invention;

[0071] Figure 2 Schematic diagram of the feature enhancement block for contrastive learning in the present invention;

[0072] Figure 3 Schematic diagram of the self-guided bounding box matching algorithm in the present invention. DETAILED DESCRIPTION

[0073] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0074] Example

[0075] One embodiment of the present invention provides a method for detecting colon polyps based on endoscopic imaging, comprising the following steps:

[0076] (1) Multi-scale feature extraction method based on dilated convolution, as follows:

[0077] The designed network structure is as follows Figure 1 As shown, this method enhances the features of small objects from two aspects;

[0078] First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information;

[0079] Secondly, from the perspective of receptive field expansion, atrous convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use atrous convolution. The multi-branch convolution structure includes four branches, of which the first branch is a residual structure, which performs 1×1 convolution and 3×3 standard convolution on the input feature map; the second and third branches are atrous convolution branches, which first perform 1×1 convolution on the input feature map to adjust the number of channels, and then perform 1×3 and 3×1 standard convolutions in sequence, and finally extract features through 3×3 atrous convolution with an expansion rate of 5; the fourth branch performs 1×1 convolution for feature fusion and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations with convolution kernel sizes of 1×3, 3×1 and 3×3, respectively. Atrous convolution layers are added to the two middle branches so that the extracted feature maps can retain more context information. The mathematical expressions are as follows:

[0080] (1);

[0081] (2);

[0082] (3);

[0083] (4);

[0084] in, and Represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3 respectively; Represents a dilated convolution operation with a dilation rate of 5; Represents the connection operation of the feature map; ⊕ represents the element-by-element addition operation of the feature map; F is the input feature map; and Represents the output feature maps of the first three branches after standard convolution and dilated convolution; Y is the output feature map of the feature extraction module FEM, and FEM is a feature extraction unit composed of the multi-branch convolution structure and the residual structure;

[0085] Compared with RFB-s, FEM has a lighter structure and enables the model to learn richer local context features through multi-branch dilated convolution, thereby enhancing the ability to express polyp characteristics.

[0086] (2) Feature enhancement method combining adjacent frame-assisted contrast learning, as follows:

[0087] In colonoscopy videos, some hidden polyps are very similar to the intestinal wall in color and texture, especially when encountering interference such as lighting changes or obvious shadows. Therefore, an advanced training strategy is needed to distinguish this homogeneity; Figure 2 As shown in the figure, inspired by the recent research on supervised contrastive learning, contrastive learning is performed through the guidance of real frames; specifically, in order to solve problems such as lighting and shadows, it is proposed to use adjacent frame alignment. First, the foreground feature is the in-frame feature of the colonoscopy polyp detection frame, which is the original anchor feature Fa. The attention map is calculated through the channel attention mechanism, and the attention map is combined with Fa through a jump connection to obtain the enhanced anchor feature. The channel attention mechanism is used to temporally align the foreground features; the attention map is calculated by weighted dot product, and these attention maps are weightedly combined with the original anchor features to achieve temporal alignment of the foreground; finally, the enhanced anchor feature is obtained by combining the attention map with the original feature through a jump connection. The formula is as follows:

[0088] (5);

[0089] in, is the weight that is adaptively adjusted according to the similarity metric, that is, the parameter to be adjusted in the neural network, It is the original anchor feature, which can be understood as the in-frame feature of the colonoscopy polyp detection frame. It is the result of normalizing the adjacent frames. Normalization can be intuitively understood as averaging.

[0090] Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows:

[0091] (6);

[0092] Use 3×3 deformable convolution to transform the dynamic field of the background Incorporating enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows:

[0093] (7);

[0094] The above formula is to add the obtained dynamic field to the anchor feature to achieve feature alignment. For the enhanced anchor feature and reference features , using the inter-frame difference method, calculate the dynamic field of the reference frame and the target frame, given a batch of And the corresponding binary image ;

[0095] in, Represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. Represents the number of channels, such as the three RGB channels commonly seen in images. The number of channels in the feature map may vary depending on the design and calculation of the model. It reflects the different feature dimension information in the feature map. Represents the height of the feature map, Represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask;

[0096] First, the anchors and reference frames are concatenated at the batch level as and , in order to utilize cross-frame information; then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ;

[0097] Then, for each foreground channel pattern, i.e., “query”, another different foreground feature is randomly selected as the “positive sample”, and all background features in the same batch are used as “negative samples”; finally, the one-step contrast loss is calculated by InfoNCE:

[0098] (8);

[0099] Among them, this formula is the specific form of InfoNCE (a contrastive loss function). In contrastive learning, the model learns the similarities and differences between samples. middle, is the query feature, which is the pattern of each foreground channel. is the positive sample feature, Represents the dot product of query features and positive sample features, which is used to measure the similarity between them. The function exponentially scales this similarity score. is the temperature hyperparameter, which is used to adjust the sharpness of the distribution, The smaller the value, the sharper the distribution and the stricter the distinction between positive and negative samples. In the above example, all negative sample features ( Belong to the set , is the embedding set of negative samples) and query features The dot product results are exponentially scaled and summed. The denominator is used to normalize the similarity scores of positive and negative samples.

[0100] This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrastive loss:

[0101] (9);

[0102] The above calculation can obtain the calculation result of a sample loss. The above process is repeated and averaged to obtain the final loss. By combining the adjacent frame-assisted self-supervised comparative learning method, cross-frame information is effectively utilized to enhance feature learning, solving the problem of background and foreground distinction in colonoscopic polyp detection.

[0103] (3) Based on the self-guided bounding box matching algorithm, the details are as follows:

[0104] like Figure 3 As shown in the figure, a general overview of the proposed method is provided. A trained target detection model (CenterNet network is selected) is used to perform inference on a set of test images, denoted as ; For each image , and get a set of bounding boxes ;

[0105] in, , since the number of bounding boxes generated for each image during inference may be different, Representing an image The number of bounding boxes generated;

[0106] in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty;

[0107] Next, we will describe the method proposed in this paper to influence the label assignment of low-confidence bounding boxes through high-confidence bounding boxes. In the subsequent matching process, it is necessary to compare the feature similarity between bounding boxes;

[0108] Therefore, for each bounding box Introducing additional feature information , is obtained by Crop out the bounding box The image block within the image block, the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula:

[0109] (10);

[0110] It can be understood that through the training of the neural network, the weight network is initialized and the network predicts a series of detection frames. The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, recorded as and , set the confidence threshold and ;

[0111] For categories Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, similarly, for the category Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, found HF and The bounding box will be used as the initial confirmation and Class bounding box; then, from HF and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching steps. In order to achieve more efficient and effective matching performance, the classic K-means algorithm is used, and all bounding boxes that have not been added to the confirmed bounding box set will be regarded as candidate bounding boxes.

[0112] In order to ensure the reliability of subsequent multiple rounds of EMD matching and bounding box propagation, it is necessary to calculate the distance constraint for each category. Bounding box, calculate the bounding box and other HF The shortest Euclidean distance between bounding boxes, for all HF After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ;

[0113] Similarly, this method is effective for HF The same process is performed on the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, and as a constraint;

[0114] Specifically, the multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. This is done to ensure that the bounding boxes added in the early stages are more likely to be correctly assigned labels. In the second stage, the constraints are relaxed to allow the bounding boxes to propagate across a wider range of candidate boxes.

[0115] In each round of propagation, assuming there are a candidate bounding boxes and m confirmed bounding boxes, among these confirmed bounding boxes, t belong to category c1 and the remaining m−t belong to category c2. The input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box;

[0116] Specifically, Including t confirmed c1 class bounding boxes and m−t confirmed c2 class bounding boxes, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is obtained. and The matching traffic between Indicates that its potential value is 0 or 1;

[0117] The matching optimization goal is:

[0118] (11);

[0119] The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the detection box to the final result. The constraints are: each confirmed bounding box matches at most one candidate bounding box, each candidate bounding box matches at most one confirmed bounding box, and a candidate bounding box can only match at most one confirmed bounding box. When matching traffic , represents the candidate bounding box Successfully and confirmed bounding box match;

[0120] (12);

[0121] (13);

[0122] (14);

[0123] (15);

[0124] A confirmed bounding box (either c1 or c2 class) can only match with one candidate bounding box at most. Formula (13) shows that a candidate bounding box can only match with one confirmed bounding box at most. , represents the candidate bounding box Successfully and confirmed bounding box match;

[0125] In the first stage of propagation, for each The associated confirmed c1 class bounding box (where ),turn up , making ,if Less than distance constraint , the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ), then update the confirmed bounding box set To the new state, the newly added c1 and c2 class bounding boxes are added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching;

[0126] In the first stage of propagation, for the bounding box associated with the confirmed bounding box of c1, the found of , calculate its Euclidean shortest distance is ;

[0127] The second phase of propagation starts when no new bounding boxes have been added to the confirmed sets of classes c1 and c2 in the previous round of propagation, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, that is, any matching pair found in EMD matching will be used to add the candidate bounding box to the confirmed set of bounding boxes;

[0128] Specifically, for each The associated confirmed c1 class bounding box (where ), this method finds , making , and then the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where );

[0129] The termination condition of multi-round bounding box propagation is when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the set of confirmed bounding boxes are reported as the final detection results, covering all bounding boxes of categories c1 and c2.

[0130] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art will be able to modify the technical solutions described in the aforementioned embodiments or substitute equivalents for some of the technical features. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for detecting colon polyps based on endoscopic imaging, characterized in that: The steps include: S1, multi-scale feature extraction based on dilated convolution and resnet50 residual network; S2. Feature enhancement combined with frame-assisted contrast learning is as follows: First, the foreground feature is the in-frame feature of the colonoscopy polyp detection frame, which is the original anchor feature Fa. The attention map is calculated through the channel attention mechanism, and the attention map is combined with Fa through a skip connection to obtain the enhanced anchor feature. The formula is as follows: （5）； in, It is the weight that is adaptively adjusted according to the similarity metric and is determined through neural network training. It is the original anchor feature, which is understood as the in-frame feature of the colonoscopy polyp detection frame. It is the result of normalizing the adjacent frames. Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows: （6）； Use 3×3 deformable convolution to transform the dynamic field of the background Incorporating enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows: （7）； The above formula is to add the obtained dynamic field to the anchor feature. For the enhanced anchor feature and reference features , using the inter-frame difference method to calculate the dynamic field of the reference frame and the target frame, given a batch of And the corresponding binary image ; in, Represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. Represents the number of channels, Represents the height of the feature map, Represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask; First, the anchors and reference frames are concatenated at the batch level as and ; Then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ; Then, for each foreground channel pattern, i.e., "query", another different foreground feature is randomly selected as the "positive sample", and all background features in the same batch are used as "negative samples"; finally, the one-step contrast loss is calculated by InfoNCE: （8）； Among them, this formula is the specific form of InfoNCE. In contrastive learning, the model learns the similarities and differences between samples. In the numerator of the formula middle, is the query feature, which is each foreground channel mode. is the positive sample feature, Represents the dot product of query features and positive sample features, which is used to measure the similarity between them. The function exponentially scales this similarity score. is the temperature hyperparameter, which is used to adjust the sharpness of the distribution, The smaller the value, the sharper the distribution and the stricter the distinction between positive and negative samples. In the above example, all negative sample features Belong to the set , is the embedding set of negative samples, negative sample features and query features The dot product results are exponentially scaled and summed. The denominator is used to normalize the similarity scores of positive and negative samples. This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrastive loss: （9）； S3, prediction based on centernet network; S4, self-guided bounding box matching in the test phase, as follows: The label assignment of low-confidence bounding boxes is affected by high-confidence bounding boxes. In the subsequent matching process, the feature similarity between the bounding boxes needs to be compared; Therefore, for each bounding box Introducing additional feature information , is obtained by Crop out the bounding box The image block within the image block, the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula: （10）； in, The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, recorded as and , set the confidence threshold and ; For categories Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, similarly, for the category Bounding box of and confidence scores , which is regarded as a high confidence HF Bounding box, found HF and The bounding box will be used as the initial confirmation and Class bounding box; then, from HF and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching steps. The classic K-means algorithm is used, and all bounding boxes that have not been added to the set of confirmed bounding boxes are considered candidate bounding boxes. Calculate the distance constraint for each category, for each HF Bounding box, calculate the bounding box and other HF The shortest Euclidean distance between bounding boxes, for all HF After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ; Similarly, for HF The same process is performed on the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, and as a constraint; The multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. In the second stage, the constraints are relaxed. In each round of propagation, assuming there is candidate bounding boxes and confirmed bounding boxes, among these confirmed bounding boxes, belongs to the category , the rest belongs to the category , the input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box; Specifically, include Confirmed Class bounding box and Confirmed Class bounding box, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is obtained. and The matching traffic between Indicates that its potential value is 0 or 1; The matching optimization goal is: （11）； The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the detection box to the final result. The constraints are: each confirmed bounding box matches at most one candidate bounding box, each candidate bounding box matches at most one confirmed bounding box, and a candidate bounding box can only match at most one confirmed bounding box. , represents the candidate bounding box Successfully and confirmed bounding box match; （12）； （13）； （14）； （15）； In the first stage of propagation, for each Related confirmed Class bounding box, where ,turn up , making ,if Less than distance constraint , the corresponding candidate bounding box Join Confirmed The same process applies to the class bounding box collection. Related Class bounding box, where , then, update the confirmed bounding box set To the new state, the newly added and The class bounding box is added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching; In the first stage of dissemination, Confirmed bounding box associated with the bounding box found of , calculate its Euclidean shortest distance is ; In the second phase of propagation, no new bounding boxes are added to the and The confirmed set of classes begins, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, and any matching pairs found in the EMD matching will be used to add the candidate bounding box to the confirmed set of bounding boxes; For each Related confirmed Class bounding box, where ,turn up , making , and then the corresponding candidate bounding box Join Confirmed The same process applies to the class bounding box collection. Related Class bounding box, where ; The termination condition of multiple rounds of bounding box propagation is: when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the confirmed bounding box set are reported as the final detection results, covering and All bounding boxes for the category.

2. The method for detecting colon polyps based on endoscopic imaging according to claim 1, characterized in that: The multi-scale feature extraction in S1 is as follows: First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information; Secondly, from the perspective of receptive field expansion, dilated convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use dilated convolution. The multi-branch convolution structure includes four branches, of which the first branch is a residual structure that performs 1×1 convolution and 3×3 standard convolution on the input feature map. The second and third branches are dilated convolution branches. They first perform 1×1 convolution on the input feature map to adjust the number of channels, then perform 1×3 and 3×1 standard convolutions in sequence, and finally extract features through 3×3 dilated convolution with an expansion rate of 5. The fourth branch performs 1×1 convolution for feature fusion and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations with convolution kernel sizes of 1×3, 3×1, and 3×3, respectively. A dilated convolution layer is added to the two middle branches so that the extracted feature maps can retain more contextual information. The mathematical expression is as follows: （1）； （2）； （3）； （4）； in, and Represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3 respectively; Represents a dilated convolution operation with a dilation rate of 5; Represents the connection operation of the feature map; Represents the element-wise addition operation of the feature map; is the input feature map; and It represents the output feature map of the first three branches after standard convolution and dilated convolution; Y is the output feature map of the feature extraction module FEM, and FEM is a feature extraction unit composed of the multi-branch convolution structure and the residual structure.

3. The method for detecting colon polyps based on endoscopic imaging according to claim 1, characterized in that: The prediction in S3 is based on the centernet network, as follows: Use a trained centernet network target detection model to infer a set of test images, denoted as ; For each image , and get a set of bounding boxes ; in, , since the number of bounding boxes generated for each image during inference may be different, Representing an image The number of bounding boxes generated; in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty.

4. A colon polyp detection device based on endoscopic imaging, characterized by: The colon polyp detection device performs colon polyp detection using the method described in any one of claims 1-3.

Images