Colon polyp detection method and device based on endoscope image

Abstract

The invention belongs to the technical field of medical auxiliary equipment, and particularly relates to a colon polyp detection method and device based on an endoscope image, and the method comprises the following steps: S1, carrying out the multi-scale feature extraction based on cavity convolution and a resnet50 residual network; s2, feature enhancement of combined frame-assisted contrast learning; s3, prediction is carried out based on the internet network; and S4, performing self-guiding bounding box matching in a test stage. According to the method, the efficient detection method is deployed on a colonoscope detection device, real-time and accurate positioning of colonoscope polyp can be achieved in the colonoscope examination operation process, the efficiency of colonoscope polyp examination and treatment is improved, the risk caused by multiple examinations and delayed treatment of a patient is reduced, and the patient experience is improved. And a powerful guarantee is provided for early diagnosis and treatment of intestinal diseases.

Description

Technical Field

[0001] The present invention relates to the technical field of medical auxiliary equipment, and in particular to a method and device for detecting colon polyps based on endoscopic images. Background Art

[0002] Colorectal cancer is one of the most common cancers. According to the latest cancer statistics released by the International Agency for Cancer, the number of new cases and deaths from colorectal cancer ranks third and second among all cancers, respectively. The number of new cases and deaths from colorectal cancer is 1.9 million and 935,000, respectively. The above data show that colorectal cancer seriously threatens people's health, and how to prevent and diagnose colorectal cancer is crucial. Studies have found that more than 80% of colorectal cancers originate from polyps. Every 1% increase in the early detection rate of polyps and necessary removal will reduce the colorectal cancer conversion rate by 3%. Therefore, identifying and removing malignant polyps is crucial to reducing the mortality rate of colorectal cancer. Colonoscopy is regarded as the gold standard technology for early polyp screening. Doctors can observe the situation in the patient's intestines through an endoscope, and can analyze and treat polyps in time after they are found. However, its effect is affected by many factors, such as the doctor's operating skills, the patient's preoperative preparation, the visibility of colorectal polyps, and the missed diagnosis rate of early colorectal cancer. Therefore, some researchers have proposed the use of computer-aided diagnosis technology for colorectal polyp detection.

[0003] However, existing polyp detection methods are mainly designed based on the experience of previous natural video object detection research, ignoring the inherent uniqueness of colonoscopy motion patterns. Therefore, this study rethinks the task of video polyp detection and summarizes three core challenges in colonoscopy videos: First, there are substantial differences in the quality of examinations due to different operating doctors and other reasons, and the problem of missed lesions still exists; second, the video jitter caused by the movement of the camera during colonoscopy will cause image blur and prone to mirror reflection problems; third, endoscopic images usually contain strong boundary tissue structures such as colon folds, blood vessels, and bubbles, resulting in a high false positive rate of colon polyp detection. In summary, there are still huge difficulties and challenges in accurately detecting colonoscopic polyps, and the existing related technologies still need to be further explored in practical applications. Summary of the invention

[0004] 1. Technical issues to be resolved In view of the deficiencies in the prior art, the present invention provides a method and device for detecting colon polyps based on endoscopic images, which solves the problem of missed or false polyp detection caused by the difficulty in accurately identifying polyps during colorectal polyp surgery.

[0005] (II) Technical solution In order to achieve the above-mentioned purpose, the present invention specifically adopts the following technical solutions: A method for detecting colon polyps based on endoscopic images comprises the following steps: S1. Multi-scale feature extraction based on dilated convolution and resnet50 residual network, as follows: Enhance the features of small objects from two aspects; First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information; Secondly, from the perspective of expanding the receptive field, atrous convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use atrous convolution. Each branch performs a 1×1 convolution operation on the input feature map, and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations, and the convolution kernel sizes are 1×3, 3×1, and 3×3, respectively. Atrous convolution layers are added to the two middle branches, so that the extracted feature maps can retain more context information. The mathematical expression is as follows: (1) (2) (3) (4) in, , , and They represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3; represents a dilated convolution operation with a dilation rate of 5; represents the connection operation of the feature map; ⊕ represents the element-by-element addition operation of the feature map; F is the input feature map; , and It represents the output feature map of the first three branches after standard convolution and dilated convolution; Y is the output feature map of FEM; Compared with RFB-s, FEM has a lighter structure and enables the model to learn richer local context features through multi-branch hole convolution, thereby enhancing the ability to express polyp features; S2. Feature enhancement combined with frame-assisted contrastive learning, as follows: In colonoscopy videos, some hidden polyps are very similar to the intestinal wall in color and texture, especially when encountering interference such as lighting changes or obvious shadows. Therefore, an advanced training strategy is needed to distinguish this homogeneity. Inspired by recent supervised contrastive learning research, contrastive learning is guided by ground-truth boxes. Specifically, to solve problems such as lighting and shadows, adjacent frame alignment is proposed. First, the channel attention mechanism is used to temporally align the foreground features; the attention maps are calculated by weighted dot products, and these attention maps are combined with the original anchor features. The weighted combination is performed to achieve the temporal alignment of the foreground; finally, the enhanced anchor feature is obtained by combining the attention map with the original feature through a jump connection, and the formula is as follows: (5) in, is the weight that is adaptively adjusted according to the similarity metric, that is, the parameter to be adjusted in the neural network, It is the original anchor feature, which can be understood as the in-frame feature of the colonoscopy polyp detection frame. is the weight adaptively adjusted according to the similarity metric, is the result of feature normalization; Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows: (6) Use 3×3 deformable convolution to transform the background dynamic field Enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows: (7) The above formula is to add the obtained dynamic field to the anchor feature to achieve feature alignment. For the enhanced anchor feature and reference features , using the inter-frame difference method, calculate the dynamic field of the reference frame and the target frame, given a batch of intermediate feature maps And the corresponding binary image ,in, represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. Represents the number of channels, such as the three RGB channels commonly seen in images. The number of channels in the feature map may vary depending on the design and calculation of the model. It reflects the different feature dimension information in the feature map. Represents the height of the feature map, represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask; First, anchors and reference frames are concatenated at batch level as and , in order to utilize cross-frame information; then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ; Then, for each foreground channel pattern, i.e., “query”, another different foreground feature is randomly selected as the “positive sample”, while all background features in the same batch are used as “negative samples”; finally, the one-step contrast loss is calculated by InfoNCE: (8) Among them, this formula is a specific form of InfoNCE (a contrastive loss function). In contrastive learning, the model learns the similarities and differences between samples. middle, is the query feature, which is the mode of each foreground channel. is the positive sample feature, Represents the dot product of the query feature and the positive sample feature, which is used to measure the similarity between them. The function exponentiates this similarity score, is the temperature hyperparameter that adjusts the sharpness of the distribution, The smaller it is, the sharper the distribution is, and the stricter the distinction between positive and negative samples is. In the above example, all negative sample features ( is the embedding set of negative samples) and the query features The dot product results are exponentialized and summed. The denominator is used to normalize the similarity scores of positive and negative samples. This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrast loss: (9) The above calculation can get the calculation result of a sample loss. Repeat the above process and average to get the final loss. By combining the adjacent frame-assisted self-supervised contrast learning method, cross-frame information is effectively used to enhance feature learning, solving the problem of background and foreground distinction in colonoscopy polyp detection. S3, prediction based on centernet network, as follows: Use a trained centernet network target detection model to infer a set of test images, denoted as ; For each image , and get a set of bounding boxes ; in, , since the number of bounding boxes generated during inference for each image may be different, Representing images The number of bounding boxes generated; in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty; S4, self-guided bounding box matching in the test phase, as follows: The label assignment of the low-confidence bounding box is affected by the high-confidence bounding box. In the subsequent matching process, the feature similarity between the bounding boxes needs to be compared; Therefore, for each bounding box Introducing additional feature information , is done by Crop out the bounding box The image block in the image block is obtained, and the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula: (10) It can be understood that through the training of the neural network, an initialized weight network is obtained, and the network predicts a series of detection boxes. The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, denoted as and , set the confidence threshold and ; For categories The bounding box and the confidence score , which is considered as high confidence (HF) Bounding box, similarly, for the category The bounding box and the confidence score , which is considered as high confidence (HF) Bounding Box, Found (HF) and The bounding box will be used as the initial confirmation and Class bounding box; then, from (HF) and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching steps. In order to achieve more efficient and effective matching performance, the classic K-means algorithm is used. All bounding boxes that have not been added to the confirmed bounding box set will be considered as candidate bounding boxes; In order to ensure the reliability of subsequent multiple rounds of EMD matching and bounding box propagation, it is necessary to calculate the distance constraint for each category. Bounding box, calculate the bounding box and other (HF) The shortest Euclidean distance between bounding boxes, for all (HF) After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ; Similarly, this method has an effect on (HF) The same process is performed for the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, will be applied and as a constraint; Specifically, the multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. This is done to ensure that the bounding boxes added in the early stages are more likely to be correctly assigned labels. In the second stage, the constraints are relaxed to allow the bounding boxes to propagate across a wider range of candidate boxes. In each round of propagation, assuming there are a candidate bounding boxes and m confirmed bounding boxes, among these confirmed bounding boxes, t belong to category c1, and the remaining m−t belong to category c2. The input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box; Specifically, Including t confirmed c1 class bounding boxes and m−t confirmed c2 class bounding boxes, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is used to derive and The matching flow between Indicates that its potential value is 0 or 1; The matching optimization goal is: (11) The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the undetermined detection box to the final result. The constraints are: it is understood that a confirmed bounding box can only match with one candidate bounding box at most, and a candidate bounding box can only match with one confirmed bounding box at most. When matching traffic , represents the candidate bounding box Successful and confirmed bounding box match; (12) (13) (14) (15) A confirmed bounding box (whether c1 or c2) can only match with one candidate bounding box at most. Formula (3) shows that a candidate bounding box can only match with one confirmed bounding box at most. , represents the candidate bounding box Successful and confirmed bounding box match; In the first stage of propagation, for each The associated confirmed c1 class bounding box (where ),turn up ,if , the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ), then update the confirmed bounding box set To the new state, the newly added c1 and c2 class bounding boxes are added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching; in, : is the matching flow between the candidate bounding box and the confirmed bounding box, with a value of 0 or 1, 0 means no match, 1 means match. In the first stage of propagation, for the bounding box associated with the confirmed bounding box of c1, the found =1 , calculate its Euclidean shortest distance is ; The second phase of propagation starts when no new bounding boxes have been added to the confirmed sets of classes c1 and c2 in the previous round of propagation, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, that is, any matching pair found in the EMD matching will be used to add the candidate bounding box to the confirmed set of bounding boxes; Specifically, for each The associated confirmed c1 class bounding box (where ), this method finds , so that , and then the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ); The termination condition of multiple rounds of bounding box propagation is when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the set of confirmed bounding boxes are reported as the final detection results, covering all bounding boxes of categories c1 and c2.

[0006] A colon polyp detection device based on endoscopic images, the colon polyp detection device performs colon polyp detection using a colon polyp detection method based on endoscopic images; Deploying the above polyp detection algorithm to the colonoscope polyp detection device involves several key links; First, in terms of hardware integration, high-performance embedded chips or GPUs are used to accelerate the computing process to ensure timely feedback of results in real-time detection scenarios. In terms of self-guided bounding box matching during the test phase, the device needs to have precise image positioning and annotation functions. When a possible polyp area is detected, the bounding box can be accurately drawn, and the position and size of the bounding box can be continuously adjusted through the self-guided mechanism to closely fit the actual shape of the polyp. This requires the image display and processing system of the device to have high resolution and low latency so that doctors can clearly and real-time observe the test results. In addition, a lot of testing and verification work was carried out, including the use of a model that simulates the intestinal environment to verify the detection accuracy of the device. After rigorous testing and optimization, this vision-based colonoscopic polyp detection method can be deployed on the device and truly applied in clinical practice, providing doctors with accurate and reliable polyp detection assistance, thereby improving the efficiency of colonoscopic polyp detection and treatment and protecting the health of patients.

[0007] (III) Beneficial effects Compared with the prior art, the present invention provides a method and device for detecting colon polyps based on endoscopic images, which has the following beneficial effects: The present invention, by deploying the efficient detection method on a colonoscopy detection device, can achieve real-time and accurate positioning of colonoscopic polyps during the colonoscopy operation, thereby improving the efficiency of colonoscopic polyp examination and treatment, reducing the risks of patients due to multiple examinations and delayed treatment, and providing a strong guarantee for the early diagnosis and treatment of intestinal diseases. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Schematic diagram of a multi-scale feature extraction block of a dilated convolution in the present invention; Figure 2 Schematic diagram of a feature enhancement block for contrastive learning in the present invention; Figure 3 Schematic diagram of the self-guided bounding box matching algorithm in the present invention. DETAILED DESCRIPTION

[0009] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0010] Example An embodiment of the present invention provides a method for detecting colon polyps based on endoscopic images, comprising the following steps: (1) Multi-scale feature extraction method based on dilated convolution, as follows: The designed network structure is as follows Figure 1 As shown, this method enhances the features of small objects from two aspects; First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information; Secondly, from the perspective of expanding the receptive field, atrous convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use atrous convolution. Each branch performs a 1×1 convolution operation on the input feature map, and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations, and the convolution kernel sizes are 1×3, 3×1, and 3×3, respectively. Atrous convolution layers are added to the two middle branches, so that the extracted feature maps can retain more context information. The mathematical expression is as follows: (1) (2) (3) (4) in, , , and They represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3; represents a dilated convolution operation with a dilation rate of 5; represents the connection operation of the feature map; ⊕ represents the element-by-element addition operation of the feature map; F is the input feature map; , and It represents the output feature map of the first three branches after standard convolution and dilated convolution; Y is the output feature map of FEM; Compared with RFB-s, FEM has a lighter structure and enables the model to learn richer local context features through multi-branch hole convolution, thereby enhancing the ability to express polyp characteristics.

[0011] (2) The feature enhancement method combined with adjacent frame-assisted contrast learning is as follows: In colonoscopy videos, some hidden polyps are very similar to the intestinal wall in color and texture, especially when encountering interference such as lighting changes or obvious shadows; therefore, an advanced training strategy is needed to distinguish this homogeneity; Figure 2 As shown in the figure, inspired by the recent research on supervised contrastive learning, contrastive learning is performed through the guidance of real boxes; specifically, in order to solve problems such as lighting and shadows, it is proposed to use adjacent frame alignment. First, the channel attention mechanism is used to align the foreground features in time; the attention maps are calculated by weighted dot products, and these attention maps are combined with the original anchor features. The weighted combination is performed to achieve the temporal alignment of the foreground; finally, the enhanced anchor feature is obtained by combining the attention map with the original feature through a jump connection, and the formula is as follows: (5) in, is the weight that is adaptively adjusted according to the similarity metric, that is, the parameter to be adjusted in the neural network, It is the original anchor feature, which can be understood as the in-frame feature of the colonoscopy polyp detection frame. It is the result of normalizing the adjacent frames. Normalization can be intuitively understood as averaging. Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows: (6) Use 3×3 deformable convolution to transform the background dynamic field Enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows: (7) The above formula is to add the obtained dynamic field to the anchor feature to achieve feature alignment. For the enhanced anchor feature and reference features , using the inter-frame difference method, calculate the dynamic field of the reference frame and the target frame, given a batch of intermediate feature maps And the corresponding binary image ,in, represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. Represents the number of channels, such as the three RGB channels commonly seen in images. The number of channels in the feature map may vary depending on the design and calculation of the model. It reflects the different feature dimension information in the feature map. Represents the height of the feature map, represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask; First, anchors and reference frames are concatenated at batch level as and , in order to utilize cross-frame information; then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ; Then, for each foreground channel pattern, i.e., “query”, another different foreground feature is randomly selected as the “positive sample”, while all background features in the same batch are used as “negative samples”; finally, the one-step contrast loss is calculated by InfoNCE: (8) Among them, this formula is a specific form of InfoNCE (a contrastive loss function). In contrastive learning, the model learns the similarities and differences between samples. middle, is the query feature, which is the mode of each foreground channel. is the positive sample feature, Represents the dot product of the query feature and the positive sample feature, which is used to measure the similarity between them. The function exponentiates this similarity score, is the temperature hyperparameter that adjusts the sharpness of the distribution, The smaller it is, the sharper the distribution is, and the stricter the distinction between positive and negative samples is. In the above example, all negative sample features ( is the embedding set of negative samples) and the query features The dot product results are exponentialized and summed. The denominator is used to normalize the similarity scores of positive and negative samples. This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrast loss: (9) The above calculation can obtain the calculation result of a sample loss. Repeat the above process and average to get the final loss. By combining the adjacent frame-assisted self-supervised comparative learning method, cross-frame information is effectively used to enhance feature learning and solve the problem of background and foreground distinction in colonoscopy polyp detection.

[0012] (3) Based on the self-guided bounding box matching algorithm, the details are as follows: like Figure 3 As shown in Figure 1, a general overview of the proposed method is provided. A trained target detection model (CenterNet network) is used to perform inference on a set of test images, denoted as ; For each image , and get a set of bounding boxes ; in, , since the number of bounding boxes generated during inference for each image may be different, Representing images The number of bounding boxes generated; in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty; Next, we will describe the method proposed in this paper to influence the label assignment of low-confidence bounding boxes through high-confidence bounding boxes. In the subsequent matching process, it is necessary to compare the feature similarities between bounding boxes; Therefore, for each bounding box Introducing additional feature information , is done by Crop out the bounding box The image block in the image block is obtained, and the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula: (10) It can be understood that through the training of the neural network, an initialized weight network is obtained, and the network predicts a series of detection boxes. The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, denoted as and , set the confidence threshold and ; For categories The bounding box and the confidence score , which is considered as high confidence (HF) Bounding box, similarly, for the category The bounding box and the confidence score , which is considered as high confidence (HF) Bounding Box, Found (HF) and The bounding box will be used as the initial confirmation and Class bounding box; then, from (HF) and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching steps. In order to achieve more efficient and effective matching performance, the classic K-means algorithm is used. All bounding boxes that have not been added to the confirmed bounding box set will be regarded as candidate bounding boxes.

[0013] In order to ensure the reliability of subsequent multiple rounds of EMD matching and bounding box propagation, it is necessary to calculate the distance constraint for each category. Bounding box, calculate the bounding box and other (HF) The shortest Euclidean distance between bounding boxes, for all (HF) After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ; Similarly, this method has an effect on (HF) The same process is performed for the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, will be applied and as a constraint; Specifically, the multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. This is done to ensure that the bounding boxes added in the early stages are more likely to be correctly assigned labels. In the second stage, the constraints are relaxed to allow the bounding boxes to propagate across a wider range of candidate boxes. In each round of propagation, assuming there are a candidate bounding boxes and m confirmed bounding boxes, among these confirmed bounding boxes, t belong to category c1, and the remaining m−t belong to category c2. The input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box; Specifically, Including t confirmed c1 class bounding boxes and m−t confirmed c2 class bounding boxes, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is used to derive and The matching flow between Indicates that its potential value is 0 or 1; The matching optimization goal is: (11) The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the undetermined detection box to the final result. The constraints are: it is understood that a confirmed bounding box can only match with one candidate bounding box at most, and a candidate bounding box can only match with one confirmed bounding box at most. When matching traffic , represents the candidate bounding box Successful and confirmed bounding box match; (12) (13) (14) (15) A confirmed bounding box (whether c1 or c2) can only match with one candidate bounding box at most. Formula (3) shows that a candidate bounding box can only match with one confirmed bounding box at most. , represents the candidate bounding box Successful and confirmed bounding box match; In the first stage of propagation, for each The associated confirmed c1 class bounding box (where ),turn up ,if , the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ), then update the confirmed bounding box set To the new state, the newly added c1 and c2 class bounding boxes are added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching; in, : is the matching flow between the candidate bounding box and the confirmed bounding box, with a value of 0 or 1, 0 means no match, 1 means match. In the first stage of propagation, for the bounding box associated with the confirmed bounding box of c1, the found =1 , calculate its Euclidean shortest distance is ; The second phase of propagation starts when no new bounding boxes have been added to the confirmed sets of classes c1 and c2 in the previous round of propagation, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, that is, any matching pair found in the EMD matching will be used to add the candidate bounding box to the confirmed set of bounding boxes; Specifically, for each The associated confirmed c1 class bounding box (where ), this method finds , so that , and then the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ); The termination condition of multiple rounds of bounding box propagation is when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the set of confirmed bounding boxes are reported as the final detection results, covering all bounding boxes of categories c1 and c2.

[0014] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for detecting colon polyps based on endoscopic images, characterized in that: The steps include: S1, multi-scale feature extraction based on dilated convolution and resnet50 residual network; S2, feature enhancement combined with frame-assisted contrastive learning; S3, prediction based on centernet network; S4. Self-guided bounding box matching in the testing phase.

2. The method for detecting colon polyps based on endoscopic images according to claim 1, characterized in that: The multi-scale feature extraction in S1 is as follows: First, from the perspective of feature richness, a multi-branch convolutional structure is used to extract multiple discriminative semantic information; Secondly, from the perspective of expanding the receptive field, atrous convolution is applied to obtain richer local context information. This method is inspired by RFB-s. The difference is that only two branches of this method use atrous convolution. Each branch performs a 1×1 convolution operation on the input feature map, and preliminarily adjusts the number of channels for subsequent processing. The first branch is a residual structure, which is used to retain the key feature information of small objects and generate equivalent feature maps. The remaining three branches perform cascaded standard convolution operations, and the convolution kernel sizes are 1×3, 3×1, and 3×3, respectively. Atrous convolution layers are added to the two middle branches, so that the extracted feature maps can retain more context information. The mathematical expression is as follows: （1）； （2）； （3）； （4）； in, , , and They represent standard convolution operations, with kernel sizes of 1×1, 1×3, 3×1, and 3×3; represents a dilated convolution operation with a dilation rate of 5; represents the connection operation of the feature map; ⊕ represents the element-by-element addition operation of the feature map; F is the input feature map; , and It represents the output feature map of the first three branches after standard convolution and dilated convolution; Y is the output feature map of FEM.

3. The method for detecting colon polyps based on endoscopic images according to claim 1, characterized in that: The feature enhancement in S2 combined with frame-assisted contrastive learning is as follows: First, the channel attention mechanism is used to temporally align the foreground features; finally, the enhanced anchor features are obtained by combining the attention map with the original features through skip connections, as follows: （5）； in, is the weight that is adaptively adjusted according to the similarity metric, that is, the parameter to be adjusted in the neural network, It is the original anchor feature, which can be understood as the in-frame feature of the colonoscopy polyp detection frame. It is the result of normalizing the adjacent frames. Normalization can be intuitively understood as averaging. Secondly, the background alignment is performed using the inter-frame difference method. and reference features , calculate their inter-frame differences, that is, perform element-by-element subtraction on the enhanced anchor features and the reference features; then, use 1x1 convolution to process the difference results to generate a dynamic field , the specific formula is as follows: （6）； Use 3×3 deformable convolution to transform the background dynamic field Enhanced anchoring features In this way, the anchor features are aligned to obtain the final aligned anchor features. The formula is as follows: （7）； The above formula is to add the obtained dynamic field to the anchor feature. For the enhanced anchor feature and reference features , using the inter-frame difference method, calculate the dynamic field of the reference frame and the target frame, given a batch of intermediate feature maps And the corresponding binary image , in, represents the batch size, that is, the number of samples used in one training. Represents the number of frames, because we are processing video-related features, and the video consists of multiple frames. Represents the number of channels, such as the three RGB channels commonly seen in images. The number of channels in the feature map may vary depending on the design and calculation of the model. It reflects the different feature dimension information in the feature map. Represents the height of the feature map, represents the width of the feature map, is the foreground binary image, It is the background binary image, which is also called mask; First, anchors and reference frames are concatenated at batch level as and ; Then, based on Extracting cross-frame features The foreground and background channel modes correspond to and ; Then, for each foreground channel pattern, i.e., "query", another different foreground feature is randomly selected as the "positive sample", and all background features in the same batch are used as "negative samples"; finally, the one-step contrast loss is calculated by InfoNCE: （8）； Among them, this formula is a specific form of InfoNCE (a contrastive loss function). In contrastive learning, the model learns the similarities and differences between samples. middle, is the query feature, which is the mode of each foreground channel. is the positive sample feature, Represents the dot product of the query feature and the positive sample feature, which is used to measure the similarity between them. The function exponentiates this similarity score, is the temperature hyperparameter that adjusts the sharpness of the distribution, The smaller it is, the sharper the distribution is, and the stricter the distinction between positive and negative samples is. In the above example, all negative sample features ( is the embedding set of negative samples) and the query features The dot product results are exponentialized and summed. The denominator is used to normalize the similarity scores of positive and negative samples. This process is repeated until each foreground channel mode is selected, and the losses of all steps are summed to obtain the final contrast loss: （9）。 4. The method for detecting colon polyps based on endoscopic images according to claim 1, characterized in that: The prediction in S3 is based on the centernet network, as follows: Use a trained centernet network target detection model to infer a set of test images, denoted as ; For each image , and get a set of bounding boxes ; in, , since the number of bounding boxes generated during inference for each image may be different, Representing images The number of bounding boxes generated; in, Represents the position attribute of the corresponding bounding box, is the coordinate of the upper left corner, Represents its width and height respectively, represents the class label of the bounding box, Represents its confidence score, ranging from 0 to 1, A high value of indicates that the model has strong confidence in the class label assignment, while a low value indicates uncertainty.

5. The method for detecting colon polyps based on endoscopic images according to claim 1, characterized in that: The self-guided bounding box matching in the test phase of S4 is as follows: The label assignment of the low-confidence bounding box is affected by the high-confidence bounding box. In the subsequent matching process, the feature similarity between the bounding boxes needs to be compared; Therefore, for each bounding box Introducing additional feature information , is done by Crop out the bounding box The image block in the image block is obtained, and the corresponding feature information is obtained, and the feature extractor is recorded as , It can be expressed by the following formula: （10）； in, The in-box features of the preliminary prediction results, self-guided bounding box propagation takes all bounding boxes as input, and its task is to reassign the category labels of low-confidence bounding boxes. To simplify the explanation, assume that there are two categories, denoted as and , set the confidence threshold and ; For categories The bounding box and the confidence score , which is considered as high confidence (HF) Bounding box, similarly, for the category The bounding box and the confidence score , which is considered as high confidence (HF) Bounding Box, Found (HF) and The bounding box will be used as the initial confirmation and Class bounding box; then, from (HF) and A set of representative bounding boxes are selected from the bounding boxes for subsequent multiple rounds of EMD matching steps. The classic K-means algorithm is used, and all bounding boxes that have not been added to the confirmed bounding box set are considered candidate bounding boxes. Calculate the distance constraint for each category, for each (HF) Bounding box, calculate the bounding box and other (HF) The shortest Euclidean distance between bounding boxes, for all (HF) After this process, the average of all the shortest distances is calculated as the category The distance constraint is denoted as ; Similarly, for (HF) The same process is performed for the bounding box to obtain , in the initial stage of multiple rounds of EMD matching, will be applied and as a constraint; The multi-round bounding box propagation consists of two stages. In the first stage, stricter constraints are imposed on the newly confirmed bounding boxes during the propagation process. In the second stage, the constraints are relaxed. In each round of propagation, assuming there are a candidate bounding boxes and m confirmed bounding boxes, among these confirmed bounding boxes, t belong to category c1, and the remaining m−t belong to category c2. The input of EMD consists of two parts: and ,in represents a candidate bounding box, Represents the confirmed bounding box; Specifically, Including t confirmed c1 class bounding boxes and m−t confirmed c2 class bounding boxes, and The distance between , reflecting the candidate bounding box With confirmed box The cost of matching, By calculation and The Euclidean distance between features is used to derive and The matching flow between Indicates that its potential value is 0 or 1; The matching optimization goal is: （11）； The above formula is the objective function of the final self-guided bounding box operation, that is, to minimize the cost of adding the undetermined detection box to the final result. The constraints are: it is understood that a confirmed bounding box can only match with one candidate bounding box at most, and a candidate bounding box can only match with one confirmed bounding box at most. When matching traffic , represents the candidate bounding box Successful and confirmed bounding box match; （12）； （13）； （14）； （15）； A confirmed bounding box (whether c1 or c2) can only match with one candidate bounding box at most. Formula (3) shows that a candidate bounding box can only match with one confirmed bounding box at most. , represents the candidate bounding box Successful and confirmed bounding box match; In the first stage of propagation, for each The associated confirmed c1 class bounding box (where ),turn up ,if , the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ), then update the confirmed bounding box set To the new state, the newly added c1 and c2 class bounding boxes are added, and the remaining candidate bounding boxes continue to enter the next round of EMD matching; in, : is the matching flow between the candidate bounding box and the confirmed bounding box, with a value of 0 or 1, 0 means no match, 1 means match. In the first stage of propagation, for the bounding box associated with the confirmed bounding box of c1, the found =1 , calculate its Euclidean shortest distance is ; The second phase of propagation starts when no new bounding boxes have been added to the confirmed sets of classes c1 and c2 in the previous round of propagation, so and The constraints are relaxed to allow more bounding boxes to be added to the confirmed set, and any matching pairs found in the EMD matching are used to add the candidate bounding box to the confirmed set of bounding boxes; For each The associated confirmed c1 class bounding box (where ),turn up , so that , and then the corresponding candidate bounding box Add the confirmed c1 class bounding box set, and the same process applies to The associated c2 class bounding box (where ); The termination condition of multiple rounds of bounding box propagation is: when the number of candidate bounding boxes in P is 0; at this time, all bounding boxes in the set of confirmed bounding boxes are reported as the final detection results, covering all bounding boxes of categories c1 and c2.

6. A colon polyp detection device based on endoscopic imaging, characterized in that: The colon polyp detection device performs colon polyp detection using the method described in any one of claims 1-5.

Images