Colonoscope video key frame extraction method, apparatus and device, and storage medium

Abstract

The invention relates to the technical field of video key frame extraction, and particularly discloses a colonoscope video key frame extraction method and device, equipment and a storage medium, and the method comprises the steps: carrying out the detection of a colonoscope video frame sequence artifact region; a ResNet50 model is adopted to detect polyp frames, colonoscope video frames containing polyps are sent into a similarity analysis network composed of two VGG16 sub-networks sharing weights, similarity scores between the video frames are calculated, and grouping is carried out according to the similarity scores; inputting the colonoscope video frame sequence into an iAFF-HSFYOLO network to carry out polyp positioning; and introducing a comprehensive evaluation mechanism according to the artifact region area ratio, the polyp region area ratio, the distance of the polyp close to the center and the confidence score, and selecting a key frame from each group of similar video frames to complete key frame extraction. According to the method, the polyp detection model, the ResNet architecture, the similarity analysis network and the attention mechanism are combined, so that the examination burden of doctors can be remarkably reduced, and the detection accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video key frame extraction, and in particular to a method, device, equipment and storage medium for extracting key frames from a colonoscopy video. Background Art

[0002] The collection of colonoscopy videos is of great significance in the diagnosis, retrospective analysis and comprehensive examination of colorectal cancer. Endoscopists need to find pathological features such as lesions, polyps, erosions, and ulcers from the complete colonoscopy video. Experienced endoscopists need to make professional and careful judgments to identify the location, nature and morphology of the lesions. However, traditional methods rely on the professional experience of endoscopists and are easily affected by factors such as bubbles, feces, and motion blur. Endoscopists may also miss detections due to fatigue, stress and other interferences during long-term work. Therefore, finding a key frame extraction method to extract key information in colonoscopy is very important to improve the diagnostic efficiency of doctors and reduce the diagnostic burden of doctors.

[0003] Existing keyframe extraction methods for colonoscopy videos usually rely only on manually set features, such as threshold setting, entropy-based frame quality assessment, and clustering-based keyframe selection. These methods do not fully consider the basic properties of the colonoscopy video itself, nor do they consider the impact of artifacts (bubbles, feces, reflections, motion blur, etc.) in colonoscopy on the colonoscopy video. Summary of the invention

[0004] In order to solve the problems existing in the prior art, the present invention provides a colonoscopy video key frame extraction method, device, equipment and storage medium for automated colonoscopy video key frame extraction, including multiple modules, each module working together to achieve efficient and accurate polyp detection and key frame extraction, solving the problems mentioned in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a colonoscopy video key frame extraction method, comprising the following steps:

[0006] S1. Artifact area detection in colonoscopy video frame sequence:

[0007] S2, using the ResNet50 model to detect polyp frames, using the cross entropy loss function and the Adam optimizer to train the model, inputting the colonoscopy video frame sequence retained after being deleted in step S1 into the trained ResNet50 model, identifying the video frames containing polyps, and discarding frames without information;

[0008] S3, sending the retained colonoscopy video frames containing polyps to a similarity analysis network composed of two VGG16 sub-networks with shared weights, calculating the similarity scores between the video frames, and grouping them according to the similarity scores;

[0009] S4, inputting the colonoscopy video frame sequence of each group into the iAFF-HSFYOLO network for polyp localization;

[0010] S5. Based on the polyp localization results of colonoscopy video frames, a comprehensive evaluation mechanism is introduced according to the area ratio of the artifact area, the area ratio of the polyp area, the distance of the polyp from the center and the confidence score. Key frames are selected from each group of similar video frames in the grouping results to complete the key frame extraction.

[0011] Preferably, in step S1, it specifically includes:

[0012] S11. Use the EndoCV2020 dataset to build a video frame artifact detection dataset and train a video frame artifact detection model based on the EndoNet model;

[0013] S12, inputting the colonoscopy video frame to be processed into the artifact detection model EndoNet, calculating the area of ​​the artifact detection box and its proportion in the image through the model, and deleting the video frames with the proportion of the artifact area exceeding the threshold.

[0014] Preferably, in step S11, the video frame artifact detection model based on the EndoNet model uses ResNet as a feature extraction skeleton, extracts multi-scale features through residual connections, and uses a feature pyramid network FPN to fuse features at different levels; the model predicts the bounding box and category label of the artifact respectively through a regression head and a classification head, and the detection process generates multiple candidate boxes on each feature layer. After classification and regression, the artifact detection box is screened out and deduplication is performed through non-maximum suppression NMS.

[0015] Preferably, in step S3, the similarity analysis network is composed of two VGG16 sub-networks with shared weights, each sub-network is based on the VGG16 architecture, the average pooling layer and the classifier are removed, and only the convolution layer for feature extraction is retained;

[0016] The two adjacent frames of the retained colonoscopy video containing polyps are respectively extracted with the VGG16 sub-network, and then their L1 distance is calculated. The feature difference is compared by the L1 distance between the two feature vectors.

[0017] L1=∑|x1 i -x2 i |

[0018] Among them, x1 i and x2 i They represent the elements in the feature vectors of two adjacent frames respectively. The obtained feature differences are processed by two levels of fully connected layers. The first fully connected layer has 512 neurons, and the second fully connected layer outputs the similarity score;

[0019] The similarity score is converted into a probability value between 0 and 1 through the Sigmoid function. A higher probability value indicates a high similarity. The calculation formula is as follows:

[0020]

[0021] Among them, s is the original similarity score generated by the similarity analysis network for the input adjacent frames, and the result of grouping the frame sequence is obtained based on the similarity score.

[0022] Preferably, in step S4, the iAFF-HSFYOLO network specifically adopts the HSFPN module to replace the SPPF layer in the YOLOv8 backbone network, and at the same time, introduces the iterative attention feature fusion module iAFF in the feature fusion stage;

[0023] In the HSFPN module, the input feature maps are subjected to adaptive average pooling and adaptive maximum pooling operations to generate global average features and maximum features; these feature maps are then reduced in dimension by a 1x1 convolutional layer conv1, and after passing through the ReLU activation function ReLU, they are then restored to the number of channels by another 1x1 convolutional layer conv2; the processed average features and maximum features are added and fused to generate a comprehensive feature map; the comprehensive feature map is used to generate an attention weight map through the Sigmoid activation function; finally, the weight map is dot-multiplied with the original input feature map to generate the final feature representation;

[0024] In the iterative attention feature fusion module iAFF, local and global contextual information are combined to generate weighted feature maps; by dynamically adjusting the weights of the input features, the model can better capture and locate polyps of different scales; finally, the enhanced feature maps are processed by the model's detection head to generate high-precision polyp localization results.

[0025] Preferably, in step S5, the calculation formula of the comprehensive evaluation mechanism is as follows:

[0026] Score = ω1·(1-A artifact )+ω2·A polyp +ω3·(1-D center )+ω4·C confidence

[0027] Among them, ω1, ω2, ω3 and ω4 are weight coefficients, A artifact Represents the area ratio of the artifact area, A polyp represents the polyp area ratio, D center represents the distance of the polyp from the center, and C confidence Represents the confidence score.

[0028] On the other hand, in order to achieve the above-mentioned purpose, the present invention also provides the following technical solution: a colonoscopy video key frame extraction device, comprising the following modules:

[0029] Artifact region detection module, colonoscopy video frame sequence artifact region detection;

[0030] The polyp detection module uses the ResNet50 model to detect polyp frames, uses the cross entropy loss function and the Adam optimizer to train the model, inputs the colonoscopy video frame sequence retained after deletion in step S1 into the trained ResNet50 model, identifies the video frames containing polyps, and discards the frames without information;

[0031] The similarity analysis module sends the retained colonoscopy video frames containing polyps to a similarity analysis network composed of two VGG16 sub-networks with shared weights, calculates the similarity scores between the video frames, and groups them according to the similarity scores;

[0032] The polyp localization module inputs the colonoscopy video frame sequence of each group into the iAFF-HSFYOLO network for polyp localization;

[0033] The key frame extraction module, based on the polyp localization results of colonoscopy video frames, introduces a comprehensive evaluation mechanism according to the area ratio of artifact area, the area ratio of polyp area, the distance of polyp to the center and the confidence score, selects key frames from each group of similar video frames in the grouping results, and completes the key frame extraction.

[0034] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: an electronic device, the electronic device comprising: a processor; and a memory for storing one or more programs;

[0035] When the one or more programs are executed by the processor, the processor executes the colonoscopy video key frame extraction method.

[0036] On the other hand, to achieve the above-mentioned purpose, the present invention also provides the following technical solution: a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the colonoscopy video key frame extraction method when executed by a processor.

[0037] The beneficial effects of the present invention are: the colonoscopy video key frame extraction method of the present invention is used to automatically extract key frames in colonoscopy videos. Combined with the polyp detection model, ResNet architecture, similarity analysis network, and attention mechanism, the method can significantly reduce the doctor's examination burden, improve the accuracy and efficiency of detection, and has broad clinical application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Schematic diagram of the steps of the colonoscopy video key frame extraction method in an embodiment of the present invention;

[0039] Figure 2 Schematic diagram of a module of a colonoscopy video key frame extraction device in an embodiment of the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present invention;

[0041] In the figure, 110 - artifact area detection module; 120 - polyp detection module; 130 - similarity analysis module; 140 - polyp positioning module; 150 - key frame extraction module; 210 - processor; 220 - storage. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1-Figure 3 ,The present invention provides a technical solution: a colonoscopy video key frame extraction method, such as Figure 1 As shown, the following steps are included:

[0044] S1. Artifact region detection in colonoscopy video frame sequence

[0045] The EndoCV2020 dataset was used to construct a video frame artifact detection dataset, and a video frame artifact detection model based on the EndoNet model was trained. The model uses ResNet as the feature extraction skeleton, extracts multi-scale features through residual connections, and uses the feature pyramid network (FPN) to fuse features at different levels to enhance the accuracy of artifact detection. The model uses regression and classification heads to predict the bounding box and category label of the artifact respectively. The detection process generates multiple candidate boxes on each feature layer, and the artifact detection box is screened out after classification and regression, and deduplication is performed through non-maximum suppression (NMS). Finally, the model calculates the area of ​​the artifact detection box and its proportion in the image, and deletes video frames with an artifact area proportion higher than the threshold. That is, frames with poor quality are screened out, and frames with high quality and clinical diagnostic value are retained.

[0046] S2. Use the ResNet50 model to detect polyp frames, use the cross entropy loss function and Adam optimizer to train the model, input the colonoscopy video frame sequence retained after deletion in step S1 into the trained ResNet50 model, identify the video frames containing polyps, and discard the frames without information.

[0047] S3. The retained colonoscopy video frames containing polyps are sent to a similarity analysis network consisting of two VGG16 sub-networks with shared weights, the similarity scores between the video frames are calculated, and the frames are grouped according to the similarity scores.

[0048] The similarity analysis network consists of two VGG16 sub-networks with shared weights. Each sub-network is based on the VGG16 architecture, with the average pooling layer and classifier removed, leaving only the convolutional layer for feature extraction.

[0049] The two adjacent frames of the retained colonoscopy video containing polyps are respectively extracted with the VGG16 sub-network, and then their L1 distance is calculated. The feature difference is compared by the L1 distance between the two feature vectors.

[0050] L1=∑|x1 i -x2 i |

[0051] Among them, x1 i and x2 i They represent the elements in the feature vectors of two adjacent frames respectively. The obtained feature differences are processed by two levels of fully connected layers. The first fully connected layer has 512 neurons, and the second fully connected layer outputs the similarity score;

[0052] The similarity score is converted into a probability value between 0 and 1 through the Sigmoid function. A higher probability value indicates a high similarity. The calculation formula is as follows:

[0053]

[0054] Among them, s is the original similarity score generated by the similarity analysis network for the input adjacent frames, and the result of grouping the frame sequence is obtained based on the similarity score.

[0055] S4. Input the colonoscopy video frame sequence of each group into the iAFF-HSFYOLO network for polyp localization.

[0056] In step S4, after steps S1, S2 and S3, high-quality colonoscopy video frames are obtained, and the iAFF-HSFYOLO model is used to locate polyps in colonoscopy videos. SPPF extracts spatial pyramid features through maximum pooling operations, but this method may lose some fine feature information. In order to reduce redundant features, combine high-level semantic information, and better capture the details of polyps of different sizes, the iAFF-HSFYOLO network specifically replaces the SPPF layer in the YOLOv8 backbone network with the HSFPN module to improve the accuracy and sensitivity of polyp positioning. At the same time, the iterative attention feature fusion module (iAFF) is introduced in the feature fusion stage. By combining local and global context information, a weighted feature map is generated to further enhance the fusion effect of multi-scale features. The HSFPN (Hierarchical Screening-Feature Pyramid Network) module optimizes the model's detection ability for polyps of different scales through channel attention mechanism and feature fusion.

[0057] In the HSFPN module, the input feature map is subjected to adaptive average pooling and adaptive max pooling respectively to generate global average features and maximum features; these feature maps are then reduced in dimension by a 1x1 convolutional layer conv1, and after passing through the ReLU activation function ReLU, they are restored to the number of channels by another 1x1 convolutional layer conv2; the processed average features and maximum features are added and fused to generate a comprehensive feature map; the comprehensive feature map is used to generate an attention weight map by the Sigmoid activation function; finally, the weight map is dot-multiplied with the input original feature map to generate the final feature representation. This adaptive feature selection and fusion strategy enables the HSFPN module to effectively improve the performance of the model in polyp detection tasks, especially when dealing with polyps of different sizes and shapes, providing more refined and accurate feature expressions, and improving the overall detection accuracy and robustness.

[0058] In the iterative attention feature fusion module iAFF, local and global context information is combined to generate weighted feature maps; by dynamically adjusting the weights of input features, the model can better capture and locate polyps of different scales; finally, the enhanced feature maps are processed by the model's detection head to generate high-precision polyp localization results, significantly improving the model's detection performance in complex scenes.

[0059] In the specific implementation, the input feature maps x and y are first added by principal element to generate a fused feature map x a . Then, the initial fusion feature map x aAfter being processed by the local attention module (LA) and the global attention module (GA), a local feature map x is generated. l and the global feature map x g The first round of attention weight map m1 is generated by the following calculation.

[0060] m1=σ(LA(x a )+GA(x a ))

[0061] The calculation formulas of LA and GA are:

[0062] LA(x a )=BN(Conv2(SiLU(BN(Conv1(x a )))))

[0063] GA(x a )=BN(Conv2(SiLU(BN(Conv1(GAP(x a ))))))

[0064] Where GAP is global average pooling. The initial feature maps x and y are weighted combined through the attention weight map m1. The intermediate fusion feature map x is generated. u :

[0065] x u =x·m1+y·(1-m1)

[0066] In the second round of fusion, the intermediate fusion feature map x u Again through the local and global attention modules, a second round of attention weight map m2 is generated. These enhanced feature maps are processed by the detection head of the model to generate high-precision polyp localization results.

[0067] S5. Based on the polyp localization results of colonoscopy video frames, a comprehensive evaluation mechanism is introduced according to the area ratio of the artifact area, the area ratio of the polyp area, the distance of the polyp from the center and the confidence score. Key frames are selected from each group of similar video frames in the grouping results to complete the key frame extraction.

[0068] The comprehensive evaluation mechanism calculates the final score of each cluster based on the following four parameters: artifact area ratio (A artifact ), polyp area ratio (A polyp ), the distance to the center (D center ) and confidence score (C confidence ). The calculation formula is as follows:

[0069] Score = ω1·(1-A artifact )+ω2·A polyp +ω3·(1-Dcenter )+ω4·C confidence

[0070] Among them, ω1, ω2, ω3 and ω4 are weight coefficients, which can be set to ω1 = 0.25, ω2 = 0.35, ω3 = 0.15 and ω4 = 0.25, which can be adjusted according to the needs of actual applications to optimize the selection effect of key frames; A artifact Represents the area ratio of the artifact area, A polyp represents the polyp area ratio, D center represents the distance of the polyp from the center, and C confidence Represents the confidence score.

[0071] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides a colonoscopy video key frame extraction device, which can implement the functions provided by the above method embodiment, such as Figure 2 As shown, the device includes the following modules:

[0072] Artifact region detection module 110, colonoscopy video frame sequence artifact region detection;

[0073] The polyp detection module 120 uses a ResNet50 model to detect polyp frames, uses a cross entropy loss function and an Adam optimizer to train the model, inputs the colonoscopy video frame sequence retained after being deleted in step S1 into the trained ResNet50 model, identifies video frames containing polyps, and discards frames without information;

[0074] The similarity analysis module 130 sends the retained colonoscopy video frames containing polyps to a similarity analysis network composed of two VGG16 sub-networks with shared weights, calculates similarity scores between the video frames, and groups them according to the similarity scores;

[0075] The polyp localization module 140 inputs the colonoscopy video frame sequence of each group in the grouping into the iAFF-HSFYOLO network for polyp localization;

[0076] The key frame extraction module 150 is based on the polyp localization results of the colonoscopy video frame, introduces a comprehensive evaluation mechanism according to the area ratio of the artifact area, the area ratio of the polyp area, the distance of the polyp from the center and the confidence score, selects key frames from each group of similar video frames in the grouping results, and completes the key frame extraction.

[0077] Based on the same inventive concept as the above method embodiment, the present application embodiment also provides an electronic device, such as Figure 3 As shown, the device includes: a processor 210; and a memory 220, for storing one or more programs;

[0078] When the one or more programs are executed by the processor 210, the processor is caused to execute the colonoscopy video key frame extraction method.

[0079] The colonoscopy video key frame extraction method specifically includes the following steps:

[0080] Artifact region detection in colonoscopy video frame sequence:

[0081] The ResNet50 model was used to detect polyp frames. The cross entropy loss function and Adam optimizer were used to train the model. The retained colonoscopy video frame sequence was input into the trained ResNet50 model to identify the video frames containing polyps and discard the frames without information.

[0082] The retained colonoscopy video frames containing polyps are sent to a similarity analysis network consisting of two VGG16 sub-networks with shared weights to calculate the similarity scores between the video frames and group them according to the similarity scores;

[0083] The colonoscopy video frame sequence of each group in the grouping was input into the iAFF-HSFYOLO network for polyp localization;

[0084] Based on the polyp localization results of colonoscopy video frames, a comprehensive evaluation mechanism was introduced according to the area ratio of artifact area, area ratio of polyp area, distance of polyp to the center and confidence score. Key frames were selected from each group of similar video frames in the grouping results to complete key frame extraction.

[0085] Based on the same inventive concept as the above method embodiment, the embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by the processor 210, the colonoscopy video key frame extraction method is implemented.

[0086] The colonoscopy video key frame extraction method specifically includes the following steps:

[0087] Artifact region detection in colonoscopy video frame sequence:

[0088] The ResNet50 model was used to detect polyp frames. The cross entropy loss function and Adam optimizer were used to train the model. The retained colonoscopy video frame sequence was input into the trained ResNet50 model to identify the video frames containing polyps and discard the frames without information.

[0089] The retained colonoscopy video frames containing polyps are sent to a similarity analysis network consisting of two VGG16 sub-networks with shared weights to calculate the similarity scores between the video frames and group them according to the similarity scores;

[0090] The colonoscopy video frame sequence of each group in the grouping was input into the iAFF-HSFYOLO network for polyp localization;

[0091] Based on the polyp localization results of colonoscopy video frames, a comprehensive evaluation mechanism was introduced according to the area ratio of artifact area, area ratio of polyp area, distance of polyp to the center and confidence score. Key frames were selected from each group of similar video frames in the grouping results to complete key frame extraction.

[0092] In several embodiments provided in the embodiments of the present invention, it should be understood that the disclosed apparatus and method can also be implemented in other ways. The apparatus and method embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the apparatus, method and computer program product according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0093] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0094] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions to enable a computer device (which can be a personal computer, an electronic device, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, RandomAccess Memory), a disk or an optical disk. It should be noted that, in this article, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0095] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0096] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0097] The word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0098] The "first\second" mentioned in the embodiments is only to distinguish similar objects, and does not represent a specific order for the objects. It is understandable that the "first\second" can be interchanged with the specific order or sequence where permitted. It should be understood that the objects distinguished by "first\second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than those illustrated or described herein.

[0099] Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A colonoscopy video key frame extraction method, characterized in that: The steps include: S1, artifact region detection in colonoscopy video frame sequence; S2, using the ResNet50 model to detect polyp frames, using the cross entropy loss function and the Adam optimizer to train the model, inputting the colonoscopy video frame sequence retained after being deleted in step S1 into the trained ResNet50 model, identifying the video frames containing polyps, and discarding frames without information; S3, sending the retained colonoscopy video frames containing polyps to a similarity analysis network composed of two VGG16 sub-networks with shared weights, calculating the similarity scores between the video frames, and grouping them according to the similarity scores; S4, inputting the colonoscopy video frame sequence of each group into the iAFF-HSFYOLO network for polyp localization; S5. Based on the polyp localization results of colonoscopy video frames, a comprehensive evaluation mechanism is introduced according to the area ratio of the artifact area, the area ratio of the polyp area, the distance of the polyp from the center and the confidence score. Key frames are selected from each group of similar video frames in the grouping results to complete the key frame extraction.

2. The colonoscopy video key frame extraction method according to claim 1, characterized in that: In step S1, it specifically includes: S11. Use the EndoCV2020 dataset to build a video frame artifact detection dataset and train a video frame artifact detection model based on the EndoNet model; S12, inputting the colonoscopy video frame to be processed into the artifact detection model EndoNet, calculating the area of ​​the artifact detection box and its proportion in the image through the model, and deleting the video frames with the proportion of the artifact area exceeding the threshold.

3. The colonoscopy video key frame extraction method according to claim 2, characterized in that: In step S11, the video frame artifact detection model based on the EndoNet model uses ResNet as the feature extraction skeleton, extracts multi-scale features through residual connections, and uses the feature pyramid network FPN to fuse features at different levels; the model predicts the bounding box and category label of the artifact respectively through the regression head and the classification head. The detection process generates multiple candidate boxes on each feature layer. After classification and regression, the artifact detection box is screened out and deduplication is performed through non-maximum suppression NMS.

4. The colonoscopy video key frame extraction method according to claim 1, characterized in that: In step S3, the similarity analysis network consists of two VGG16 sub-networks with shared weights. Each sub-network is based on the VGG16 architecture, with the average pooling layer and classifier removed, leaving only the convolutional layer for feature extraction. The two adjacent frames of the retained colonoscopy video containing polyps are respectively extracted with the VGG16 sub-network, and then their L1 distance is calculated. The feature difference is compared by the L1 distance between the two feature vectors. L1=∑|x1 i -x2 i | Among them, x1 i and x2 i They represent the elements in the feature vectors of two adjacent frames respectively. The obtained feature differences are processed by two levels of fully connected layers. The first fully connected layer has 512 neurons, and the second fully connected layer outputs the similarity score; The similarity score is converted into a probability value between 0 and 1 through the Sigmoid function. A higher probability value indicates a high similarity. The calculation formula is as follows: Among them, s is the original similarity score generated by the similarity analysis network for the input adjacent frames, and the result of grouping the frame sequence is obtained based on the similarity score.

5. The colonoscopy video key frame extraction method according to claim 1, characterized in that: In step S4, the iAFF-HSFYOLO network specifically adopts the HSFPN module to replace the SPPF layer in the YOLOv8 backbone network, and at the same time, introduces the iterative attention feature fusion module iAFF in the feature fusion stage; In the HSFPN module, the input feature map undergoes adaptive average pooling and adaptive maximum pooling operations to generate global average features and maximum features; These feature maps are then processed by a 1x1 convolutional layer conv1 for dimensionality reduction, and after passing through the ReLU activation function ReLU, they are then passed through another 1x1 convolutional layer conv2 to restore the number of channels; the processed average features and maximum features are added and fused to generate a comprehensive feature map; the comprehensive feature map is used to generate an attention weight map through the Sigmoid activation function; finally, the weight map is dot-multiplied with the original input feature map to generate the final feature representation; In the iterative attention feature fusion module iAFF, local and global context information are combined to generate a weighted feature map; By dynamically adjusting the weights of the input features, the model is able to better capture and locate polyps of different scales; ultimately, the enhanced feature map is processed by the model's detection head to generate high-precision polyp localization results.

6. The colonoscopy video key frame extraction method according to claim 1, characterized in that: In step S5, the calculation formula of the comprehensive evaluation mechanism is as follows: Score=ω1·(1-A artifact )+ω2·A polyp +ω3·(1-D center )+ω4·C confidence Among them, ω1, ω2, ω3 and ω4 are weight coefficients, A artifact Represents the area ratio of the artifact area, A polyp represents the polyp area ratio, D center represents the distance of the polyp from the center, and C confidence Represents the confidence score.

7. A key frame extraction device according to the colonoscopy video key frame extraction method according to any one of claims 1 to 6, characterized in that: Includes the following modules: Artifact region detection module (110), colonoscopy video frame sequence artifact region detection; A polyp detection module (120) uses a ResNet50 model to detect polyp frames, uses a cross entropy loss function and an Adam optimizer to train the model, inputs the colonoscopy video frame sequence retained after being deleted in step S1 into the trained ResNet50 model, identifies video frames containing polyps, and discards frames without information; A similarity analysis module (130) sends the retained colonoscopy video frames containing polyps to a similarity analysis network composed of two VGG16 sub-networks with shared weights, calculates similarity scores between the video frames, and groups them according to the similarity scores; A polyp localization module (140) inputs the colonoscopy video frame sequence of each group in the grouping into the iAFF-HSFYOLO network for polyp localization; The key frame extraction module (150) introduces a comprehensive evaluation mechanism based on the polyp positioning results of the colonoscopy video frame according to the area ratio of the artifact area, the area ratio of the polyp area, the distance of the polyp from the center and the confidence score, selects key frames from each group of similar video frames in the grouping results, and completes the key frame extraction.

8. An electronic device, characterized in that: The electronic device comprises: a processor (210); and a memory (220) for storing one or more programs; When the one or more programs are executed by the processor (210), the processor is caused to execute the colonoscopy video key frame extraction method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by the processor (210), the method for extracting key frames from a colonoscopy video as described in any one of claims 1 to 6 is implemented.

Images