A visual slam front-end feature collection and semantic association method and system

Abstract

The application discloses a visual SLAM front-end feature acquisition and semantic association method and system for bronchoscope polyp perception. The method comprises the following steps: acquiring a monocular bronchoscope image frame, obtaining a frame-level detection mark and a candidate region through a polyp detection network; performing segmentation on the image in which the polyp is detected and generating a binary mask under the constraint of the candidate region, screening ORB features according to the mask, and obtaining a polyp-related two-dimensional feature subset; projecting three-dimensional map points in visual SLAM to the image plane according to the current camera pose, giving a semantic label according to the spatial membership relationship between the projected points and the binary mask, and realizing the association between the two-dimensional lesion features and the three-dimensional map points. The image in which the polyp is not detected is directly subjected to regular ORB-SLAM front-end processing. The application introduces polyp semantic information while maintaining the original geometric tracking process, reduces unnecessary segmentation calculation of the polyp-free frame, and provides a data basis for lesion perception mapping and bronchoscope navigation.

Description

Technical Field

[0001] This application relates to the fields of medical endoscopic image processing, computer vision and simultaneous localization mapping technology, and in particular to a method and system for integrating polyp detection, polyp segmentation, mask-constrained ORB feature selection and two-dimensional-three-dimensional semantic association in the front end of a monocular bronchoscopy visual SLAM. Background Technology

[0002] Bronchoscopic robots can be used for airway examination, lesion localization, and interventional procedures. Preoperative computed tomography (CT) scans can provide a static prior knowledge of airway anatomy, but they are difficult to reflect the real-time changes in the appearance of the bronchial lumen and the state of lesions during surgery. Visual SLAM (Simultaneous Localization and Mapping) based on monocular endoscopic images can estimate camera pose and reconstruct the local environment online, and has good application potential because it does not rely on additional localization sensors.

[0003] Existing ORB-SLAM methods mainly revolve around geometric feature extraction, matching, tracking, and map point maintenance. In bronchoscopic scenarios with weak textures, significant lighting variations, specular reflections, and unstable observations, although the geometric front end can extract scattered ORB (Oriented Fast and Rotated BRIEF) features, it usually cannot distinguish whether the features originate from polyp regions, thus failing to directly form a lesion-oriented two-dimensional feature set and three-dimensional semantic representation.

[0004] Deep learning-based object detection can quickly identify candidate lesion regions at the image level, and pixel-level segmentation can further delineate lesion boundaries. However, existing detection and segmentation methods are mostly used as independent two-dimensional image analysis modules, separate from feature acquisition and map point processing in the visual SLAM front-end. Performing complete segmentation on every frame would also introduce unnecessary computational overhead for image frames without polyps.

[0005] Therefore, a front-end processing method is needed that can activate segmentation branches on demand using frame-level detection results, constrain ORB feature acquisition using polyp masks, and associate two-dimensional polyp information with visual SLAM three-dimensional map points, so as to obtain lesion perception capability while retaining the original geometric tracking mechanism. Summary of the Invention

[0006] In view of this, the purpose of this application is to provide a method and system for feature acquisition and semantic association of visual SLAM front-end for bronchoscope polyp detection, which solves the problems of existing visual SLAM front-ends lacking explicit lesion semantics, difficulty in corresponding two-dimensional detection and segmentation results with three-dimensional map points, and repeated semantic segmentation of polyp-free image frames.

[0007] Based on the above objectives, this application proposes a visual SLAM front-end feature acquisition and semantic association method for bronchoscopic polyp detection, including: Step S1: Acquire the monocular bronchoscopy image frame at the current moment, input the image frame into the polyp detection network, and output the frame-level detection flag and polyp candidate region of the current image frame from the polyp detection network.

[0008] Step S2: Perform branch gating based on frame-level detection flags. For image frames containing polyps, perform polyp segmentation under polyp candidate region constraints and generate a binary polyp mask; for image frames without polyps, do not activate segmentation branches and directly perform preset ORB-SLAM front-end processing.

[0009] Step S3: Extract ORB features from the current image frame. By determining whether the image coordinates of each ORB feature are located within the binary polyp mask, a subset of polyp-related two-dimensional features is obtained from the full image ORB feature set.

[0010] Step S4: Obtain the 3D map points observed in the current image frame in the visual SLAM map, and project the 3D map points onto the image plane according to the current camera pose and camera projection model; by determining whether the projected coordinates are located within the binary polyp mask, assign polyp semantic labels to the current map point observation.

[0011] Step S5: Provide the polyp-related 2D feature subset and 3D map point observation results with polyp semantic labels to the subsequent semantic map construction, lesion localization or bronchoscopy navigation module, while maintaining the visual SLAM geometric tracking and map point maintenance process.

[0012] This application also provides a system for performing the above method, the system comprising: The image acquisition module is used to acquire monocular bronchoscopy image frames; The polyp detection and gating module is used to output frame-level detection flags and polyp candidate regions, and select semantic processing branches or preset ORB-SLAM front-end branches based on the frame-level detection flags. The segmentation and masking module is used to generate binary polyp masks under the constraint of polyp candidate regions; The feature filtering module is used to obtain a subset of polyp-related two-dimensional features from the ORB feature set based on the binary polyp mask; Visual SLAM geometric front end, used to perform camera pose estimation and 3D map point maintenance; The 2D-3D association module is used to project 3D map points onto the image plane and generate polyp semantic labels based on the spatial membership relationship between the projected points and the binary polyp mask.

[0013] Compared with existing technologies, this application has the following advantages: By constructing a visual SLAM front-end framework that integrates polyp detection, detection-guided segmentation, binary mask-constrained ORB feature selection, and 2D-3D semantic association, the segmentation and semantic processing branches are gated using frame-level polyp detection results. Polyp segmentation and mask generation are performed only on image frames containing polyps, and polyp-related ORB features are selected based on the binary polyp mask. Simultaneously, semantic associations are established between 2D polyp regions and 3D map points in the visual SLAM. For image frames where no polyps are detected, conventional ORB-SLAM front-end processing continues. Therefore, while maintaining the original geometric tracking and map point maintenance mechanisms, this application can provide explicit 2D features and 3D semantic labels for lesion regions and reduce unnecessary segmentation calculations on polyp-free image frames, providing a structured data foundation for lesion perception mapping, polyp localization, and bronchoscopy navigation. Attached Figure Description

[0014] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in this application and should not be construed as limiting the scope of this application.

[0015] Figure 1 This is a flowchart of a visual SLAM front-end feature acquisition and semantic association method for bronchoscopic polyp perception implemented according to this application.

[0016] Figure 2 This is the overall framework diagram of the polyp detection front-end for bronchoscopy feature acquisition and semantic association in this application. In this case, monocular bronchoscopy image frames are first judged at the frame level by a polyp detection network. Image frames without polyps directly enter the conventional ORB-SLAM3 front-end. Image frames containing polyps are further segmented to generate a binary polyp mask. The original image frames and the corresponding mask are used together to screen polyp-related ORB features constrained by the mask, and further complete the semantic association between two-dimensional features and three-dimensional map points.

[0017] Figure 3 This is a schematic diagram of the experimental setup and experimental scenario for this application, including endoscopic equipment, a bronchial model, representative bronchoscope images, and a polyp model.

[0018] Figure 4 The image shows a representative processing result of an image frame containing polyps. The upper left is the original bronchoscopy image frame, the lower left is the binary polyp mask, and the right is the result of polyp-related ORB feature acquisition and semantic association under mask constraints. Blue dots represent ordinary intraluminal ORB feature points, and green dots represent polyp-related ORB feature points.

[0019] Figure 5The image shows a representative processing result of a polyp-free image frame. The left side is the original bronchoscopy image frame, and the right side is the result of conventional ORB feature extraction. Since polyp-related features were not activated, only ordinary ORB feature points in the lumen are shown in blue in the result image. Detailed Implementation

[0020] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0021] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] This application proposes a method and system for visual SLAM front-end feature acquisition and semantic association for bronchoscopy polyp detection. It constructs a visual SLAM front-end processing framework that integrates polyp detection, detection-guided segmentation, binary mask-constrained ORB feature selection, and 2D-3D semantic association. The semantic processing branch is gated using frame-level polyp detection results, performing polyp segmentation and mask generation only on image frames containing polyps. Polyp-related ORB features are selected based on the binary polyp mask, and semantic associations are established between 2D polyp regions and 3D map points in the visual SLAM. For image frames where no polyps are detected, conventional ORB-SLAM front-end processing continues. The process and experimental results of this embodiment demonstrate that this method can supplement bronchoscopy image frames with polyp-related 2D features and 3D map point semantic labels while maintaining the original geometric tracking and map point maintenance mechanisms of ORB-SLAM3. Furthermore, frame-level gating reduces unnecessary segmentation calculations on polyp-free image frames, providing a data foundation for lesion detection mapping, polyp localization, and bronchoscopy navigation.

[0023] In order to achieve the purpose of this application, such as Figure 1 and Figure 2 As shown, this embodiment adds polyp detection, detection-guided segmentation, binary mask generation, mask-constrained ORB feature selection, and two-dimensional-three-dimensional semantic association processing to the ORB-SLAM3 front end. Figure 1 This application illustrates the processing flow of its method. Figure 2The diagram illustrates the front-end framework and its dual-branch data flow: Monocular bronchoscopy image frames first undergo frame-level judgment via a polyp detection network; polyp-free image frames directly enter the conventional ORB-SLAM3 front-end, while polyp-containing image frames undergo further polyp segmentation to generate a binary polyp mask; the original image frames and corresponding masks are used together to filter polyp-related ORB features and further complete the semantic association between 2D features and 3D map points. The added semantic processing branch does not replace the geometric tracking link of ORB-SLAM3, but extracts supplementary information related to polyps from the current image frame and its map point observations, including the following steps: Step S1: Acquire the monocular bronchoscopy image frame at the current moment, input the image frame into the polyp detection network, and output the frame-level detection flag and polyp candidate region of the current image frame from the polyp detection network.

[0024] Step S2: Perform branch gating based on frame-level detection flags. For image frames containing polyps, perform polyp segmentation under polyp candidate region constraints and generate a binary polyp mask; for image frames without polyps, do not activate the segmentation branch and directly perform conventional ORB-SLAM front-end processing.

[0025] Step S3: Extract ORB features from the current image frame. By determining whether the image coordinates of each ORB feature are located within the binary polyp mask, a subset of polyp-related two-dimensional features is obtained from the full image ORB feature set.

[0026] Step S4: Obtain the 3D map points observed in the current image frame in the visual SLAM map, and project the 3D map points onto the image plane according to the current camera pose and camera projection model; by determining whether the projected coordinates are located within the binary polyp mask, assign polyp semantic labels to the current map point observation.

[0027] Step S5: Provide the polyp-related 2D feature subset and 3D map point observation results with polyp semantic labels to the subsequent semantic map construction, lesion localization or bronchoscopy navigation module, while maintaining the original geometric tracking and map point maintenance process of visual SLAM.

[0028] In the visual SLAM front-end feature acquisition and semantic association method and system for bronchoscopic polyp detection described in this application, step S1 involves acquiring the current monocular bronchoscope image frame, determining whether a polyp exists in the current image frame using a polyp detection network, and outputting candidate regions where the polyp is located. Let time... The acquired monocular bronchoscopy image frames are .in, and These represent the height and width of the image frame, respectively, and the number 3 indicates that the image frame contains three color channels. (The image frame is then...) Input polyp detection network The output of the polyp detection network is represented as .in, This is the frame-level detection flag for the current image frame. This is a candidate region for polyps. When the polyp detection network outputs at least one polyp detection result in the current image frame, it will... Setting it to 1 indicates that the current image frame contains polyps; when the polyp detection network does not output polyp detection results, it will... Setting it to 0 indicates that the current image frame is a polyp-free image frame. Polyp candidate region. The polyp detection can be determined by the rectangular detection boxes output by the polyp detection network. When multiple detection results exist in an image frame, each detection box can be processed as a candidate region, or the union of the corresponding regions of multiple detection boxes can be used as the polyp candidate region for the current image frame. Candidate regions provide coarse-grained location priors for subsequent polyp segmentation, allowing the segmentation network to focus on processing the suspected polyp regions located by the detection network, reducing interference from irrelevant backgrounds such as bronchial wall texture, specular reflection, dark areas, and image edges on the segmentation results. Frame-level detection flags. It is also used to control the activation state of semantic processing branches. When At that time, activation polyp segmentation, binary mask generation, mask-constrained ORB feature selection, and 2D-3D semantic association processing are performed; when At this time, the polyp segmentation network is not activated, and the current image frame directly enters the regular ORB-SLAM3 front end, which performs full-image ORB feature extraction, feature matching, camera pose estimation, and map point maintenance.

[0029] In one specific implementation, the polyp detection network employs the YOLOv5 object detection network. YOLOv5 acquires image features at different levels through a backbone feature extraction network, integrates shallow spatial information and deep semantic information using a multi-scale feature fusion structure, and outputs the polyp category, detection box coordinates, and corresponding confidence score through the detection head. Since the polyp detection network in this application is primarily used as a frame-level semantic gating module, as long as the current image frame outputs at least one polyp detection result that meets the detection conditions, the image frame can be sent to the subsequent segmentation branch.

[0030] In one specific training method, the polyp detection network uses publicly available polyp images to construct training data, and manually annotates the polyp regions in the training images with bounding boxes. The detection network is trained for 100 epochs using a stochastic gradient descent optimizer, with a batch size of 16 and an initial learning rate of [value missing]. .

[0031] In step S2, branch gating is performed based on the frame-level detection flag output in step S1, and a binary polyp mask corresponding to the current image frame is generated under the guidance of the polyp candidate region. The polyp segmentation network is defined as follows: The binary polyp mask corresponding to the current image frame is The detection-guided polyp segmentation process is represented as follows:

[0032] when At that time, the current image frame will be... and polyp candidate areas Input to the polyp segmentation network, and the polyp segmentation network outputs pixel-level polyp segmentation results; when At that time, the polyp segmentation network is not invoked, and Set to a zero-mask of the same size as the current image frame. To suppress bronchial walls, dark areas, specular reflections, and other background responses outside the polyp candidate region, a candidate region constraint input can be constructed based on the polyp candidate region. The candidate region constraint input is represented as:

[0033] in, Represents the pixel coordinates in the current image frame. Located in the candidate region. Pixels within the candidate region retain their original pixel values, while pixels outside the candidate region are set to zero, thereby suppressing irrelevant background responses outside the candidate region.

[0034] In another equivalent implementation, candidate regions can be cropped from the original image frame based on the detection bounding box, and the cropped candidate image can be scaled to the input size required by the segmentation network. After the segmentation network outputs the polyp segmentation results within the candidate regions, the segmentation results are restored to the coordinate system of the original image frame according to the coordinate transformation relationship recorded during the cropping and scaling process. The restored candidate region segmentation results are filled into a binary image of the same size as the original image frame, and the pixels outside the candidate regions are set to 0, thereby obtaining a complete binary polyp mask. The polyp segmentation network first outputs a probability map of polyp pixels. Then according to the preset segmentation threshold Binarize the probability graph:

[0035] in, Represents pixels It was identified as an area of ​​polyps. This indicates that the pixel is determined to be a non-polyp region. The resulting binary polyp mask is used both to filter polyp-related ORB features in the current image frame and to determine whether the projected coordinates of a 3D map point fall within a polyp region.

[0036] In one specific implementation, the polyp segmentation network uses PraNet. PraNet progressively refines the polyp region through multi-scale feature fusion and a reverse attention mechanism, further refining the polyp boundaries based on the overall polyp location. The segmentation network is trained using the publicly available polyp segmentation datasets Kvasir-SEG and CVC-ClinicDB, with Adam as the optimizer, 60 training epochs, a batch size of 16, and an initial learning rate of [missing information]. Input resolution set to Pixel.

[0037] In step S3, ORB feature extraction is performed on the current image frame, and the binary polyp mask obtained in step S2 is used to spatially filter the ORB features to obtain a subset of polyp-related two-dimensional features. Let's assume that from the current image frame... The extracted full-image ORB feature set is as follows:

[0038] in, This represents the total number of ORB features extracted in the current image frame. Indicates the first One ORB feature, This represents the pixel coordinates of the ORB feature in the current image frame. Each ORB feature can also contain information such as feature scale, orientation, response value, and binary descriptor. This is used to obtain the full-image ORB feature set. Next, a mask lookup is performed on the image coordinates of each ORB feature. When the binary mask value at the feature coordinate location is 1, the ORB feature is identified as a polyp-related feature; when the binary mask value at the feature coordinate location is 0, the feature is not added to the polyp-related two-dimensional feature subset. The polyp-related two-dimensional feature subset is represented as follows:

[0039] in, This represents a subset of two-dimensional features related to polyps in the current image frame. When the ORB feature coordinates are non-integer coordinates, the corresponding mask pixel coordinates can be obtained using the nearest neighbor rounding method, or the mask value at the corresponding position can be queried using bilinear interpolation. When the resolution of the image pyramid layer where the ORB features are located is inconsistent with the resolution of the binary polyp mask, the feature coordinates need to be transformed to the mask coordinate system. Let the width and height of the original image frame be... and The width and height of the binary polyp mask are respectively and Then the characteristic coordinates can be transformed according to the following formula:

[0040]

[0041] Based on the transformed coordinates By querying a binary polyp mask, erroneous spatial membership determinations can be avoided due to inconsistencies in resolution or image pyramid scale. It should be noted that the mask-constrained ORB feature filtering in this application does not delete the entire image ORB feature set. This also does not change how the full-image ORB features participate in regular feature matching, camera pose estimation, and map point updates. The full-image ORB feature set is still used in the original geometric tracking process of ORB-SLAM3, while the polyp-related 2D feature subset... It is then stored as a semantic feature set in parallel with the full-map ORB feature set, and used for subsequent two-dimensional-three-dimensional association, lesion area display, and polyp-related map point marking.

[0042] In step S4, the 3D map points observed in the ORB-SLAM3 map in the current image frame are projected onto the current image frame, and based on the spatial membership relationship between the projected coordinates and the binary polyp mask, the current observation of the 3D map points is assigned a polyp semantic label. Let the first... The coordinates of the three-dimensional map point in the world coordinate system are:

[0043] Let the camera pose corresponding to the current image frame be... ,in, This represents the coordinate transformation from the world coordinate system to the current camera coordinate system. Each 3D map point is first transformed from the world coordinate system to the current camera coordinate system. .set up Then in Under these conditions, the 3D map point can be projected onto the current image plane using a camera projection model:

[0044] in, This represents the projection function determined by the camera intrinsic parameters and the imaging model. This represents the projected coordinates of a 3D map point in the current image frame. In the pinhole camera model, let the camera focal length parameter be... and The principal point coordinates are Then the projected coordinates can be expressed as:

[0045]

[0046] Before performing semantic association, it is first determined whether the 3D map point is in front of the current camera and whether its projected coordinates are within the valid boundaries of the current image frame. For map points that do not meet the depth validity condition or whose projected coordinates exceed the image boundaries, semantic label determination for the current frame is not performed. For valid projected points, the semantic label for the currently observed polyp is determined based on the spatial membership between its projected coordinates and the binary polyp mask. To avoid conflicts with image frames... Confusion, will the first Map points at time The semantic tags are denoted as Its definition is:

[0047] in, Indicates the first The projection of a 3D map point in the current image frame is located inside the polyp mask. The current observation of this map point is marked as a polyp-related observation. This indicates that the current projection of the map point is outside the polyp mask, and its current observation is marked as a non-polyp-related observation. This semantic label is used to describe the observation attributes of the 3D map point in the current image frame, without replacing the original geometric attributes of the map points in ORB-SLAM3. The location of the map point is still estimated and updated by the feature matching, triangulation, local mapping, and back-end optimization processes in ORB-SLAM3, with the polyp semantic label serving as an additional attribute to establish a correlation with the 3D map point observation. Through the above processing, the 2D polyp mask in the image domain can establish a spatial correspondence with the 3D map points in ORB-SLAM3, enabling the system to obtain the polyp semantic information corresponding to the current image frame while retaining the 3D map geometric information. This forms a correlation link from the 2D polyp region, polyp-related 2D ORB features to the 3D map point observation.

[0048] In step S5, while maintaining the original geometric tracking and map point maintenance process of ORB-SLAM3, different front-end processing paths are executed based on the frame-level detection flags, and the polyp-related 2D feature subset and 3D map point observation results with polyp semantic labels are output. Regarding the frame-level detection flags... For polyp-free image frames, the polyp segmentation network is not invoked, nor is a non-zero binary polyp mask generated. Instead, the full-image ORB feature extraction, feature matching, motion model prediction, camera pose estimation, local map tracking, and map point update are directly performed by the conventional ORB-SLAM3 front-end. This avoids unnecessary pixel-level segmentation and mask-constrained feature selection for polyp-free image frames. (Frame-level detection flags are also mentioned.) For image frames containing polyps, a binary polyp mask is first generated based on the candidate polyp regions. Then, the mask-constrained feature filtering module obtains a subset of polyp-related two-dimensional features. It then projects the currently observable 3D map points onto the image plane using the current camera pose and camera projection model, generating semantic labels for the currently observed 3D map points. When processing image frames containing polyps, the original image frame still enters the ORB-SLAM3 geometric front end, and the full-image ORB features still participate in camera tracking and map point maintenance. Polyp detection, polyp segmentation, feature selection, and 2D-3D association constitute additional semantic processing links, which do not replace the original geometric tracking links of ORB-SLAM3, nor change the basic mechanisms of the original feature matching, pose estimation, and map optimization.

[0049] In one specific system implementation, the polyp detection network and polyp segmentation network are trained and inferred using the PyTorch deep learning framework. The ORB-SLAM3 geometric front end, mask-constrained ORB feature selection, and 2D-3D association process are implemented in C++. The system processes bronchoscopy videos frame by frame in chronological order. Each frame first passes through the polyp detection network, and then the segmentation and semantic processing branches are activated based on the frame-level detection results.

[0050] To verify the feasibility of the method described in this application, in one specific embodiment, a bronchial model is used as the experimental subject, and a polyp model is set inside the model. The experimental setup and scenario are as follows. Figure 3 As shown. The image acquisition device uses an endoscope, and the acquired bronchoscopic images have a resolution of [resolution missing]. The resolution was 1000 pixels, and the video frame rate was 25Hz. Before the experiment, Zhang Zhengyou's camera calibration method was used to obtain the intrinsic parameters of the endoscope camera to support camera pose estimation and 3D map point projection.

[0051] The experimental platform was configured with a general-purpose 2.40GHz processor and two general-purpose graphics cards, each with 24GB of video memory, and a general-purpose operating system. The polyp detection network and polyp segmentation network were implemented using an open-source deep learning framework, while the ORB-SLAM3 front-end and the newly added mask constraint feature selection and 2D-3D correlation program were implemented in C++.

[0052] The complete bronchoscopy video sequence contains 16,059 frames, of which 6,482 frames were manually labeled as polyp-positive and 9,577 frames were polyp-negative. The polyp detection network predicted 6,757 positive frames and 9,302 negative frames, including 6,119 true positive frames, 8,939 true negative frames, 638 false positive frames, and 363 false negative frames.

[0053] Based on the above detection results, the polyp detection network achieved an accuracy of 0.938, a precision of 0.906, a recall of 0.944, a specificity of 0.933, an F1 score of 0.924, a false positive rate of 0.067, a false negative rate of 0.056, and a balanced accuracy of 0.939. These results indicate that the frame-level polyp detection module can send most polyp-containing image frames to the subsequent segmentation and semantic processing branches, while allowing most polyp-free image frames to continue using the conventional ORB-SLAM3 front-end processing path.

[0054] To analyze the computational overhead of different processing branches, 10 frames containing polyps and 10 frames without polyps were randomly selected from the complete video sequence for single-frame runtime statistics. The polyp-containing samples required polyp detection, polyp segmentation, binary mask generation, mask-constrained ORB feature selection, and 2D-3D correlation, with an average runtime of 56.206 ms. The average runtime for all polyp-containing image frames in the complete sequence was 56.565 ms.

[0055] For polyp-free samples, the polyp segmentation branch is not activated; only polyp detection and conventional ORB-SLAM3 front-end processing are performed, with an average runtime of 31.532 ms. The average runtime for all polyp-free image frames in the complete sequence is 32.374 ms. This demonstrates that frame-level detection gating can prevent polyp-free image frames from undergoing unnecessary polyp segmentation, binary mask generation, and mask-constrained feature selection.

[0056] Representative processing results of image frames containing polyps are as follows: Figure 4 As shown. Figure 4 The top left image is the original bronchoscopic image frame, the bottom left image is the binary polyp mask, and the right image shows the results of polyp-related ORB feature acquisition and semantic association constrained by the mask. Blue dots represent ordinary intraluminal ORB feature points, and green dots represent polyp-related ORB feature points located within the polyp mask. Representative processing results for polyp-free image frames are shown below. Figure 5 As shown. Figure 5 The left side shows the original bronchoscopic image frame, and the right side shows the conventional ORB feature extraction result. Since no polyps were detected in this frame, polyp segmentation and polyp-related feature branches were not activated, and only ordinary ORB feature points within the lumen are displayed in blue. These results demonstrate that a binary polyp mask can constrain the acquisition range of polyp-related ORB features without altering the conventional geometric front-end processing of polyp-free image frames.

[0057] The above experimental results verify the feasibility of this application in frame-level polyp detection, detection-guided segmentation, mask-constrained ORB feature acquisition, and 2D-3D semantic association. It should be noted that the experimental results of this embodiment are mainly used to verify the front-end feature acquisition and semantic association processes, and should not be directly interpreted as an improvement in the absolute trajectory error, relative pose error, or overall tracking accuracy of ORB-SLAM3.

[0058] The device model, network structure, training parameters, image resolution, and operating platform in this embodiment are only used to illustrate one specific implementation of this application and do not constitute a limitation on the scope of protection of this application. Without departing from the technical concept of this application, the detection network, segmentation network, feature descriptors, coordinate transformation methods, and semantic label expression methods can be adjusted or equivalently replaced according to the actual bronchoscopy equipment, computing platform, and lesion type.

[0059] It should be noted that: The algorithms and displays provided herein are not inherently related to any pre-existing computer, virtual system, or other device. Various general-purpose systems can also be used in conjunction with the teachings herein. The required structure for constructing such systems is apparent from the above description. Furthermore, this application is not directed to any pre-existing programming language. It should be understood that the content of this application described herein can be implemented using various programming languages, and the above description of pre-existing languages ​​is for the purpose of disclosing the best mode of implementation of this application.

[0060] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0061] Similarly, it should be understood that, in order to simplify this application and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of this application, various features of this application are sometimes grouped together into a single embodiment, figure, or description thereof. However, this method of disclosure should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the following claims, inventive aspects lie in fewer than all features of a single foregoing disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0062] Those skilled in the art will understand that modules in the device of the embodiments can be adaptively changed and placed in one or more devices different from that embodiment. Modules, units, or components in the embodiments can be combined into a single module, unit, or component, and further, they can be divided into multiple sub-modules, sub-units, or sub-components. Except where at least some of such features and / or processes or units are mutually exclusive, any combination can be used to combine all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or device so disclosed. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0063] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the following claims, any of the claimed embodiments can be used in any combination.

[0064] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors can be used in practice to implement some or all of the functions of some or all of the components in the virtual machine creation system according to the embodiments of this application. This application can also be implemented as a device or system program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0065] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several systems, several of these systems may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0066] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in this application, and these should all be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A visual SLAM front-end feature acquisition and semantic association method for bronchoscopic polyp detection, characterized in that, include: Step S1: Acquire the monocular bronchoscopy image frame at the current moment, input the image frame into the polyp detection network, and obtain the frame-level detection flag and polyp candidate region used to indicate whether the current image frame contains polyps. Step S2: When the frame-level detection flag indicates that the current image frame contains polyps, a binary polyp mask is generated by constraining the input or output range of the polyp segmentation network according to the polyp candidate region; when the frame-level detection flag indicates that the current image frame does not contain polyps, the polyp segmentation network is not activated, and the current image frame is sent to the preset visual SLAM front end. Step S3: Extract ORB features from the current image frame, and retain the ORB features located within the binary polyp mask according to the spatial membership relationship between the image coordinates of the ORB features and the binary polyp mask, to obtain a subset of polyp-related two-dimensional features; Step S4: Obtain the 3D map points observed in the current image frame in the visual SLAM map, project the 3D map points onto the current image frame according to the current camera pose, and assign polyp semantic labels to the current observation of the 3D map points according to the spatial membership relationship between the projected coordinates and the binary polyp mask. Step S5: While maintaining the visual SLAM geometric tracking and map point maintenance process, output the polyp-related two-dimensional feature subset and the three-dimensional map point observation results with polyp semantic labels.

2. The method according to claim 1, characterized in that, Step S1 includes: Let the current image frame be Polyp detection network is The polyp detection network outputs ,in The frame-level detection flag, This refers to the candidate polyp region; When the polyp detection network outputs at least one polyp detection result, Set to 1, otherwise Set to 0.

3. The method according to claim 2, characterized in that, Step S2 includes: when At that time, will be located Pixel retention within; Will be located Pixels outside the segmentation range are set to zero or masked in the segmentation loss and segmentation output to form candidate region constraint inputs; The candidate region constraints are input into the polyp segmentation network. A binary polyp mask was obtained. ; when season It is a zero-mask.

4. The method according to claim 1, characterized in that, Step S3 includes: Construct the ORB feature set extracted from the current image frame as follows ;in, This represents the total number of ORB features extracted in the current image frame. Indicates the first One ORB feature, This indicates the pixel coordinates of the ORB feature in the current image frame; If features The image coordinates are located in the binary polyp mask. Within the foreground pixel range, this feature is retained in the polyp-related two-dimensional feature subset. Otherwise, this feature will not be added. .

5. The method according to claim 3, characterized in that, Step S4 includes: Definition of the first The coordinates of the three-dimensional map point in the world coordinate system are: The current camera pose is ;pass Obtain the projected coordinates of the 3D map points in the current image frame. , where π(·) is the camera projection function; like lie in Inside, the semantic labels of the currently observed polyps will be... Set to 1 otherwise set to 0.

6. The method according to claim 1, characterized in that, The preset ORB-SLAM front-end includes: For image frames where no polyps were detected, full-image ORB feature extraction, feature matching, camera pose estimation, and map point update were performed, but polyp segmentation, binary mask generation, and mask-constrained feature filtering were not performed.

7. The method according to claim 1, characterized in that, The polyp detection network includes: Construct a single-stage or two-stage object detection network; The polyp segmentation network adopts an encoder-decoder structure, a reverse attention structure, or a multi-scale feature fusion structure.

8. The method according to claim 1, characterized in that, include: The binary polyp mask has the same pixel coordinate reference as the current image frame; When the feature coordinates and the mask resolution are inconsistent, the feature coordinates or the mask are transformed according to the image scaling ratio before the spatial membership determination is performed.

9. A visual SLAM front-end feature acquisition and semantic association system for bronchoscopic polyp detection, characterized in that, include: The image acquisition module is used to acquire monocular bronchoscopy image frames; The polyp detection and gating module is used to output frame-level detection flags and polyp candidate regions, and select semantic processing branches or preset ORB-SLAM front-end branches based on the frame-level detection flags. The segmentation and masking module is used to generate binary polyp masks under the constraint of polyp candidate regions; The feature filtering module is used to obtain a subset of polyp-related two-dimensional features from the ORB feature set based on the binary polyp mask; Visual SLAM geometric front end, used to perform camera pose estimation and 3D map point maintenance; The 2D-3D association module is used to project 3D map points onto the image plane and generate polyp semantic labels based on the spatial membership relationship between the projected points and the binary polyp mask.

10. The system according to claim 9, characterized in that, include: The polyp detection and gating module shuts down the segmentation and masking module when the frame-level detection flag is not set, and sends the image frame directly to the visual SLAM geometric front end. When the frame-level detection flag is set to "yes", the segmentation and masking module, the feature selection module, and the 2D-3D association module are activated sequentially.

Images