Door handle detection and segmentation method and apparatus

Abstract

The application discloses a door handle detection and segmentation method and device, and relates to the technical field of target detection. A target detection model is used for fast coarse positioning of a door handle, and a boundary frame output by the target detection model is input into a lightweight instance segmentation model as a position prompt to guide the instance segmentation model to generate a high-precision pixel-level mask, so that a 'detection-segmentation' cascade process is formed, and the recognition robustness under complex background, reflection or shielding conditions is significantly improved. The MobileSAM model can be used as the instance segmentation model, the segmentation quality is guaranteed, the real-time requirement of an embedded platform (such as a Jetson AGX Orin) is met, the segmentation result is output in the form of a structured binary mask, and the mask quality is quantitatively evaluated, so that seamless integration with a downstream grasping planning or pose estimation module is supported.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a method and apparatus for detecting and segmenting door handles based on multi-stage visual perception. Background Technology

[0002] In service robots, smart homes, and autonomous inspection systems, accurate door handle detection is a crucial prerequisite for completing the basic human-computer interaction task of "opening the door." Currently, related technologies mainly rely on two types of methods: one is a rule-based method based on manual features, such as using color thresholds, geometric templates, or edge contour matching for door handle localization; the other is a general object detection or instance segmentation model based on deep learning, such as the YOLO series, Faster R-CNN, or Mask R-CNN. However, rule-based methods exhibit extremely poor robustness when faced with diverse door handle types (such as knob-type and lever-type), complex materials (reflective metal, transparent glass), changes in lighting, and partial occlusion in real-world scenarios, easily leading to missed or false detections.

[0003] While existing deep learning methods have improved accuracy, they still have significant limitations: object detection models can only output rectangular bounding boxes, failing to accurately depict the irregular curved surfaces and hollow structures of door handles; general instance segmentation models (such as SAM), although capable of pixel-level output, are unstable in segmenting small targets or low-contrast regions without effective guidance cues, and their original versions are computationally expensive, making them difficult to deploy on resource-constrained embedded robot platforms. Furthermore, publicly available datasets generally lack high-quality labeled samples specifically for door handles, resulting in poor domain adaptability and limited generalization ability in model training. Therefore, current technologies have not yet achieved an effective balance between accuracy, real-time performance, robustness, and deployability, making it difficult to meet the practical needs of service robots for reliable visual perception of door handles in open, dynamic environments.

[0004] The existing technology, "A Vision-Based Robot Grasping and Localization System and Method" (Patent No.: CN116460842A), extracts the geometric features of the target using a combination of frequency domain analysis and morphological processing. The process is as follows: Figure 9 As shown, the first three modules are mainly relevant to this invention: ① Image preprocessing: Manually label or perform simple threshold segmentation on the acquired image of the grasped object to obtain the binarized foreground region; ② Spectrum generation: Apply a windowed Fourier transform to the binary image to convert the spatial domain information into a frequency domain representation, thereby generating a spectrum. ③ Feature extraction: The spectrogram is binarized and a skeleton extraction algorithm is applied to obtain the center line of the target contour.

[0005] The traditional methods based on Fourier transform and skeleton extraction have certain shortcomings in the door handle perception task. The fundamental reason is that they rely on low-level image features and lack high-level semantic understanding capabilities, which are specifically manifested in the following three aspects: ① Highly sensitive to the geometric regularity of the target and image quality. This method assumes that the door handle has an approximately regular contour in order to form an identifiable main energy direction in the spectrogram. However, door handles in real-world scenes have diverse shapes and are often affected by factors such as metallic reflection and weak texture, resulting in diffused spectral energy and blurred main directions, making it impossible to reliably extract effective features.

[0006] ② The binarization and skeleton extraction processes are highly susceptible to background noise. This process requires a high-quality foreground mask as input, but in complex backgrounds, traditional thresholding segmentation struggles to accurately separate the target from the background. The resulting noise or broken areas can directly lead to skeleton extraction failure, causing trajectory planning to deviate completely from the physical structure.

[0007] ③ It lacks semantic discrimination capabilities and cannot adapt to open environments. This method only analyzes the spatial distribution pattern of pixels and cannot distinguish between "door handles" and non-target objects with similar appearances (such as door locks, decorative parts, and hanging objects). Furthermore, in scenarios where multiple door handles coexist, the frequency domain energies overlap, making it impossible to separate individual instances. This severely limits its practicality in unstructured environments such as homes and hotels. Summary of the Invention

[0008] In view of the above problems, the present invention provides a door handle detection and segmentation method and apparatus to overcome or at least partially solve the above problems. This method achieves high-precision pixel-level perception of door handles by constructing a collaborative processing flow from target detection to instance segmentation.

[0009] This invention provides the following solution: A method for detecting and segmenting door handles, comprising: Obtain the raw RGB image containing the door handle; The original RGB image is input into a trained object detection model to generate the bounding box information of the door handle; The original RGB image and the bounding box information are input into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. By thresholding the pixel-level probability map, a binary segmentation mask that accurately represents the outline of the door handle is generated.

[0010] Preferably: the target detection model includes a YOLOv8 network; the YOLOv8 network includes: The preprocessing module is used to scale the original RGB image Lrgb to a preset size and perform normalization processing; A backbone network, including a CSPDarknet network, is used to extract multi-scale depth features from preprocessed images; A feature fusion network, comprising a PANet structure, is used to fuse the multi-scale deep features in a top-down and bottom-up manner to enhance the semantic and spatial information of the features. The detection head is used to output the bounding box information and its corresponding category confidence based on the fused features.

[0011] Preferably, the YOLOv8 network is trained using a multi-task loss function, which consists of a weighted average of bounding box coordinate loss, confidence loss, and classification loss, expressed as follows:

[0012] In the formula: For bounding box regression loss; The target confidence loss is used to determine whether the bounding box contains the target. The loss is predicted for each category to distinguish the target category; , and These are the weighting coefficients for each type of loss.

[0013] Preferably: the instance segmentation model includes the MobileSAM model; the MobileSAM model includes: The image encoder uses MobileNetV3 as the backbone network to extract image features from the original RGB image LrgbLrgb. A cue encoder is used to encode the bounding box information BB into a cue feature vector; A mask decoder is used to perform deep fusion of the image features and the cue feature vector, and output the probability map.

[0014] Preferably, in the mask generation step, the binary segmentation mask is generated using the following formula. :

[0015] In the formula: This indicates that a threshold value is set; the default value is 0.5. This represents the confidence level that each pixel belongs to the door handle.

[0016] Preferably: the confidence level of each pixel belonging to the door handle It can be expressed by the following formula:

[0017] In the formula: Indicates MobileSAM network parameters, For the Sigmoid function, Represents the original RGB image. This indicates bounding box information.

[0018] Preferably, a mask quality assessment module is used to quantitatively score the binary segmentation mask from four dimensions: integrity, shape rationality, edge clarity, and connectivity. After adaptive weighted fusion, a comprehensive quality score is formed. When the comprehensive score is lower than the preset quality threshold, an adaptive correction mechanism is triggered. The adaptive correction mechanism includes: Re-segment by adjusting the position or type of the prompt points; If the image region still does not meet the standard, it will be segmented again after targeted enhancement preprocessing based on the scene classification results. The enhancement preprocessing includes removing highlights in reflective scenes, adaptive histogram equalization in low-light scenes, and sharpening in blurred scenes.

[0019] Preferably, before the output and integration steps, morphological operations or connected component analysis are performed on the generated binary segmentation mask to eliminate noise or holes in the mask; after the mask generation step, the binary segmentation mask is encapsulated into a standardized robot operating system ROS 2 message format and published through a preset topic for use by downstream robot control modules.

[0020] Preferably, both the object detection model and the instance segmentation model are obtained by training or fine-tuning using a specially constructed door handle dataset; the method for constructing the door handle dataset includes: RGB images of door handles of various types and materials were collected under different scenes, lighting conditions and viewing angles. For each acquired RGB image, the bounding box of the door handle and the pixel-level precise mask are simultaneously labeled; The labeled image data is divided into non-overlapping training, validation, and test sets.

[0021] A door handle detection and segmentation device is provided for performing the above-described door handle detection and segmentation method, the device comprising: The data acquisition unit is used to acquire the original RGB image containing the door handle; The target detection unit is used to input the original RGB image into the trained target detection model to generate the bounding box information of the door handle; An instance segmentation unit is used to input the original RGB image and the bounding box information into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. The mask generation unit is used to generate a binary segmentation mask that accurately represents the outline of the door handle by thresholding the pixel-level probability map.

[0022] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects: This application provides a door handle detection and segmentation method and apparatus. It employs a target detection model for rapid coarse localization of the door handle, using the output bounding box as a location cue input to a lightweight instance segmentation model. This guides the model to generate a high-precision pixel-level mask, forming a cascaded "detection → segmentation" process that significantly improves robustness under complex backgrounds, reflective conditions, or occlusion. The MobileSAM model can be used as the instance segmentation model, ensuring segmentation quality while meeting the real-time requirements of embedded platforms (such as Jetson AGX Orin). The segmentation results are output as structured binary masks, and the mask quality is quantified for evaluation, supporting seamless integration with downstream grasping planning or pose estimation modules. RGB-D images are simultaneously acquired using a depth camera in diverse real-world scenes. Pixel-level masks and bounding boxes of the door handles are manually and meticulously annotated, constructing a door handle dataset covering various types, materials, lighting conditions, and occlusion conditions. This provides a domain-adaptive data foundation for model training, fine-tuning, and performance verification. By using the detection frame as a strong spatial prior, the segmentation region of MobileSAM is constrained, effectively overcoming the segmentation blurring problem caused by the small size, large curvature, and hollow structure of the door handle, and ensuring that the output mask accurately matches the real physical contour.

[0023] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0025] Figure 1 This is a flowchart of a door handle detection and segmentation method provided in an embodiment of the present invention; Figure 2This is an overall structural diagram of a door handle detection and segmentation method provided in an embodiment of the present invention; Figure 3 This is a flowchart of the YOLOv8 target detection process provided in an embodiment of the present invention; Figure 4 This is a flowchart of the MobileSAM instance segmentation process provided in an embodiment of the present invention; Figure 5 This is a flowchart of the mask quality assessment provided in an embodiment of the present invention; Figure 6 This is a flowchart of the dataset construction process provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of a door handle detection and segmentation device provided in an embodiment of the present invention; Figure 8 This is a schematic diagram of a door handle detection and segmentation device provided in an embodiment of the present invention; Figure 9 This is a flowchart of an existing technical solution. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0027] See Figure 1 , Figure 2 This invention provides a method for detecting and segmenting door handles, as shown in the embodiments of the present invention. Figure 1 , Figure 2 As shown, the method may include: S101: Obtain the original RGB image containing the door handle; in practice, the system first obtains the original RGB image:

[0028] in, These represent the image height and width, respectively.

[0029] The raw RGB image can be acquired by any visible light imaging device, with the acquisition frequency set to 30 FPS to meet real-time processing requirements. The acquired raw data serves as the input source for subsequent visual algorithms.

[0030] S102: Input the original RGB image into the trained object detection model to generate the bounding box information of the door handle; in specific implementation, the object detection model in this application embodiment may include a YOLOv8 network; the YOLOv8 network includes: The preprocessing module is used to scale the original RGB image Lrgb to a preset size and perform normalization processing; A backbone network, including a CSPDarknet network, is used to extract multi-scale depth features from preprocessed images; A feature fusion network, comprising a PANet structure, is used to fuse the multi-scale deep features in a top-down and bottom-up manner to enhance the semantic and spatial information of the features. The detection head is used to output the bounding box information and its corresponding category confidence based on the fused features.

[0031] This application uses YOLOv8 to quickly and coarsely locate the door handle, and uses the output bounding box as a location prompt to input into a lightweight instance segmentation model to guide it to generate a high-precision pixel-level mask, forming a "detection → segmentation" cascade process, which significantly improves the robustness of recognition under complex backgrounds, reflective or occluded conditions.

[0032] To achieve rapid door handle localization, this embodiment employs a YOLOv8 target detection network. YOLOv8 is a single-stage anchor-free target detection network with advantages of high efficiency and high accuracy, making it suitable for real-time applications.

[0033] The target detection flowchart is as follows Figure 3 As shown, the model receives the original RGB image. First, preprocessing is performed, including resizing to 640×640 pixels, normalization, and aspect ratio-preserving padding. The preprocessed image is then input into the CSPDarknet backbone network to extract multi-scale features, followed by feature fusion using a PANet structure to enhance the representation of spatial and semantic information. Finally, the model outputs bounding box information. and category confidence ,in The coordinates of the center point, For frame dimensions, This indicates the credibility of the test result being "door handle".

[0034] To optimize detection performance, YOLOv8 employs a multi-task loss function for training. This function consists of a weighted average of three parts: bounding box coordinate loss, confidence loss, and classification loss, expressed as:

[0035] in: For the bounding box regression loss, this method uses CloU loss, which comprehensively considers the impact of overlapping areas, center point distance and aspect ratio on the accuracy of the bounding box. The target confidence loss is used to determine whether the bounding box contains the target. The loss is predicted for each category to distinguish the target category; , and These are the weighting coefficients for each type of loss.

[0036] Through this multi-task loss function, the model can effectively improve the accuracy and robustness of target localization, meeting the requirements for real-time and stable detection of door handles in robotic arm grasping tasks.

[0037] S103: The original RGB image and the bounding box information are input together into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. In a specific implementation, the instance segmentation model in this application embodiment may include the MobileSAM model; the MobileSAM model includes: The image encoder uses MobileNetV3 as the backbone network to extract image features from the original RGB image Lrgb. A cue encoder is used to encode the bounding box information BB into a cue feature vector; A mask decoder is used to perform deep fusion of the image features and the cue feature vector, and output the probability map.

[0038] The embodiments of this application adopt the MobileSAM model, which ensures the segmentation quality while meeting the real-time requirements of embedded platforms (such as Jetson AGX Orin); and outputs the segmentation results in the form of structured binary masks, supporting seamless integration with downstream grasping planning or pose estimation modules.

[0039] Specifically, after YOLOv8 detects the approximate location of the door handle, the system further calls the lightweight instance segmentation model MobileSAM to perform pixel-level precise segmentation of the detection box region and generate a binary mask. This allows for the extraction of the precise contours of the door handle.

[0040] MobileSAM is a mobile-optimized version of the SAM architecture. By using a lightweight visual Transformer combined with the MobileNetV3 backbone network, it significantly reduces computational overhead while maintaining high segmentation accuracy. Its core mechanism is prompt-guided segmentation, which can achieve zero-shot instance segmentation using only detection boxes as prompts, without requiring training for specific categories.

[0041] like Figure 4 As shown, the input to the MobileSAM model consists of two parts: Original RGB image Bounding box information output by YOLOv8 The bounding box serves as a location cue, guiding the model to accurately segment the target region.

[0042] The model consists of three core components: an image encoder that uses a MobileNetV3 backbone network to extract multi-scale image features; a cue encoder that encodes the input bounding boxes into cue feature vectors; and a mask decoder that fuses the image features and cue features to output a pixel-level probability map. , which represents the confidence level that each pixel belongs to the door handle.

[0043]

[0044] in, Indicates MobileSAM network parameters, The Sigmoid function is used to analyze the probability graph. Thresholding is performed to generate a binary mask. :

[0045] in, The threshold is set by default to 0.5; pixels with a mask value of 1 constitute the precise outline area of ​​the door handle, providing accurate input information for subsequent gripping point prediction.

[0046] To ensure the high reliability of the mask output to the subsequent fine-grained control module, this application embodiment can provide a mask quality evaluation module to quantitatively score the binary segmentation mask from four dimensions: integrity, shape rationality, edge clarity, and connectivity. After adaptive weighted fusion, a comprehensive quality score is formed. When the comprehensive score is lower than the preset quality threshold, an adaptive correction mechanism is triggered. The adaptive correction mechanism includes: Re-segment by adjusting the position or type of the prompt points; If the image region still does not meet the standard, it will be segmented again after targeted enhancement preprocessing based on the scene classification results. The enhancement preprocessing includes removing highlights in reflective scenes, adaptive histogram equalization in low-light scenes, and sharpening in blurred scenes.

[0047] In practice, as shown in Figure 5, after MobileSAM generates the initial mask, a mask quality evaluation module is introduced to quantitatively score the mask from four dimensions: integrity, shape rationality, edge sharpness, and connectivity. ① Integrity check: Calculate the ratio of the mask area to the bounding box area to determine whether the segmentation result completely covers the target object, avoiding missed detections or over-segmentation; ② Shape rationality check: Verify whether the segmentation result conforms to the typical geometric characteristics of a door handle by using shape indices such as the aspect ratio of the minimum bounding rectangle and the roundness of the outline; ③ Edge sharpness check: Calculate the overlap between the mask edge and the image gradient field to evaluate the accuracy of the segmentation boundary, especially for degraded scenes such as reflection and blur. ④ Connectivity check: Count the number of connected regions in the mask. Ideally, it should be 1 to ensure the structural integrity of the target object and effectively handle fragmentation caused by occlusion.

[0048] The scoring of the above four dimensions adopts a scene-aware adaptive weight fusion mechanism. First, the door handle area is analyzed through lightweight scene classification to identify lighting conditions (reflective / normal / low light), occlusion degree (none / partial / severe), and door handle type (lever-type / circular) to generate the current scene label. Then, based on a predefined scene-weight mapping table, differentiated baseline weights are assigned to different scenes—for example, increasing the weight of edge sharpness and shape rationality in reflective scenes, and increasing the weight of integrity and connectivity in occluded scenes. Finally, dynamic fine-tuning is performed based on the dispersion of the scores of each dimension. When the score of a certain dimension is significantly low, its weight is automatically increased, so that the comprehensive score can better reflect the "barrel effect" of segmentation defects.

[0049] For scenario S, the baseline weight vector is:

[0050] in, These represent four quality dimensions. For the first The baseline weights for each dimension, For the first Real-time ratings across multiple dimensions. It is a natural constant. and To adjust the parameters and give lower-scoring dimensions a higher weight, the overall score is adjusted. The calculation is as follows:

[0051] After adaptive weighted fusion, a comprehensive quality score is generated. When the comprehensive score is lower than a preset quality threshold, an adaptive correction mechanism is triggered. ① First, re-segment by adjusting the position or type of the prompts; ②If it still does not meet the standard, the image region is further segmented after targeted enhancement preprocessing based on the scene classification results—such as removing highlights in reflective scenes, adaptive histogram equalization in low-light scenes, and sharpening in blurry scenes.

[0052] This mechanism can flexibly cope with complex and ever-changing real-world environments, ensuring that the mask output to the subsequent fine-grained control module has high reliability and effectively avoiding operational errors caused by poor segmentation quality.

[0053] S104: By thresholding the pixel-level probability map, a binary segmentation mask that accurately represents the outline of the door handle is generated.

[0054] Furthermore, embodiments of this application can provide that, after the mask generation step, the binary segmentation mask is encapsulated into a standardized robot operating system ROS 2 message format and published through a preset topic for use by downstream robot control modules. Before the output and integration step, morphological operations or connected component analysis are performed on the generated binary segmentation mask to eliminate noise or holes in the mask.

[0055] The final output of the method provided in this application is a pixel-level binary segmentation mask of the door handle. This mask is published to downstream task modules via a standard communication protocol, enabling seamless integration with the robot system. The specific implementation is as follows: ① Mask publishing: Convert the mask image output by MobileSAM into sensor_msgs / Image message format and publish it through the ROS2 topic / door_handle / mask; ② Optional post-processing: Before release, morphological closing operations can be performed on the mask to fill internal holes according to application requirements, or the maximum area region can be preserved through connected component analysis to improve robustness; ③ Downstream task support: This mask can be used as input for subsequent modules such as grasp point prediction, pose estimation, and robotic arm motion planning, but its specific application is not within the scope of protection of this invention.

[0056] It is understood that the core innovation of the method provided in this application lies in the generation process from an RGB image to a pixel-level mask, rather than the downstream control logic. The existence of the output module is only to illustrate that this method has good system compatibility and engineering practicality.

[0057] Both the object detection model and the instance segmentation model are obtained by training or fine-tuning using a specially constructed door handle dataset; the method for constructing the door handle dataset includes: To better train the network, embodiments of this application can provide RGB images of door handles of various types and materials under different scenes, lighting conditions and viewing angles; For each acquired RGB image, the bounding box of the door handle and the pixel-level precise mask are simultaneously labeled; The labeled image data is divided into non-overlapping training, validation, and test sets.

[0058] To support the training and performance verification of the multi-stage visual perception model provided in this application, a dedicated image dataset for door handles called DoorHandle-RealWorld (DHRW) can be constructed. The construction of this dataset is completed before system deployment, belonging to the offline preparation phase, and does not participate in the real-time operation process, but it provides high-quality, domain-adaptive supervision signals for the YOLOv8 detection model and the MobileSAM segmentation model.

[0059] like Figure 6 The diagram shown is a flowchart of the dataset construction process. The specific construction steps are as follows: ①Scenario and sample diversity design: Images of door handles were collected covering typical indoor environments such as homes, office buildings, and hotel corridors, ensuring sufficient diversity in lighting conditions (natural light, artificial lighting, backlighting), background complexity (solid color door panels, sticker decorations), and occlusion (hanging objects, hands obstructing view, partial field of vision restriction). Door handle types include common structures such as knobs, levers, and push-pull handles, and materials include metal, plastic, wood, and composite materials to enhance the model's generalization ability.

[0060] ②Image acquisition: RGB images of the door handle were acquired using a visible light imaging device at different time periods and from different viewpoints, with a total sample size of no less than 5,000 images. All images had a resolution of no less than 640×480 to ensure that the door handle area had sufficient pixel detail to support fine segmentation.

[0061] ③ Detailed annotation standards: Double-label each image: Boundary box annotation: Annotate the smallest bounding rectangle of the door handle, in the format ( ), used to train the YOLOv8 object detection model; Pixel-level mask annotation: Fine-grained manual annotation is performed along the actual outline of the door handle using a polygon tool to generate a binary mask. The foreground (door handle) is labeled as 1 and the background as 0, which is used to train and evaluate the MobileSAM instance segmentation model.

[0062] All annotations were completed by professionals and cross-checked to ensure consistency and accuracy.

[0063] ④ Dataset partitioning and augmentation strategies: All samples were divided into training, validation, and test sets in a 7:2:1 ratio to ensure no overlap in door handle type or scene distribution. During model training, data augmentation strategies such as random brightness / contrast adjustment, Gaussian noise, motion blur, and simulated occlusion were applied to the training set images to further improve the model's robustness in real-world scenarios.

[0064] As can be seen, the door handle detection and segmentation method provided in this application has several advantages. First, it breaks through the limitation of traditional object detection that only outputs rectangular boxes, achieving pixel-level accurate segmentation of door handles of any shape and accurately restoring their complex geometric details such as curvature and hollowing. Second, it maintains stable recognition performance under complex interference conditions such as strong reflection, weak texture, and partial occlusion, significantly improving the system's adaptability in real-world scenarios. Third, through lightweight model design and a multi-stage collaborative mechanism, it keeps the end-to-end inference latency within 100 milliseconds while ensuring segmentation quality, meeting the needs of real-time robot control. Fourth, it constructs a dedicated multimodal dataset for door handles for robot door opening tasks, covering high-quality labeled samples under various types, materials, and environmental conditions, providing standardized data support for model training, fine-tuning, and performance evaluation, thereby comprehensively improving the feasibility and engineering value of the door handle visual perception system.

[0065] In summary, the door handle detection and segmentation method provided in this application uses an object detection model for rapid coarse localization of the door handle. The output bounding box is then used as a location cue input to a lightweight instance segmentation model, guiding it to generate a high-precision pixel-level mask. This forms a cascaded "detection → segmentation" process, significantly improving the robustness of recognition under complex backgrounds, reflective conditions, or occlusion. The MobileSAM model can be used as the instance segmentation model, ensuring segmentation quality while meeting the real-time requirements of embedded platforms (such as Jetson AGX Orin). The segmentation results are output as structured binary masks, and the mask quality is quantified for evaluation, supporting seamless integration with downstream grasping planning or pose estimation modules. RGB-D images are simultaneously acquired using a depth camera in diverse real-world scenes, and the pixel-level masks and bounding boxes of the door handles are manually and meticulously annotated. This constructs a door handle dataset covering various types, materials, lighting conditions, and occlusion conditions, providing a domain-adaptive data foundation for model training, fine-tuning, and performance verification. By using the detection frame as a strong spatial prior, the segmentation region of MobileSAM is constrained, effectively overcoming the segmentation blurring problem caused by the small size, large curvature, and hollow structure of the door handle, and ensuring that the output mask accurately matches the real physical contour.

[0066] See Figure 7This application embodiment can also provide a door handle detection and segmentation device, such as... Figure 6 As shown, the device for performing the above-described door handle detection and segmentation method may include: The data acquisition unit 701 is used to acquire the original RGB image containing the door handle; The target detection unit 702 is used to input the original RGB image into the trained target detection model to generate the bounding box information of the door handle; The instance segmentation unit 703 is used to input the original RGB image and the bounding box information into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. The mask generation unit 704 is used to generate a binary segmentation mask that accurately represents the outline of the door handle by thresholding the pixel-level probability map.

[0067] This application embodiment can also provide a door handle detection and segmentation device, the device including a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the steps of the door handle detection and segmentation method described above according to the instructions in the program code.

[0068] like Figure 8 As shown in the figure, a door handle detection and segmentation device provided in this application embodiment may include: a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, memory 11, and communication interface 12 all communicate with each other through the communication bus 13.

[0069] In this embodiment, the processor 10 may be a central processing unit (CPU), a graphics processing unit (GPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic devices.

[0070] The processor 10 can call the program stored in the memory 11. Specifically, the processor 10 can execute the operations in the embodiments of the door handle detection and segmentation method.

[0071] The memory 11 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment, the memory 11 stores at least a program for implementing the following functions: Obtain the raw RGB image containing the door handle; The original RGB image is input into a trained object detection model to generate the bounding box information of the door handle; The original RGB image and the bounding box information are input into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. By thresholding the pixel-level probability map, a binary segmentation mask that accurately represents the outline of the door handle is generated.

[0072] In one possible implementation, the memory 11 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function (such as file creation or data read / write). The data storage area may store data created during use, such as initialization data.

[0073] In addition, memory 11 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device or other volatile solid-state storage device.

[0074] Communication interface 12 can be an interface for a communication model, used to connect with other devices or systems.

[0075] Of course, it should be noted that, Figure 8 The structure shown does not constitute a limitation on the door handle detection and segmentation device in the embodiments of this application. In practical applications, the door handle detection and segmentation device may include devices that are more advanced than those described above. Figure 8 More or fewer components as shown, or combinations of certain components.

[0076] This application embodiment may also provide a computer-readable storage medium for storing program code for executing the steps of the above-described door handle detection and segmentation method.

[0077] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0078] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this application.

[0079] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0080] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A method for detecting and segmenting door handles, characterized in that, include: Obtain the raw RGB image containing the door handle; The original RGB image is input into a trained object detection model to generate the bounding box information of the door handle; The original RGB image and the bounding box information are input into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. By thresholding the pixel-level probability map, a binary segmentation mask that accurately represents the outline of the door handle is generated.

2. The door handle detection and segmentation method according to claim 1, characterized in that, The target detection model includes a YOLOv8 network; the YOLOv8 network includes: The preprocessing module is used to scale the original RGB image Lrgb to a preset size and perform normalization processing; A backbone network, including a CSPDarknet network, is used to extract multi-scale depth features from preprocessed images. A feature fusion network, comprising a PANet structure, is used to fuse the multi-scale deep features in a top-down and bottom-up manner to enhance the semantic and spatial information of the features. The detection head is used to output the bounding box information and its corresponding category confidence based on the fused features.

3. The door handle detection and segmentation method according to claim 2, characterized in that, The YOLOv8 network is trained using a multi-task loss function, which consists of a weighted average of bounding box coordinate loss, confidence loss, and classification loss, expressed as follows: In the formula: For bounding box regression loss; The target confidence loss is used to determine whether the bounding box contains the target. The loss is predicted for each category to distinguish the target category; , and These are the weighting coefficients for each type of loss.

4. The door handle detection and segmentation method according to claim 1, characterized in that, The instance segmentation model includes the MobileSAM model; the MobileSAM model includes: The image encoder uses MobileNetV3 as the backbone network to extract image features from the original RGB image LrgbLrgb. A cue encoder is used to encode the bounding box information BB into a cue feature vector; A mask decoder is used to perform deep fusion of the image features and the cue feature vector, and output the probability map.

5. The door handle detection and segmentation method according to claim 1, characterized in that, In the mask generation step, the binary segmentation mask is generated using the following formula. : In the formula: This indicates that a threshold value is set; the default value is 0.

5. This represents the confidence level that each pixel belongs to the door handle.

6. The door handle detection and segmentation method according to claim 5, characterized in that, Confidence that each pixel belongs to the door handle It can be expressed by the following formula: In the formula: Indicates MobileSAM network parameters, For the Sigmoid function, Represents the original RGB image. This indicates bounding box information.

7. The door handle detection and segmentation method according to claim 1, characterized in that, A mask quality assessment module is used to quantitatively score the binary segmentation mask from four dimensions: integrity, shape rationality, edge clarity, and connectivity. After adaptive weighted fusion, a comprehensive quality score is formed. When the comprehensive score is lower than the preset quality threshold, an adaptive correction mechanism is triggered. The adaptive correction mechanism includes: Re-segment by adjusting the position or type of the prompt points; If the image region still does not meet the standard, it will be segmented again after targeted enhancement preprocessing based on the scene classification results. The enhancement preprocessing includes removing highlights in reflective scenes, adaptive histogram equalization in low-light scenes, and sharpening in blurred scenes.

8. The door handle detection and segmentation method according to claim 1, characterized in that, Before the output and integration steps, morphological operations or connected component analysis are performed on the generated binary segmentation mask to eliminate noise or holes in the mask; after the mask generation step, the binary segmentation mask is encapsulated into a standardized robot operating system ROS 2 message format and published through a preset topic for use by downstream robot control modules.

9. The door handle detection and segmentation method according to claim 1, characterized in that, Both the object detection model and the instance segmentation model were obtained by training or fine-tuning using a specially constructed door handle dataset; The method for constructing the door handle dataset includes: RGB images of door handles of various types and materials were collected under different scenes, lighting conditions and viewing angles. For each acquired RGB image, the bounding box of the door handle and the pixel-level precise mask are simultaneously labeled; The labeled image data is divided into non-overlapping training, validation, and test sets.

10. A door handle detection and segmentation device, characterized in that, The apparatus for performing the door handle detection and segmentation method according to any one of claims 1-9, the apparatus comprising: The data acquisition unit is used to acquire the original RGB image containing the door handle; The target detection unit is used to input the original RGB image into the trained target detection model to generate the bounding box information of the door handle; An instance segmentation unit is used to input the original RGB image and the bounding box information into a lightweight instance segmentation model, so that the instance segmentation model uses the bounding box information as a spatial location cue to perform pixel-level classification of the image and generate a pixel-level probability map. Each element value in the pixel-level probability map represents the confidence that the corresponding pixel belongs to the door handle. The mask generation unit is used to generate a binary segmentation mask that accurately represents the outline of the door handle by thresholding the pixel-level probability map.

Images