YOLO-SAM object three-dimensional target detection and extraction method
By combining the YOLO11-PQDA and MobileSAM models, the robustness and computational efficiency issues of 3D target detection and extraction in complex environments are solved, achieving efficient and lightweight 3D target detection and extraction, which meets the real-time requirements of edge computing devices.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-05-25
- Publication Date
- 2026-07-10
AI Technical Summary
Existing technologies suffer from insufficient algorithm robustness and limited computing power of edge computing devices in the processing of 3D point clouds in complex dynamic environments, making it difficult to achieve high-precision, real-time 3D target detection and extraction.
The YOLO11-PQDA model is used for pruning, quantization, and knowledge distillation. Combined with the MobileSAM model, the SternYOLO-SAM-Lite framework is constructed for target detection and extraction. By utilizing the polarization self-attention mechanism and zero-shot transfer characteristics, lightweight and efficient 3D target extraction is achieved.
It achieves efficient and lightweight 3D target detection and extraction in complex scenarios, improving detection accuracy and robustness, and adapting to the real-time needs of edge computing devices.
Smart Images
Figure CN122368962A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision and multimodal perception technology, and more specifically to a YOLO-SAM method for three-dimensional object detection and extraction. Background Technology
[0002] In the fields of intelligent transportation systems and port automation, environmental perception based on 3D point clouds is a core technology for achieving autonomous operations. Specifically, the accuracy and real-time performance of 3D feature extraction in ship berthing detection and obstacle (such as vehicle, pedestrian, and irregularly shaped object) recognition during intelligent vehicle operation directly determine the safety and efficiency of ship pose estimation, autonomous driving path planning, and obstacle avoidance strategies. However, 3D point cloud processing based on LiDAR faces two common technical challenges in these complex application scenarios: (1) The algorithm's robustness is insufficient due to noise in complex dynamic environments.
[0003] In unstructured real-world scenarios, the quality of point cloud data is highly susceptible to interference from environmental factors.
[0004] In port operations, subtle interference from cables, multipath reflections from sea waves, and dynamic facilities in the work area often lead to false detections due to clutter and artifacts. In intelligent vehicle driving scenarios, the complex topology of urban roads, the highly dynamic traffic flow, the noise brought by adverse weather conditions such as rain, fog, and snow, and the mutual occlusion of vehicles / pedestrians make it difficult to accurately and completely identify and segment target features (especially edge and contour features).
[0005] (2) The computing power constraints of the edge side and the vehicle terminal require the algorithm to be both lightweight and real-time.
[0006] Whether it's edge computing boxes in ports or in-vehicle computing platforms in smart cars (such as embedded domain controllers), their computing resources, memory capacity, and power consumption budgets are relatively limited. Existing high-precision deep learning models often have a huge number of parameters and high computational complexity, making it difficult to support long-term, high-frame-rate real-time inference on edge devices with limited computing power, and thus failing to meet the millisecond-level response requirements of autonomous driving and automated operations.
[0007] Therefore, while ensuring detection accuracy, existing technologies still have significant room for improvement in terms of reducing computational overhead, improving inference speed, and adapting to the stringent deployment requirements of port edges and vehicle-mounted mobile terminals through algorithm optimization. Designing a lightweight, efficient algorithm with strong robustness under complex interference to achieve real-time and accurate extraction of 3D targets in complex scenes is a pressing challenge for current intelligent sensing technologies. Summary of the Invention
[0008] In view of this, the present invention provides a YOLO-SAM method for 3D object detection and extraction. It addresses the problems of complex noise interference, insufficient algorithm robustness, and limited computing power of edge computing devices in the existing 3D point cloud processing technology. It provides a framework (SternYOLO-SAM-Lite) that integrates YOLO11-PQDA object detection and MobileSAM lightweight segmentation to achieve efficient object detection and accurate 3D point cloud extraction.
[0009] To achieve the above objectives, the present invention adopts the following technical solution: A YOLO-SAM method for 3D object detection and extraction includes: Obtain the original 3D point cloud data of the target object and perform preprocessing to obtain the 2D point cloud intensity map of the object; The two-dimensional point cloud intensity map is input into the target detection module to detect the target object and output the bounding box coordinates of the target region. The two-dimensional point cloud intensity map and bounding box coordinates are used as prompt information and input into the target extraction module to accurately segment the target object and output a two-dimensional mask of the target region. The pixel coordinates of the two-dimensional mask are converted into three-dimensional coordinates, and the corresponding points are extracted from the three-dimensional point cloud data of the original target object based on the three-dimensional coordinates to obtain the final target three-dimensional point cloud.
[0010] Preferably, the preprocessing of the original object's three-dimensional point cloud data includes coordinate correction and dimensionality reduction; the dimensionality reduction operation projects the three-dimensional point cloud data onto a two-dimensional plane and generates a corresponding two-dimensional point cloud intensity map based on the reflection intensity information of the point cloud.
[0011] Preferably, the target detection module uses the YOLO11-PQDA model, and optimizes the YOLO11-PQDA model by pruning, quantization, knowledge distillation and attention mechanism. The optimized YOLO11-PQDA model is used to perform preliminary localization of the target object and output the bounding box coordinates.
[0012] Preferably, the target extraction module, based on the zero-sample transfer characteristic of the MobileSAM model, accurately segments the target object with bounding box cues and generates a high-quality two-dimensional segmentation mask.
[0013] Preferably, a pre-calibrated coordinate transformation matrix is used to convert the pixel coordinates of the two-dimensional mask into three-dimensional coordinates. Based on the transformed three-dimensional coordinates, all points constituting the target object are extracted from the three-dimensional point cloud data of the original target object to obtain the final target three-dimensional point cloud.
[0014] Preferably, a YOLO-SAM-based 3D object detection and extraction system includes: Data preprocessing module: used to acquire the original 3D point cloud data of the target object and preprocess it to obtain the 2D point cloud intensity map of the object; Target detection module: Detects target objects based on 2D point cloud intensity maps and outputs the bounding box coordinates of the target region; Target extraction module: Uses the 2D point cloud intensity map and bounding box coordinates as prompt information to accurately segment the target object and output a 2D mask of the target region; 3D extraction module: Converts the pixel coordinates of the 2D mask into 3D coordinates, and extracts the corresponding points from the 3D point cloud data of the original target object based on the 3D coordinates to obtain the final target 3D point cloud.
[0015] Preferably, a computer-readable storage medium stores a computer program that, when executed by a processor, implements a three-dimensional target detection and target extraction method.
[0016] As can be seen from the above technical solution, compared with the prior art, the present invention discloses a YOLO-SAM method, system, and storage medium for three-dimensional object detection and extraction, with the following beneficial effects: 1. High efficiency and lightweight: By pruning, quantizing and knowledge distilling the YOLO11 network, and combining it with the lightweight MobileSAM model, the number of model parameters and computational complexity are significantly reduced, and the inference speed is improved, enabling it to be efficiently deployed on edge computing devices with limited computing power.
[0017] 2. High accuracy and robustness: The polarization self-attention (PSA) mechanism introduced in the target detection stage effectively compensates for the accuracy loss caused by model compression; the target extraction stage utilizes MobileSAM's powerful zero-shot segmentation capability to achieve refined target extraction. This scheme effectively overcomes noise interference in complex scenes, ensuring the accuracy of detection and extraction.
[0018] 3. Technological Breakthrough: This invention breaks through the limitations of traditional two-dimensional detection. By using multimodal feature fusion and coordinate mapping, it constructs an extraction system from two-dimensional image features to three-dimensional spatial point clouds, providing an efficient and lightweight solution for three-dimensional target detection and extraction in complex scenes. Attached Figure Description
[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0020] Figure 1 The method flowchart provided by the present invention.
[0021] Figure 2 The SternYOLO-SAM-Lite algorithm framework diagram provided by this invention. Detailed Implementation
[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0023] like Figure 1 As shown in the figure, this invention discloses a YOLO-SAM method for 3D object detection and extraction, including: The three-dimensional point cloud data of the original target object is obtained and preprocessed to obtain the two-dimensional point cloud intensity map of the object, as shown in the following formula; ; The two-dimensional point cloud intensity map is input into the target detection module to detect the target object and output the bounding box coordinates of the target region. The two-dimensional point cloud intensity map and bounding box coordinates are used as prompt information and input into the target extraction module to accurately segment the target object and output a two-dimensional mask of the target region. The pixel coordinates of the two-dimensional mask are converted into three-dimensional coordinates, and the corresponding points are extracted from the three-dimensional point cloud data of the original target object based on the three-dimensional coordinates to obtain the final target three-dimensional point cloud.
[0024] Specifically, the preprocessing of the original object's 3D point cloud data includes coordinate correction and dimensionality reduction; the dimensionality reduction operation projects the 3D point cloud data onto a 2D plane and generates a corresponding 2D point cloud intensity map based on the point cloud's reflection intensity information.
[0025] In a specific embodiment of the present invention, the data preprocessing process includes: First, 3D point cloud data of the original target object is acquired using sensors such as LiDAR. To facilitate subsequent processing, the 3D point cloud data needs to be preprocessed, including coordinate correction and dimensionality reduction. Dimensionality reduction projects the 3D point cloud onto a 2D plane and generates a 2D point cloud intensity map based on information such as the point cloud's reflection intensity. This map preserves key geometric and physical features while significantly reducing the data dimensionality, laying the foundation for rapid processing of the subsequent lightweight model.
[0026] Specifically, the target detection module uses the YOLO11-PQDA model. The model is optimized through pruning, quantization, knowledge distillation, and attention mechanisms. The optimized YOLO11-PQDA model is then used to perform preliminary localization of the target object and output the bounding box coordinates. The formula is as follows: .
[0027] In the formula, Asp(X) represents the output feature after processing by the polarization space attention mechanism; X represents the input intermediate feature tensor. Wq and Wv represent the 1×1 convolution weights used to generate query and value features, respectively; FGP stands for Global Pooling Operation; FSM stands for Softmax function; FSG stands for Sigmoid activation function; σ1, σ2, and σ3 represent tensor reshaping operations used for feature dimension transformation.
[0028] In a specific embodiment provided by this invention, the target detection module is a deep optimization of YOLO11 for edge computing scenarios, the core of which lies in the closed-loop strategy of "compression-compensation": Compression: Reduce computation nodes through structured pruning (such as channel pruning based on L1 norm), compress floating-point parameters into integers using weighted quantization to reduce model size, and transfer general knowledge from large teacher models to lightweight student models through knowledge distillation.
[0029] Compensation: To compensate for the potential decrease in accuracy caused by the compression operation, a polarization self-attention (PSA) module is innovatively embedded. This module can more effectively capture long-range dependencies between features and compensate for the loss of feature expressive power.
[0030] After processing, this module outputs the preliminary positioning information of the target object on the two-dimensional intensity map, namely the bounding box coordinates.
[0031] Specifically, the target extraction module, based on the zero-shot transfer characteristic of the MobileSAM model, accurately segments the target object with bounding box cues and generates a high-quality two-dimensional segmentation mask. The formula is as follows:
[0032] In the formula, SAM represents the semantic similarity measure between image features and cue features; X represents the target image feature vector extracted by the image encoder; Y represents the cue feature vector generated by the cue encoder; Indicates the transpose of Y; and Let X and Y represent the modulus (norm) of vectors X and Y, respectively. The inverse cosine function is used to calculate the angle between feature vectors to determine similarity.
[0033] In another specific embodiment provided by the present invention, the two-dimensional intensity map and the output bounding box coordinates are sent together to the target extraction module. The original object in this embodiment is the stern of a ship.
[0034] This embodiment uses the MobileSAM model, a lightweight segmentation model optimized for mobile and edge devices. Its core advantage lies in its powerful zero-shot segmentation capability, meaning that without fine-tuning based on specific stern data, it can accurately segment the target object within the bounding box with just a cue (in this example, the bounding box). The MobileSAM model can accurately delineate the outline of the stern and generate a high-quality binary 2D mask, where the target area is 1 and the background is 0.
[0035] Specifically, a pre-defined coordinate transformation matrix converts the pixel coordinates of the two-dimensional mask into three-dimensional coordinates. Based on the transformed three-dimensional coordinates, all points constituting the target object are extracted from the original three-dimensional point cloud data of the target object to obtain the final target three-dimensional point cloud.
[0036] In one specific embodiment of this invention, using the coordinate transformation matrix determined during the data preprocessing stage, the pixel coordinates of all pixels with a value of 1 in the generated two-dimensional mask are mapped back to the original three-dimensional point cloud space. By searching for these mapped three-dimensional coordinates, all points constituting the target object can be accurately extracted from the original three-dimensional point cloud. Ultimately, the output is a clean and accurate three-dimensional point cloud containing only the structure of the target object, which can be used for subsequent advanced tasks such as pose calculation.
[0037] Specifically, such as Figure 2 As shown, a YOLO-SAM-based 3D object detection and extraction system includes: Data preprocessing module: used to acquire the original 3D point cloud data of the target object and preprocess it to obtain the 2D point cloud intensity map of the object; Target detection module: Detects target objects based on 2D point cloud intensity maps and outputs the bounding box coordinates of the target region; Target extraction module: Uses the 2D point cloud intensity map and bounding box coordinates as prompt information to accurately segment the target object and output a 2D mask of the target region; 3D extraction module: Converts the pixel coordinates of the 2D mask into 3D coordinates, and extracts the corresponding points from the 3D point cloud data of the original target object based on the 3D coordinates to obtain the final target 3D point cloud.
[0038] Specifically, a computer-readable storage medium stores a computer program that, when executed by a processor, implements a method for three-dimensional target detection and extraction.
[0039] In summary, this invention, through an innovative end-to-end processing flow, achieves lightweight and high efficiency algorithms while ensuring high accuracy, providing a reliable and efficient technical solution for real-time detection and extraction of 3D targets in complex environments such as ports.
[0040] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.
[0041] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A YOLO-SAM method for 3D object detection and extraction, characterized in that, include: The three-dimensional point cloud data of the original target object is acquired and preprocessed to obtain the two-dimensional point cloud intensity map of the object. The two-dimensional point cloud intensity map is input into the target detection module to detect the target object and output the bounding box coordinates of the target region. The two-dimensional point cloud intensity map and bounding box coordinates are used as prompt information and input into the target extraction module to accurately segment the target object and output a two-dimensional mask of the target region. The pixel coordinates of the two-dimensional mask are converted into three-dimensional coordinates, and the corresponding points are extracted from the three-dimensional point cloud data of the original target object based on the three-dimensional coordinates to obtain the final target three-dimensional point cloud.
2. The YOLO-SAM method for three-dimensional object detection and extraction according to claim 1, characterized in that, The preprocessing of the original object's 3D point cloud data includes coordinate correction and dimensionality reduction. The dimensionality reduction operation projects the 3D point cloud data onto a 2D plane and generates a corresponding 2D point cloud intensity map based on the reflection intensity information of the point cloud.
3. The YOLO-SAM method for three-dimensional object detection and extraction according to claim 1, characterized in that, The target detection module uses the YOLO11-PQDA model. The YOLO11-PQDA model is optimized by pruning, quantization, knowledge distillation and attention mechanism. The optimized YOLO11-PQDA model is used to perform preliminary localization of the target object and output the bounding box coordinates.
4. The YOLO-SAM method for three-dimensional object detection and extraction according to claim 1, characterized in that, The target extraction module, based on the zero-shot transfer characteristic of the MobileSAM model, accurately segments the target object with bounding box cues and generates a high-quality two-dimensional segmentation mask.
5. The YOLO-SAM method for three-dimensional object detection and extraction according to claim 1, characterized in that, A pre-defined coordinate transformation matrix converts the pixel coordinates of a two-dimensional mask into three-dimensional coordinates. Based on the transformed three-dimensional coordinates, all points constituting the target object are extracted from the original three-dimensional point cloud data of the target object to obtain the final target three-dimensional point cloud.
6. A YOLO-SAM-based three-dimensional object detection and extraction system, characterized in that, include: Data preprocessing module: used to acquire the original 3D point cloud data of the target object and preprocess it to obtain the 2D point cloud intensity map of the object; Target detection module: Detects target objects based on 2D point cloud intensity maps and outputs the bounding box coordinates of the target region; Target extraction module: Uses the 2D point cloud intensity map and bounding box coordinates as prompt information to accurately segment the target object and output a 2D mask of the target region; 3D extraction module: Converts the pixel coordinates of the 2D mask into 3D coordinates, and extracts the corresponding points from the 3D point cloud data of the original target object based on the 3D coordinates to obtain the final target 3D point cloud.
7. A computer-readable storage medium storing a computer program, characterized in that, When the program is executed by the processor, it implements the three-dimensional target detection and target extraction method according to any one of claims 1-5.