Micro-target detection and localization method based on lidar

Abstract

This invention discloses a micro-target detection and localization method based on lidar, belonging to the field of lidar micro-target detection technology. The method includes acquiring first data of the micro-target to be detected and preprocessing the first data; constructing a multi-branch deep learning model based on the preprocessed first data and extracting features from the first data; performing detection and material classification based on the feature extraction results; and pre-setting a dynamic feature library for online learning of unknown materials based on the feature extraction results. This invention significantly improves the accuracy of micro-target detection and material classification by constructing a multi-branch deep learning model that integrates geometric and spectral features. With the help of a dynamic feature library and online learning mechanism, it achieves automatic identification of unknown materials. A dedicated feature extraction branch is designed for multi-wavelength lidar data to fully exploit material-specific information, providing a high-precision, highly adaptable, and low-maintenance-cost micro-target detection and localization solution.

Description

Technical Field

[0001] This invention relates to the field of micro-target detection technology, and in particular to a micro-target detection and localization method based on lidar. Background Technology

[0002] With the development of industrial automation, intelligent equipment and remote sensing technology, the demand for micro-target detection and positioning in high-precision manufacturing, autonomous driving, and drone inspection is increasing. As a high-precision active remote sensing device, lidar can acquire the three-dimensional coordinate information and multi-wavelength reflection intensity data of the target, and has become one of the core technologies for micro-target detection. By analyzing the spatial structure features and spectral reflection characteristics in point cloud data, it is possible to determine the existence of micro-targets, classify their materials and accurately locate them.

[0003] However, the common solutions currently available have many drawbacks, including: traditional methods often use a single branch network to process point cloud data, which cannot simultaneously integrate the spatial structure features and spectral material features of micro-targets. When relying solely on geometric features, it is easy to miss tiny targets with very few point clouds and cannot distinguish targets with similar materials but similar geometric structures. When relying solely on spectral features, it is difficult to locate the spatial position of the target and is significantly affected by lidar noise. Summary of the Invention

[0004] The purpose of this section is to outline some aspects of embodiments of the present invention and to briefly describe some preferred embodiments. Simplifications or omissions may be made in this section, as well as in the abstract and title of this application, to avoid obscuring the purpose of these documents; however, such simplifications or omissions should not be construed as limiting the scope of the invention.

[0005] In view of the problems existing in the above-mentioned LiDAR-based micro-target detection and localization methods, this invention is proposed.

[0006] Therefore, the purpose of this invention is to provide a micro-target detection and localization method based on lidar. This method is applicable to solving the problems of traditional point cloud analysis, which only extracts geometric information such as three-dimensional coordinates and point cloud density, which has the risk of missing detection for micro-targets with sparse point clouds or complex spatial structures, does not effectively utilize multi-wavelength reflection intensity data, resulting in low material classification accuracy, and uses fixed weights or simple concatenation in some multi-modal fusion methods, which do not adaptively learn the complementary relationship between geometric and spectral features according to the characteristics of micro-targets, making it difficult to cope with the complex detection needs of multi-material and multi-scale targets.

[0007] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0008] In a first aspect, embodiments of the present invention provide a micro-target detection and localization method based on lidar, which includes acquiring first data of the micro-target to be tested and preprocessing the first data; constructing a multi-branch deep learning model based on the preprocessed first data and extracting features from the first data; performing detection and material classification according to the feature extraction results; and pre-setting a dynamic feature library to learn unknown materials online according to the feature extraction results.

[0009] As a preferred embodiment of the micro-target detection and localization method based on lidar described in this invention, the first data is point cloud data containing multi-wavelength reflection intensity and three-dimensional coordinates; the first data is acquired by multi-wavelength lidar; the multi-branch deep learning model is used to extract features from the first data; the dynamic feature library includes known material templates and unknown sample temporary storage areas.

[0010] As a preferred embodiment of the micro-target detection and localization method based on lidar described in this invention, the construction of a multi-branch deep learning model based on the preprocessed first data includes the following steps: dividing the data into geometric and spectral branches according to the data characteristics and clarifying the function of each branch; using PointNet point cloud network to extract geometric features in the geometric branch; using CNN to extract spectral material features in the spectral branch; and constructing a multi-branch deep learning model by fusing the features output by the geometric and spectral branches.

[0011] As a preferred embodiment of the lidar-based micro-target detection and localization method of the present invention, the specific formula of the multi-branch deep learning model is as follows:

[0012] ,

[0013] In the formula, The fused feature vector; These are the weighting coefficients for geometric features; Center point ; geometric eigenvectors; This is the spectral eigenvector.

[0014] As a preferred embodiment of the micro-target detection and localization method based on lidar described in this invention, the micro-target detection based on the feature extraction results includes the following steps: calculating the magnitude of the fused feature vector; the micro-target detection based on the feature extraction results includes the following steps: when the magnitude of the fused feature vector is greater than a first threshold, it indicates that the micro-target exists and that the target has significant geometric structural features and spectral anomaly features, the location of the target area is recorded, and the material classification process is triggered; when the magnitude of the fused feature vector is less than or equal to the first threshold, it indicates that the target does not exist, the current area is recorded as background, the material classification process is not triggered, the detection window is moved to the next area, and the feature extraction and threshold judgment are repeated.

[0015] As a preferred embodiment of the micro-target detection and localization method based on lidar described in this invention, the specific steps of the material classification process are as follows: Calculate the similarity between the spectral feature vector and the known material templates in the dynamic feature library; if the maximum similarity is greater than the second threshold, the material of the micro-target is a known material category, and the material classification result is directly output; if the maximum similarity is less than or equal to the second threshold, it indicates that the spectral features of the micro-target do not match the matching degree of all known material templates in the dynamic feature library, and cannot be accurately classified as any known material, indicating that the material is an unknown material, and the unknown material processing process is performed.

[0016] As a preferred embodiment of the micro-target detection and localization method based on lidar described in this invention, the specific steps of the unknown material processing procedure are as follows: extract the spectral feature vector and geometric features of the micro-target and mark them as unknown samples; add the unknown samples to the dynamic feature library and store them in partitions with known material templates; prompt the operator to manually confirm the unknown samples and label them with material tags; store the spectral feature vector and material tags corresponding to the labeled unknown samples in the known material template area of ​​the dynamic feature library.

[0017] Secondly, to further solve the above-mentioned technical problems, the present invention provides a micro-target detection and positioning system based on lidar, which includes: a data acquisition module for acquiring first data of the target to be tested and performing preprocessing; a model building module for building a multi-branch deep learning model and extracting features from the first data; a result processing module for performing detection and material classification based on the feature extraction results; and a model updating module for performing online learning on unknown materials based on the feature extraction results.

[0018] Thirdly, embodiments of the present invention provide a computer device, including a memory and a processor, wherein the memory stores a computer program, and the computer program, when executed by the processor, implements any step of the micro-target detection and localization method based on lidar as described in the first aspect of the present invention.

[0019] Fourthly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the micro-target detection and localization method based on lidar as described in the first aspect of the present invention.

[0020] The beneficial effects of this invention are as follows: By constructing a multi-branch deep learning model that integrates geometric and spectral features, this invention significantly improves the accuracy of micro-target detection and material classification, solving the problem of incomplete description of single-modal features; with the help of a dynamic feature library and online learning mechanism, it achieves automatic identification of unknown materials, greatly reducing the cost of manual annotation and shortening the adaptation cycle of new materials; by integrating feature threshold linkage detection and classification processes, it optimizes real-time processing efficiency and enhances robustness in complex environments; at the same time, a dedicated feature extraction branch is designed for multi-wavelength lidar data to fully explore material-specific information, providing a high-precision, highly adaptable, and low-maintenance-cost micro-target detection and positioning solution. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. Wherein:

[0022] Figure 1 This is a flowchart illustrating the implementation of the present invention in Example 1.

[0023] Figure 2 This is a dynamic adjustment diagram of the multi-branch deep learning model in Example 1. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Example 1

[0028] Reference Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides a micro-target detection and localization method based on lidar, including the following steps:

[0029] S1: Obtain the first data of the micro-target to be tested and preprocess the first data.

[0030] Specifically, the first data is point cloud data containing multi-wavelength reflection intensity and three-dimensional coordinate information.

[0031] Furthermore, the first data was acquired using a multi-wavelength lidar.

[0032] Furthermore, three-dimensional coordinate information supports the spatial positioning and existence determination of the target, while multi-wavelength reflection intensity supports material classification and feature differentiation extraction.

[0033] In this embodiment, the preprocessing of multi-wavelength reflection intensity is to eliminate the influence of noise and equipment differences, and enhance the relative reflection characteristics of materials to different wavelengths; the preprocessing of three-dimensional coordinate information is for spatial structure noise reduction and positioning accuracy optimization.

[0034] Preferably, by performing spatial structure noise reduction on the three-dimensional coordinate information and eliminating noise and equipment differences in multi-wavelength reflection intensity, interference factors in the data are reduced, making the point cloud data more accurately reflect the micro-target features. The preprocessing of the three-dimensional coordinates optimizes the positioning accuracy and supports the spatial positioning and existence judgment of the target. The preprocessing of multi-wavelength reflection intensity enhances the relative reflection characteristics of the material to different wavelengths, improves the reliability of material classification and feature differentiation extraction, and provides a high-quality data foundation for subsequent micro-target detection, recognition and other tasks. The preprocessing operation makes the data format more standardized, which is convenient for subsequent algorithms or models to process efficiently, and enhances data compatibility and task processing efficiency.

[0035] S2: Based on the preprocessed first data, construct a multi-branch deep learning model and extract features from the first data.

[0036] It should be noted that multi-branch deep learning models are used to perform feature extraction on the first data.

[0037] Tax collection.

[0038] Specifically, based on the preprocessed first data, the construction of a multi-branch deep learning model includes the following steps: dividing the data into geometric branches and spectral branches according to the data characteristics, and clarifying the function of each branch.

[0039] The geometric branch uses the PointNet point cloud network to extract geometric features, and the specific formula is as follows:

[0040] ,

[0041] In the formula, Center point Geometric feature vectors describe local spatial structure information; The three-dimensional coordinates of the center point are used to calculate the relative positional relationship with the nearest neighbor points; for The coordinates of the j-th nearest neighbor provide spatial distribution information of the local neighborhood; k is the number of nearest neighbors in the nearest neighbor algorithm, used to control the size of the local neighborhood. For the nearest neighbor algorithm, The set of k nearest neighbors.

[0042] The spectral branch uses CNN to extract spectral material features, and the specific formula is as follows:

[0043] ,

[0044] In the formula, M is the spectral feature vector used to distinguish materials; M is the number of spectral bands, which determines the dimension of the spectral features. The weight of the m-th band is used to adjust the contribution of each band to the final feature; is the reflectance of the m-th band, representing the reflectivity of an object at a given wavelength.

[0045] A multi-branch deep learning model is constructed by fusing the features output from the geometric branch and the spectral branch.

[0046] Specifically, the formula for a multi-branch deep learning model is as follows:

[0047] ,

[0048] In the formula, The fused feature vector concatenates all dimensions of geometric and spectral features. These are the weighting coefficients for geometric features, controlling the relative importance of the two types of features, and are automatically optimized during model training. Center point ; geometric eigenvectors; This is the spectral eigenvector.

[0049] It should be noted that if geometric features are more critical in the detection task, the model will automatically increase in size. If material classification is the focus, the model size will automatically decrease. This increases the weighting of spectral features.

[0050] It should be noted that PointNet is a deep learning network suitable for point cloud data, mainly used to extract geometric features, while CNN is a convolutional neural network, mainly used to extract spectral material features.

[0051] In this embodiment, point cloud data is disordered and sparse. PointNet can effectively capture three-dimensional geometric structures by aligning the relative coordinates of nearest neighbors. Spectral data has correlations between bands and spatial locality. The convolution operation of CNN can extract local spectral patterns through a sliding window, and the weights can adaptively learn the importance of different bands.

[0052] S3: Perform micro-target detection and material classification based on feature extraction results.

[0053] Preferably, micro-target detection based on feature extraction results includes the following steps:

[0054] Calculate the magnitude of the fused feature vector;

[0055] When the magnitude of the fused feature vector is greater than the first threshold, it indicates that a micro-target exists and that the target has significant geometric and spectral features. The location of the target area is recorded, and the material classification process is triggered.

[0056] When the magnitude of the fused feature vector is less than or equal to the first threshold, it indicates that the target does not exist. The current region is recorded as the background, the material classification process is not triggered, the detection window is moved to the next region, and the feature extraction and threshold judgment are repeated.

[0057] Specifically, the dynamic adjustment steps for the first threshold are as follows:

[0058] Continuously collect the most recent fusion feature data, calculate the average value and fluctuation range of the fusion feature data, and avoid the fixed threshold from becoming invalid due to environmental changes or changes in target characteristics by dynamically statistically analyzing real-time data.

[0059] Based on the average value and fluctuation range in real-time data, the detection threshold for the current frame is dynamically generated. When the data fluctuates greatly, the threshold is automatically increased to filter noise; when the data is stable, the threshold is kept moderate to balance the detection sensitivity.

[0060] If noise is mistakenly identified as a target for 5 consecutive frames, it indicates that the threshold is too low. The threshold will be automatically increased to tighten the detection standard and reduce false positives.

[0061] If a real target is missed for 5 consecutive frames, it means that the threshold is too high. The threshold will be automatically lowered to relax the detection standard and avoid missed detection.

[0062] It should be noted that the magnitude of each threshold adjustment should be limited to avoid frequent fluctuations in the detection results.

[0063] Specifically, the material classification process involves the following steps: calculating the similarity between the spectral feature vector and known material templates in the dynamic feature library, using the following formula:

[0064] ,

[0065] In the formula, The similarity between the current micro-target's spectral characteristics and the template of the i-th known material type; The spectral feature vector of the micro-target is extracted by a CNN network with a spectral branch. is the template vector of the i-th known material in the dynamic feature library.

[0066] If the maximum similarity If the value is greater than the second threshold, the material of the micro-target is a known material category, and the material classification result is output directly.

[0067] If the maximum similarity If the value is less than or equal to the second threshold, it indicates that the spectral characteristics of the micro-target do not match the matching degree of any known material templates in the dynamic feature library, and cannot be accurately classified as any known material. This indicates that the material is an unknown material, and the unknown material processing procedure is performed.

[0068] It should be noted that the second threshold is obtained by setting it based on the similarity characteristics of known material templates, combined with the accuracy requirements of the classification task.

[0069] Preferably, micro-target detection is performed by fusing geometric and spectral features, and a dynamic threshold adjustment mechanism is used to adapt to environmental changes, which effectively improves detection robustness and reduces false negatives and false negatives; material classification uses spectral features as the primary weight and combines geometric auxiliary judgment to improve the accuracy of material identification.

[0070] S4: Preset dynamic feature library, which learns about unknown materials online based on feature extraction results.

[0071] Preferably, the specific steps of the unknown material processing procedure are as follows:

[0072] Extract the spectral feature vector and geometric features of the micro-target and label it as an unknown sample;

[0073] Add unknown samples to the dynamic feature library and store them separately from known material templates;

[0074] The operator is prompted to manually confirm the unknown sample and label the unknown sample with a material label.

[0075] The spectral feature vectors and material labels corresponding to the labeled unknown samples are stored in the known material template area of ​​the dynamic feature library.

[0076] Preferably, the spectral and geometric features of unknown samples are automatically stored and managed in partitions. New materials are incorporated into known templates through manual annotation and incremental training, so that the model can adapt to the detection requirements of new materials without offline retraining. This mechanism significantly reduces the cost of manual annotation, shortens the adaptation cycle of new materials, and improves the model's generalization ability by continuously expanding the feature library. It is especially suitable for industrial detection scenarios where the types of materials are frequently updated.

[0077] For example, point cloud data containing three-dimensional coordinates and dual-band reflection intensity is acquired by multi-wavelength lidar. After three-dimensional noise reduction and spectral reflectance enhancement preprocessing, the data is input into a multi-branch model composed of PointNet geometric branches and CNN spectral branches, and a comprehensive feature vector is generated by fusion. Micron-sized metal particles are accurately detected by dynamically adjusting the first threshold, and material classification is achieved based on cosine similarity. When unknown copper-nickel alloy particles are detected, the features are automatically stored, manually labeled, and the dynamic feature library is updated, triggering online learning of the model. This shortens the adaptation cycle of new materials from 2 weeks in traditional methods to 2 hours, and improves the particle detection rate and material classification accuracy to 96% and 95%, respectively, significantly enhancing the efficiency of defect detection and process traceability in semiconductor manufacturing.

[0078] In summary, this invention significantly improves the accuracy of micro-target detection and material classification by constructing a multi-branch deep learning model that integrates geometric and spectral features, solving the problem of incomplete description by single-modal features. Utilizing a dynamic feature library and online learning mechanism, it achieves automatic identification of unknown materials, greatly reducing manual annotation costs and shortening the adaptation cycle for new materials. By integrating feature thresholds to link detection and classification processes, it optimizes real-time processing efficiency and enhances robustness in complex environments. Furthermore, a dedicated feature extraction branch is designed for multi-wavelength LiDAR data to fully exploit material-specific information, providing a high-precision, highly adaptable, and low-maintenance micro-target detection and positioning solution.

[0079] Example 2

[0080] This embodiment also provides a micro-target detection and localization system based on LiDAR, including: a data acquisition module for acquiring first data of the target to be tested and performing preprocessing; a model building module for building a multi-branch deep learning model and extracting features from the first data; a result processing module for performing detection and material classification based on the feature extraction results; and a model update module for online learning of unknown materials based on the feature extraction results.

[0081] Example 3

[0082] This embodiment provides a computer device applicable to the micro-target detection and localization method based on lidar, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the micro-target detection and localization method based on lidar as proposed in the above embodiment.

[0083] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0084] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the micro-target detection and localization method based on lidar as proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0085] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A micro-target detection and localization method based on lidar, characterized in that: include: Acquire the first data of the micro-target to be tested, and preprocess the first data; Based on the preprocessed first data, a multi-branch deep learning model is constructed and features are extracted from the first data; Based on the preprocessed first data, constructing a multi-branch deep learning model includes the following steps: Based on the data characteristics, divide the data into geometric and spectral branches, and clarify the function of each branch; The geometric branch uses the PointNet point cloud network to extract geometric features; The spectral branch utilizes CNN to extract spectral material features; By fusing the features output from the geometric branch and the spectral branch, a multi-branch deep learning model is constructed. The specific formula for the multi-branch deep learning model is as follows: In the formula, The fused feature vector; These are the weighting coefficients for geometric features; Center point ; geometric eigenvectors; It is a spectral eigenvector; Detection and material classification are performed based on the feature extraction results; Micro-target detection based on the feature extraction results includes the following steps: Calculate the magnitude of the fused feature vector; When the magnitude of the fused feature vector is greater than the first threshold, it indicates that a micro-target exists and that the target has significant geometric structural features and spectral anomaly features. The location of the target area is recorded, and the material classification process is triggered. When the magnitude of the fused feature vector is less than or equal to the first threshold, it indicates that the target does not exist. The current region is recorded as the background, the material classification process is not triggered, the detection window is moved to the next region, and the feature extraction and threshold judgment are repeated. A preset dynamic feature library is used to learn about unknown materials online based on the feature extraction results; The specific steps of the material classification process are as follows: Calculate the similarity between the spectral feature vector and the known material templates in the dynamic feature library; If the maximum similarity is greater than the second threshold, then the material of the micro-target is a known material category, and the material classification result is directly output. If the maximum similarity is less than or equal to the second threshold, it indicates that the spectral features of the micro-target do not match the matching degree of all known material templates in the dynamic feature library, and cannot be accurately classified as any known material. This indicates that the material is an unknown material, and the unknown material processing procedure is performed.

2. The micro-target detection and localization method based on lidar as described in claim 1, characterized in that: The first data is point cloud data containing multi-wavelength reflection intensity and three-dimensional coordinates; The first data was acquired using a multi-wavelength lidar. The multi-branch deep learning model is used to extract features from the first data; The dynamic feature library includes known material templates and a temporary storage area for unknown samples.

3. The micro-target detection and localization method based on lidar as described in claim 1, characterized in that: The specific steps of the unknown material processing procedure are as follows: Extract the spectral feature vector and geometric features of the micro-target and label it as an unknown sample; Add unknown samples to the dynamic feature library and store them separately from known material templates; The operator is prompted to manually confirm the unknown sample and label the unknown sample with a material label. The spectral feature vectors and material labels corresponding to the labeled unknown samples are stored in the known material template area of ​​the dynamic feature library.

4. A micro-target detection and localization system based on lidar, based on the lidar-based micro-target detection and localization method according to any one of claims 1 to 3, characterized in that: include, The data acquisition module is used to acquire the first data of the target under test and perform preprocessing. The model building module is used to build multi-branch deep learning models and extract features from the first data. The results processing module is used for detection and material classification based on the feature extraction results; The model update module is used to learn about unknown materials online based on the feature extraction results.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the micro-target detection and localization method based on lidar as described in any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the micro-target detection and localization method based on lidar as described in any one of claims 1 to 3.

Images