Abnormal target detection method based on point cloud and image weighted fusion
Through the detection method based on point cloud and image weighted fusion, combined with fit curve denoising and deep learning algorithm, the problem of high false alarm rate of foreign object detection in bad weather is solved, and high accuracy and low error under harsh conditions are achieved Foreign object detection of the rate of reporting.
Patent Information
- Application Number
- CN202411096676.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-29
AI Technical Summary
In the prior art, in severe weather conditions, the false alarm rate of foreign matter detection is high, the detection rate is low, and the real-time detection efficiency is low, making it difficult to achieve reliable monitoring in all time periods and all climatic conditions.
Using an abnormal object detection method based on point cloud and image weighted fusion, weighted confidence calculation is performed to improve detection accuracy by pre-processing of lidar point cloud data and video image data, fitting curve denoising, PointCNN and YOLOv7 algorithms. and reduce false positive rate.
Effectively remove noise, improve foreign object detection accuracy and accuracy, reduce false alarm rate, and realize reliable monitoring and real-time detection under severe weather conditions.
Smart Images

Figure CN119048737A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of radar detection and video detection technology, and in particular to foreign object detection and recognition technology, and more specifically to an abnormal target detection method based on weighted fusion of point cloud and image. Background Art
[0002] Foreign object detection and recognition is widely used in the fields of autonomous driving, rail transportation, etc. Radar detection and video detection are two commonly used detection methods and have been widely used in practice.
[0003] At present, in the field of rail transit, the risk of foreign body intrusion is generally prevented and controlled by combining track rolling vehicle inspection, manual inspection and video surveillance. The inspection efficiency is low, and it is impossible to achieve real-time detection, with a certain time lag. At the same time, at night, in rain, fog and other weather conditions with low visibility, it is easy to have false detection and missed detection.
[0004] Based on the above problems, the industry has proposed a method for detecting foreign objects on the track surface based on the fusion of LiDAR and video acquisition. By fusing LiDAR point cloud data with video data, combined with big data analysis and foreign object recognition algorithms, reliable monitoring of foreign objects on the track surface under all time periods and all weather conditions can be achieved. When risks occur, they can be detected in real time and an alarm can be generated to the monitoring platform to reduce the harm caused by the risk of foreign object intrusion. This system will provide important support for the operation and maintenance and safety of the subway.
[0005] There are many existing methods and technologies for detecting foreign objects based on images or lidar, but they generally have problems such as high false alarm rate and significant impact from weather. That is, under conditions of heavy rain, heavy fog, hail, and heavy snow, the system detection rate will be seriously reduced and a large number of false alarms may be generated. Summary of the invention
[0006] The main purpose of this application is to provide an abnormal target detection method based on weighted fusion of point cloud and image, so as to solve the problem that the foreign object detection method in the prior art is greatly affected by weather and has a high false alarm rate.
[0007] In order to achieve the above object, according to one aspect of the specific implementation of the present application, a method for detecting abnormal targets based on weighted fusion of point cloud and image is provided, characterized in that it includes the following steps:
[0008] a. Collect the laser radar point cloud data and video image data at the same time, process the laser radar point cloud data and video image data, and remove noise in the data;
[0009] b. Set the fitting curve, find the global isolated points in the point cloud data by finding the deviation through the fitting curve, and filter them out;
[0010] c. The point cloud data and image data after the above preprocessing are used as input, and the PointCNN model and YOLOv7 algorithm are used for detection respectively. The two-dimensional foreign body features in the video image extracted by the YOLOv7 algorithm and the three-dimensional foreign body features in the point cloud extracted by the PointCNN model are weightedly calculated;
[0011] d. A weighted confidence of the final detection result is obtained based on the weighted calculation result. When the weighted confidence is greater than a set threshold, it indicates that a foreign object has been detected.
[0012] In some embodiments, in step a, lidar point cloud data and video image data are collected at equal periods.
[0013] In some embodiments, the fitting curve function expression is z=f(x, y; θ), where θ is a parameter vector, and x, y, z are spatial rectangular coordinate systems.
[0014] In some embodiments, the fitting curve is a linear function, a polynomial function, a high-order function, or a radial basis function.
[0015] In some embodiments, step b comprises the following steps:
[0016] b1. Define the loss function z i is the actual observed value of the ith value; f(x i ,y i ; θ) the i-th fitting function value; i = 1, 2...n, n is the total number of data;
[0017] b2. Solve the parameter vector θ to minimize the loss function L(θ);
[0018] b3. Calculate the residual e of each data point i =z i -f(x i ,y i ;θ), e i It reflects the deviation between the fitting surface and the actual data;
[0019] b4, remove e i >T, and obtain the preprocessed point cloud data, where T is the set threshold.
[0020] In some embodiments, in step b2, the least squares method is used to solve the parameter vector θ so that the loss function L(θ) is minimized.
[0021] In certain embodiments, T=0.1.
[0022] In some embodiments, step d specifically includes: using a weighting function to perform superposition calculation on the detection confidences output by the two models to obtain a weighted confidence of the final detection result.
[0023] In some embodiments, the weighting function is expressed as:
[0024]
[0025] Among them, λ∈(0,1) represents the weight of the point cloud features during fusion, P is the overall confidence, and α i , β j They are the confidence levels of PointCNN and YOLOv7 outputs respectively; i = 1, 2, ..., k, where k is the confidence level α i The total number of; j = 1, 2...m, m is the confidence level β j The total number of .
[0026] In some embodiments, λ=0.7 is set, and when the overall confidence level P>0.7, it indicates that a foreign object is detected.
[0027] According to the technical solution of the present application and the further improved technical solution in certain exemplary embodiments thereof, the present application has the following beneficial effects:
[0028] 1) Different from the prior art that directly uses lidar point cloud data for detection, or uses Gaussian filters, mean filters, median filters, bilateral filters, etc. for filtering, this application preprocesses the input lidar point cloud data, and uses a fitting curve to calculate the deviation to remove discrete isolated noise points in the scene. It can effectively remove discrete noise generated by clouds, fog, rain, snow, etc., and will not cause loss of foreign object features, greatly improving the precision and accuracy of foreign object detection.
[0029] 2) Different from the existing technology, which uses traditional detection methods (background modeling to find foreign objects), this application uses a convolutional neural network (PointCNN) to identify foreign objects, which can explore the deep feature information in the data and accurately find the location of foreign objects.
[0030] 3) Different from the existing method of using a single technical means to detect foreign objects, this method commonly uses a comprehensive analysis of laser radar and video sensors to perform weighted confidence calculations on the output results to ensure accuracy while reducing the false alarm rate.
[0031] The present application is further described below in conjunction with the accompanying drawings and specific embodiments. Additional aspects and advantages of the present application will be partially given in the following description, partially become apparent from the following description, or be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The drawings constituting a part of this application are used to provide a further understanding of this application. The specific implementation modes, illustrative embodiments and their descriptions of this application are used to explain this application and do not constitute an improper limitation on this application. In the drawings:
[0033] Figure 1 It is a schematic diagram of the process of a foreign body detection method according to a specific implementation method of the present application;
[0034] Figure 2 This is a schematic diagram comparing the number of false positives of the detection method (fusion algorithm) of this application and other detection methods;
[0035] Figure 3 This is a schematic diagram comparing the detection quantity of the detection method (fusion algorithm) of this application and other detection methods. DETAILED DESCRIPTION
[0036] It should be noted that, in the absence of conflict, the specific implementation modes, exemplary embodiments and features thereof in this application can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and in combination with the following content.
[0037] In order to enable those skilled in the art to better understand the solution of this application, the following will be combined with the drawings in the specific implementation methods and exemplary embodiments of this application to clearly and completely describe the technical solutions in the specific implementation methods and exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of them. Based on the specific implementation methods and exemplary embodiments in this application, all other implementation methods and embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of this application.
[0038] PointCNN and YOLO involved in this application are both commonly used neural network models, which are used for lidar point cloud data processing and video image processing respectively.
[0039] PointCNN is a data processing model that applies standard convolution to point clouds.
[0040] YOLO is an object detection algorithm. It is a deep learning method that redefines the object detection task as a regression problem. YOLOv7 is one of its different versions.
[0041] Example
[0042] The following describes the technical solution of the present application in detail by taking foreign object detection on rail transit lines as an example.
[0043] The process of foreign body detection based on radar data and video data in this example is as follows Figure 1 shown.
[0044] S101 collects the lidar point cloud data and video image data at the same time, processes the lidar point cloud data and video image data, and removes noise in the data.
[0045] This example uses an integrated semi-solid laser radar and a starlight-level camera to collect point cloud data and video image data of the track area in real time.
[0046] In this step, firstly, the lidar point cloud and camera video image data at the same time are collected in an equal periodic manner (preferably 3 seconds in this example), and the point cloud data and image data are preprocessed by Gaussian filtering to remove noise in the data.
[0047] Point cloud data can be recorded as p n =[x n ,y n ,z n ,r n ], where x n ,y n 、z n is the coordinate information of the point cloud data, r n is the reflectivity, n is the data number. n All data in , removing r n = 0, and use the remaining data as the original data.
[0048] S102 sets a fitting curve, finds global isolated points in the point cloud data by calculating deviations through the fitting curve, and filters them out.
[0049] The main function of this step is to remove discrete noise. By fitting the curve to find the deviation, the global isolated points in the point cloud data are found and filtered out.
[0050] Since the noise clouds generated by rain, snow, clouds and fog are relatively discrete, while the point clouds generated by foreign objects are relatively concentrated, filtering out noise points will not cause loss of foreign object features.
[0051] This example calculates the distance between each data point and other points, and then finds points that are far away from other points as isolated points based on the set threshold. This process can be achieved through matrix expressions. The specific steps are as follows:
[0052] 1. Set the function form of the fitting surface, which can be a linear function, a polynomial function, a high-order function, a radial basis function, etc. Assume that the selected fitting function is z=f(x, y; θ) where θ is a parameter vector.
[0053] 2. Define the loss function z i is the actual observed value of the ith value; f(x i ,yi ;θ) the i-th fitting value, i = 1, 2...n, n is the total number of data. i and the fitted value f(x i ,y i ; θ) is used as the loss function.
[0054] 3. Solve the parameter vector θ so that the loss function L(θ) is minimized. Here, the least squares method or other optimization algorithms can be used to solve the parameter vector θ.
[0055] 4. Calculate the residual e for each data point i =z i -f(x i ,y i ;θ), e i It reflects the deviation between the fitted surface and the actual data.
[0056] 5. Determine whether there are isolated points based on the residual value, that is, remove e i >T, and obtain the preprocessed point cloud p n '. Wherein T is the set threshold (in this example, T = 0.1 is preferred).
[0057] This step can effectively remove discrete noise in the point cloud data without causing loss of foreign body features. It is particularly suitable for preprocessing of lidar point cloud data. It can effectively avoid the interference of bad weather such as rain, snow, and fog on lidar data collection, improve the validity of the data and the reliability of foreign body detection, and play a very important role in reducing the false alarm rate.
[0058] S103, taking the point cloud data and image data after the above preprocessing as input, using the PointCNN model and the YOLOv7 algorithm for detection respectively, and performing weighted calculation on the two-dimensional foreign body features in the video image extracted by the YOLOv7 algorithm and the three-dimensional foreign body features in the point cloud extracted by the PointCNN model.
[0059] In this example, the preprocessed point cloud data p n ' and image data as input, and use PointCNN model and YOLOv7 model for detection respectively. YOLOv7 algorithm is used to extract two-dimensional foreign body features in video images and PointCNN is used to extract point cloud data p n The three-dimensional foreign body features in the 'are weighted and calculated. By optimizing the weights, the accuracy of the detection method of the present application is improved.
[0060] In this example, according to the characteristics of the rail transit system, the PointCNN detection process first uses PointCNN to segment the point cloud data p nThe track, tunnel entrance, and overhead wire in the ' are of high confidence due to sufficient data samples. n ' removes the point cloud data of track, tunnel entrance and contact network to obtain the point cloud data p n '. Use Open3D clustering function to cluster the point cloud data n In '', points with intervals less than a set value (such as 0.1 meters) are clustered to obtain the cluster point set b n . n The clustered point sets in are input into PointCNN one by one for recognition to further improve the accuracy.
[0061] The weighted calculation in this example is to calculate the confidence of the detection results output by the PointCNN model and the YOLOv7 algorithm, and use the weighted function to perform superposition calculation to obtain the weighted confidence of the final detection result. The weighted function in this example is expressed as:
[0062]
[0063] Among them, λ∈(0,1) represents the weight of the point cloud features during fusion, P is the overall confidence, and α i , β j They are the confidence levels of PointCNN and YOLOv7 outputs respectively; i = 1, 2, ..., k, where k is the confidence level α i The total number of; j = 1, 2...m, m is the confidence level β j The total number of .
[0064] S104, obtaining a weighted confidence of the final detection result according to the weighted calculation result, and when the weighted confidence is greater than a set threshold, it indicates that a foreign object is detected.
[0065] In this example, λ=0.7 in formula (1) is preferred, which means that the output result of PointCNN is trusted, and when the overall confidence P>0.7, it means that a foreign object is detected and the system outputs an alarm signal.
[0066] See also Figure 2 and Figure 3 In this embodiment, laser radar and camera are used to collect point cloud data and video image data of the track section, and PointCNN and YOLOv7 are used for fusion detection. Compared with the traditional detection method, the number of false positives is significantly reduced. Figure 2 and as shown in Table 1. At the same time, the foreign matter detection effect of the detection method of the present application is significantly improved, such as Figure 3 And as shown in Table 2.
[0067] Table 1
[0068] date Fusion Algorithm YOLOv5 PointCNN Day 1 37 123 53 Day 2 35 117 50 Day 3 31 103 44 Day 4 38 127 54 Day 5 32 107 46 Day 6 33 110 47 Day 7 17 57 24 Day 8 16 53 23 Day 9 12 40 17 Day 10 0 30 67 Day 11 0 22 20 Day 12 0 49 81 Day 13 0 10 46 Day 14 0 25 65 Day 15 0 27 81 Day 16 0 45 80 Day 17 2 7 3 Day 18 1 3 1 Day 19 0 25 33 Day 20 0 30 47 Day 21 0 24 14 Day 22 0 35 43 Day 23 0 12 29 Day 24 0 29 16 Day 25 10 33 14 Day 26 11 37 16 Day 27 8 27 11 Day 28 7 23 10 Day 29 6 20 9 Day 30 3 10 4 Day 31 2 7 3 Day 32 2 7 3 Day 33 0 32 40 Day 34 0 24 38 Day 35 3 10 4 Day 36 0 42 26 Day 37 0 24 35
[0069] Table 2
[0070]
[0071]
Claims
1. A method for detecting abnormal targets based on weighted fusion of point cloud and image, characterized in that: The steps include: a. Collect the laser radar point cloud data and video image data at the same time, process the laser radar point cloud data and video image data, and remove noise in the data; b. Set the fitting curve, find the global isolated points in the point cloud data by finding the deviation through the fitting curve, and filter them out; c. The point cloud data and image data after the above preprocessing are used as input, and the PointCNN model and YOLOv7 algorithm are used for detection respectively. The two-dimensional foreign body features in the video image extracted by the YOLOv7 algorithm and the three-dimensional foreign body features in the point cloud extracted by the PointCNN model are weightedly calculated; d. A weighted confidence of the final detection result is obtained based on the weighted calculation result. When the weighted confidence is greater than a set threshold, it indicates that a foreign object has been detected.
2. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 1 is characterized in that: In step a, lidar point cloud data and video image data are collected periodically.
3. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 1 is characterized in that: The fitting curve function expression is z=f(x, y; θ), wherein θ is a parameter vector, and x, y, z are a spatial rectangular coordinate system.
4. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 3 is characterized in that: The fitting curve is a linear function, a polynomial function, a high-order function or a radial basis function.
5. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 3 is characterized in that: The step b comprises the following steps: b1. Define the loss function z i is the actual observed value of the ith value; f(x i ,y i ; θ) the i-th fitting value, i = 1, 2...n, n is the total number of data; b2. Solve the parameter vector θ to minimize the loss function L(θ); b3. Calculate the residual e of each data point i =z i -f(x i ,y i ;θ); b4, remove e i >T, and obtain the preprocessed point cloud data, where T is the set threshold.
6. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 5 is characterized in that: In step b2, the least squares method is used to solve the parameter vector θ so that the loss function L(θ) is minimized.
7. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 5 is characterized in that: T=0.1。 8. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 1 is characterized in that: Step d specifically includes: using a weighted function to perform superposition calculation on the detection confidences output by the two models to obtain a weighted confidence of the final detection result.
9. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 8, characterized in that: The weighting function is expressed as: Among them, λ∈(0,1) represents the weight of the point cloud features during fusion, P is the overall confidence, and α i , β j They are the confidence levels of PointCNN and YOLOv7 outputs respectively; i = 1, 2, ..., k, where k is the confidence level α i The total number of; j = 1, 2...m, m is the confidence level β j The total number of .
10. The abnormal target detection method based on weighted fusion of point cloud and image according to claim 9, characterized in that: When λ=0.7, the overall confidence level P>0.7 indicates that a foreign object is detected.
Citation Information
Cited By
Battery compartment foreign matter identification method and system based on laser scanning and AI identification
CN120747147A
