Camera and radar extrinsic parameter optimization method based on operation site moving target

Abstract

The embodiment of the invention discloses a camera and radar external parameter optimization method based on a working site moving target, and the method comprises the steps: collecting a continuous frame image and a point cloud, and extracting an image moving target and a point cloud moving target through an optical flow method and a frame difference method; projecting a moving target to an image by using an existing calibration matrix, performing rigid body transformation matching, and storing a matching result into M; image and point cloud edge features are extracted and obtained through matching; and obtaining an external parameter by using a PNP algorithm, calculating an error, outputting the error if the error reaches the standard, otherwise, rejecting the error and repeating feature matching and subsequent steps. According to the method, iterative optimization of the external parameter matrix is carried out by using the point cloud and image features of the moving target, the moving target with a large error is eliminated, camera and laser radar external parameter calibration based on any scene moving target is realized, the problems that calibration depends on a calibration board and external parameter correction cannot be carried out in the field use process are effectively solved, and the calibration accuracy is improved. And the measurement requirements of the transformer substation can be well met.

Description

Technical Field

[0001] This invention relates to the field of substation management technology, specifically to a method for optimizing camera and radar extrinsic parameters based on moving targets at the work site. Background Technology

[0002] In the safety management of substations, it is necessary to detect the distance between energized equipment and surrounding objects, which poses a risk of electric shock during the measurement process. Traditional manual measurement methods suffer from problems such as difficulty in reaching the measurement location, low efficiency, and low accuracy.

[0003] To improve measurement accuracy and efficiency while ensuring personnel safety, current methods primarily utilize the fusion of camera and lidar data to accurately measure the distance between charged conductors and surrounding objects. While current extrinsic parameter calibration methods for cameras and lidar can achieve a certain level of accuracy in the overall fusion of images and point clouds, errors in camera and lidar calibration inevitably occur with the use of the equipment. This can lead to significant deviations in the fusion of conductor point clouds and images, making it difficult to meet the requirements for measuring the distance between charged conductors and surrounding objects in large-scale substation scenarios. Summary of the Invention

[0004] To address the shortcomings of the aforementioned technologies, this invention provides a method for optimizing camera and radar extrinsic parameters based on moving targets at the work site.

[0005] This invention provides a method for optimizing camera and radar extrinsic parameters based on moving targets at a work site. The method includes the following steps: S1: The camera and lidar to be calibrated simultaneously acquire continuous frames of image and point cloud data, let the image set be... Point cloud set as The specific expression is: ; ;

[0006] The image data acquired in the nth acquisition is: Point cloud data is Each acquired image and point cloud is stored sequentially. and middle; S2: Extraction of the middle part using optical flow method i Images acquired in this session Moving targets in ; S3: Extracting the first frame using the frame difference method i Point cloud collected in the second time Moving targets in ; S4: Using the device's existing calibration matrix, project the point cloud of the moving target set onto the image. The projected point cloud is... ; S5: Matching using rigid body transformation method and And store the matched moving targets in ,Right now ; For all matched moving targets in the k acquisitions, i.e., moving targets in the image and point cloud. ,in , For the k-th group, match the moving targets in the image and the moving targets in the point cloud; S6: If The number of moving targets is less than the threshold If the condition is met, return to step S2; otherwise, iterate through the previous steps. Extracting using the Canny operator Image edge features are extracted using a normal estimation method. The point cloud edge features were analyzed, and the image was registered with the point cloud edge features using the Hungarian matching algorithm to obtain... The set of image and point cloud features of all moving targets in the dataset, i.e.: ; S7: Employs the PNP algorithm, utilizing... Perform extrinsic parameter calibration of the camera and lidar to obtain the extrinsic parameter calibration matrix. ; S8: Using the external parameter calibration matrix Will All moving targets in the point cloud are projected onto the image, and the Euclidean distance between them and the corresponding image features is calculated as the projection error. S9: Calculate the root mean square error of the projection error. If the root mean square error is less than the threshold... Then the extrinsic calibration matrix is ​​considered to be... For the extrinsic parameter calibration matrix of the camera and lidar, otherwise in Remove the moving target with the largest projection error and return to S6.

[0007] Preferably, step S2 includes the following process: S2.1: Preprocess the image by converting it into a grayscale image and using a Gaussian filtering algorithm to smooth out noise in the image; S2.2: Use the Sobel operator to calculate the grayscale value of each pixel in the grayscale image. x shaft and y The rate of change of grayscale along the axial direction yields the image. x shaft and y Spatial gradient matrix along the axis; S2.3: Using the first i The pixel value of the second captured image minus the first i- The pixel values ​​at corresponding locations in a single image capture are used to obtain the temporal gradient of each pixel. S2.4: Within a 3×3 neighborhood window of each pixel, calculate x Sum of squares of spatial gradients of orientation Gxx The specific expression is: ;

[0008] in For the image The grayscale value of the location; The product of the spatial gradients in the x-direction and y-direction, Gxy, is calculated using the following expression: ;

[0009] The sum of squares of the spatial gradients in the y-direction, Gyy, is calculated using the following expression: ;

[0010] Within the neighborhood window, calculate x The sum of the products of the spatial and temporal gradients in the y-direction, Gxt, and the sum of the products of the spatial and temporal gradients in the y-direction, Gyt, are: ; ;

[0011] in, To capture the time difference between two images; S2.5: Construct a system of linear equations, the specific expression of which is: ;

[0012] S2.6: Solve the system of linear equations to obtain the optical flow velocity of each pixel, thereby obtaining the moving target in the image. Then the... i The set of moving targets in the frame image is .

[0013] Preferably, step S3 includes the following process: S3.1: Point Cloud Perform filtering, using Gaussian filtering to remove noise points; S3.2: Calculating Point Clouds With point clouds The Euclidean distance of each point in space, if the distance is greater than a threshold Then this point is considered a point of difference; S3.3: After obtaining the difference points, perform connected component analysis on them, merging adjacent difference points into a single connected component; then cluster the difference points, grouping spatially similar difference points into one cluster, forming multiple clusters. Each cluster may correspond to a moving target. Therefore, the set of moving targets in the point cloud of the i-th frame is: .

[0014] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages: The present invention extracts moving targets from images and point clouds using optical flow and differential methods, and uses the point cloud and image features of the moving targets to iteratively optimize the extrinsic parameter matrix, eliminating moving targets with large errors, and realizing the extrinsic parameter calibration of cameras and lidar based on moving targets in any scene. This effectively solves the problem that calibration depends on calibration boards and that extrinsic parameter correction cannot be performed during field use, and can well meet the needs of measuring the distance between energized bodies and surrounding objects in large substation scenarios. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart of the camera and radar extrinsic parameter optimization method based on moving targets at the work site according to the present invention; Detailed Implementation

[0017] This invention provides a method and system for optimizing camera and radar extrinsic parameters based on moving targets at the work site. By collecting dynamic targets on site, the extrinsic parameters of the camera and lidar are optimized, providing a fast, high-precision, and highly reliable method for laser point cloud and image fusion for substation inspection sites.

[0018] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0019] Please see Figure 1 , Figure 1 The flowchart of this invention shows that this invention provides a method for optimizing camera and radar extrinsic parameters based on moving targets at a work site. The method includes the following steps: S1: The camera and lidar to be calibrated simultaneously acquire continuous frames of image and point cloud data, let the image set be... Point cloud set as The specific expression is: ; ;

[0020] The image data acquired in the nth acquisition is: Point cloud data is Each acquired image and point cloud is stored sequentially. and middle; The core of this step is to ensure the time synchronization and sequence consistency of the data acquired by the camera and LiDAR. This provides the foundational data support for subsequent cross-modal (image and point cloud) moving target association and matching. Only image and point cloud data acquired at the same time can accurately correspond to the same spatial state at the work site. During this step, the camera and LiDAR to be calibrated are started synchronously, acquiring n consecutive frames of data to construct an image set and a point cloud set, respectively. The image data acquired in the nth acquisition is... Point cloud data is Each acquired image and point cloud are stored in the corresponding set in the order of acquisition.

[0021] S2: Extraction of the middle part using optical flow method i Images acquired in this session Moving targets in ; In this step, optical flow is used to track the motion trajectories of pixels in consecutive frames of images. Based on the continuous change in pixel grayscale values ​​of the same object in adjacent frames, moving regions are distinguished from static backgrounds, thereby locating moving targets in the image. During runtime, the acquired image set is processed... The Middle i Images acquired in this session The optical flow method is used to calculate the displacement vector of pixels between adjacent frames. Regions with obvious motion characteristics are selected by filtering out the displacement vectors, and these regions are defined as moving targets in the image. .

[0022] S3: Extracting the first frame using the frame difference method i Point cloud collected in the second time Moving targets in ; In this step, the frame difference method calculates the spatial position difference between two adjacent frames of point cloud data and filters out point cloud clusters whose positions have changed. The point cloud of a static target remains basically unchanged in consecutive frames, while the point cloud of a moving target will undergo significant displacement with the frame sequence, thereby realizing the extraction of moving targets from the point cloud.

[0023] S4: Using the device's existing calibration matrix, project the point cloud of the moving target set onto the image. The projected point cloud is... ; In this step, the existing initial calibration matrix (describing the spatial pose relationship between the camera and the LiDAR) is used to convert the 3D point cloud data into coordinates on a 2D image plane through perspective projection, achieving spatial alignment between the point cloud and the image, preparing for subsequent target matching. During runtime, the existing calibration matrix is ​​invoked to match the point cloud data within the moving target set extracted in step S3. Projection operations are performed to convert the 3D point cloud coordinates into 2D coordinates on the image plane. The projected point cloud forms a set.

[0024] S5: Matching using rigid body transformation method and And store the matched moving targets in ,Right now ; For all matched moving targets in the k acquisitions, i.e., moving targets in the image and point cloud. ,in , For the k-th group, match the moving targets in the image and the moving targets in the point cloud; In this step, rigid body transformation describes the translational and rotational motion of an object in space without changing the shape and size of the target. Utilizing this property, the projected point cloud is adjusted through rigid body transformation. The pose of the target in the image is determined so that it is aligned with the moving target. The contours and positions on the two-dimensional plane are made to coincide as much as possible, thereby achieving cross-modal target matching. The calculation process in this step is as follows: The rigid body transformation method is used to transform the projected point cloud. With moving targets in the image Perform pose matching, filter out successfully matched target pairs, and store these target pairs in a set. ,Right now .in, This represents the target pair that was successfully matched in the k-th group, specifically... , and These represent the moving target in the image and the moving target in the point cloud, respectively, for the k-th matching group.

[0025] S6: If The number of moving targets is less than the threshold If the condition is met, return to step S2; otherwise, iterate through the previous steps. Extracting using the Canny operator Image edge features are extracted using a normal estimation method. The point cloud edge features were analyzed, and the image was registered with the point cloud edge features using the Hungarian matching algorithm to obtain... The set of image and point cloud features of all moving targets in the dataset, i.e.: ; In this step, edge features are among the most significant geometric features of the target. The Canny operator can efficiently extract continuous edges from the image, and normal estimation can locate point cloud edges through abrupt changes in point cloud normal vectors. The Hungarian matching algorithm, based on the principle of "maximizing feature similarity," achieves optimal pairing of image and point cloud edge features. During calculation, the set is first determined. Is the number of moving targets less than the threshold? If the value is less than the target, return to step S2 to extract the target again; if the value is not less than the target, then iterate through the target. Each set of target pairs Extraction using the Canny operator Image edge features are extracted using a normal estimation method. The point cloud edge features are then used to register the two types of edge features using the Hungarian matching algorithm, ultimately obtaining the image and point cloud feature set of all moving targets. .

[0026] In this step, the Canny operator, or Canny edge detection operator, is an edge detection algorithm widely used in image processing and computer vision, effectively achieving a good balance between noise suppression and edge localization accuracy. Normal estimation methods are fundamental techniques in computer graphics, 3D computer vision, and point cloud processing. Their purpose is to estimate a normal vector for each point on a geometric surface in 3D space (especially discrete point cloud data). This normal vector is perpendicular to the local surface at that point, reflecting the surface orientation and geometric features of that point. The Hungarian matching algorithm, also known as the Kuhn-Munkles algorithm, is a classic algorithm for solving the optimal matching problem in bipartite graphs, particularly suitable for assignment problems and minimum-cost / maximum-weight bipartite matching.

[0027] S7: Employs the PNP algorithm, utilizing... Perform extrinsic parameter calibration of the camera and lidar to obtain the extrinsic parameter calibration matrix. ; In this step, the PNP (Perspective-N-Point) algorithm can solve for the camera pose parameters by using the known correspondence between 3D points (point cloud features) and 2D points (image features). The feature set obtained in step S6 is used as a reference. Using the PNP algorithm as input data, the external parameter calibration matrices of the camera and LiDAR are obtained by establishing a spatial mapping relationship between 3D point cloud features and 2D image features. .

[0028] S8: Using the external parameter calibration matrix Will All moving targets in the point cloud are projected onto the image, and the Euclidean distance between them and the corresponding image features is calculated as the projection error. In this step, the accuracy of the extrinsic calibration matrix can be measured by the degree of agreement between the point cloud projection results and the image features: if the matrix accuracy is high, the distance between the projected point cloud position and the corresponding image feature will be small, and vice versa. Euclidean distance can quantify this difference. During calculation, the extrinsic calibration matrix obtained in step S7 is used. The moving targets of all point clouds in set M Project the image onto the image plane and calculate the Euclidean distance between each projection point and the corresponding image feature point. This distance is the projection error of a single feature pair.

[0029] S9: Calculate the root mean square error of the projection error. If the root mean square error is less than the threshold... Then the extrinsic calibration matrix is ​​considered to be... For the extrinsic parameter calibration matrix of the camera and lidar, otherwise in Remove the moving target with the largest projection error and return to S6.

[0030] In this step, the root mean square error (RMSE) comprehensively reflects the overall level of all projection errors. If the RMSE is less than the threshold... If the overall accuracy of the calibration matrix meets the requirements, then the abnormal target with the largest error (which may be a mismatched target) is removed, and feature registration and calibration are performed again to improve the matrix accuracy.

[0031] In some embodiments of the present invention, the camera may be an industrial area scan camera or a thermal imaging camera. Industrial area scan cameras, such as the Basler acA series, can be used to acquire visible light images of equipment appearance and switch status, and combined with optical flow methods to extract moving targets (such as inspection robots and operating tools). Thermal imaging cameras can identify potential equipment faults through temperature distribution at night or under complex lighting conditions, and can achieve higher-precision modeling by fusing with LiDAR point clouds. The LiDAR may be a mechanical multi-line LiDAR or a solid-state LiDAR. Mechanical multi-line LiDAR can be used to construct fine point cloud models of substation equipment for spatial positioning of moving targets (such as drones and cranes). Solid-state LiDAR has the advantages of light weight and small size, has no rotating parts, and can be easily installed on inspection robots or mobile platforms, maintaining stability under mechanical vibration environments.

[0032] Preferably, step S2 includes the following process: S2.1: Preprocess the image by converting it into a grayscale image and using a Gaussian filtering algorithm to smooth out noise in the image; The main purpose of this step is to simplify the data dimensions and suppress noise interference, laying a stable image foundation for subsequent gradient calculation and optical flow analysis. Grayscale conversion logic: The core of optical flow is to track motion through the spatiotemporal changes of pixel grayscale values. The RGB three-channel information of a color image is redundant and increases computational load. Converting the color image to a grayscale image preserves the brightness characteristics of pixels while reducing the dimensionality of the three-dimensional data to one dimension, significantly simplifying subsequent calculations. Gaussian filtering logic: Images from the work site are susceptible to dust, electromagnetic interference, etc., producing salt-and-pepper noise or random noise. This noise can cause abrupt changes in local grayscale, interfering with the accuracy of gradient calculation. Gaussian filtering, based on the principle of "weighted averaging of normal distribution," smooths the grayscale values ​​within the pixel's neighborhood, meaning points closer to the center pixel have higher weights. This effectively suppresses high-frequency noise while preserving the image's edge contours.

[0033] S2.2: Use the Sobel operator to calculate the grayscale value of each pixel in the grayscale image. x shaft and y The rate of change of grayscale along the axial direction yields the image. x shaft and y Spatial gradient matrix along the axis; In this step, the spatial gradient reflects the rate of change of pixel grayscale values ​​in a plane and is a core feature for identifying image edges, textures, and potential motion regions. The Sobel operator is a classic first-order differential operator that performs convolution operations on the smoothed grayscale image by designing "horizontal / vertical convolution kernels". The larger the absolute value of the convolution result, the more drastic the grayscale change in that direction, and the more likely it is an edge or motion boundary.

[0034] S2.3: Using the first i The pixel value of the second captured image minus the first i- The pixel values ​​at corresponding locations in a single image capture are used to obtain the temporal gradient of each pixel. In this step, the temporal gradient is based on the core assumption of optical flow, namely that the grayscale value of the same object in adjacent frames changes continuously and slowly. This is achieved by calculating the... i Frame and the i The grayscale difference of the corresponding pixel in frame -1 reflects the trend of grayscale value change over time: if the pixel is static, the time gradient is close to 0; if the pixel is a moving target, its position shifts between two frames, and the grayscale difference at the corresponding position will show a significant non-zero value.

[0035] S2.4: Within a 3×3 neighborhood window of each pixel, calculate x The sum of squares of the directional spatial gradients, Gxx, is specifically expressed as:

[0036] in For the image The grayscale value of the location; The product of the spatial gradients in the x-direction and y-direction, Gxy, is calculated using the following expression:

[0037] The sum of squares of the spatial gradients in the y-direction, Gyy, is calculated using the following expression: ;

[0038] Within the neighborhood window, calculate x The sum of the products of the spatial and temporal gradients in the y-direction, Gxt, and the sum of the products of the spatial and temporal gradients in the y-direction, Gyt, are: ; ;

[0039] in, To capture the time difference between two images; In this step, the gradient value of a single pixel is susceptible to noise interference and has poor stability. By aggregating (summing) the gradient information within a 3×3 neighborhood, the redundant information within the neighborhood can be used to offset the influence of noise, thereby improving the robustness of the gradient features. Specifically: Gxx , Gyy Reflecting respectively x, y The overall strength of the directional spatial gradient; Gxy reflect x and y Correlation of directional gradients; Gxt , Gyt This relates the spatial gradient and the temporal gradient, and is the core coefficient for constructing the optical flow equation.

[0040] S2.5: Construct a system of linear equations, the specific expression of which is:

[0041] In this step, the derivation is based on the Optical Flow Constraint Equation (OFCE). The aggregated gradient parameters calculated in S2.4 are directly substituted into the neighborhood merging form of the optical flow constraint equation to construct a system of equations.

[0042] S2.6: Solve the system of linear equations to obtain the optical flow velocity of each pixel, thereby obtaining the moving target in the image. Then the... i The set of moving targets in the frame image is .

[0043] In this embodiment, the optical flow vector of each pixel is obtained by solving a system of linear equations. The magnitude of the optical flow vector directly reflects the intensity of pixel motion. The magnitude of the optical flow vector for each pixel is calculated; pixels with a magnitude greater than a set threshold are considered moving pixels, otherwise they are considered static pixels. Connectivity analysis is performed on the moving pixels, dividing interconnected moving pixels into the same region, with each region representing a moving target. Finally, the [presumably a specific parameter or value] is obtained. i Set of moving targets in a frame image: ,in, Representing the i The first frame k A moving target (including the target's position, bounding box, pixel set, etc.).

[0044] In this embodiment, the core objective of step S2 is to separate the moving target from consecutive frame images, laying the foundation for subsequent cross-modal (e.g., point cloud) matching. Its technical basis is the optical flow constraint assumption: the pixel grayscale values ​​of the same object in adjacent frames change continuously and slowly, and the spatiotemporal gradient change of the pixel satisfies the constraint that the sum of the product of the spatial gradient and the motion velocity equals the negative value of the temporal gradient. This is achieved through quantitative analysis of the pixel's spatial (… x, y The grayscale changes in the axis and time dimension are used to solve for the pixel motion velocity (optical flow), and finally the moving pixels are aggregated to obtain the complete target.

[0045] Each step forms a closed-loop logic of input optimization, feature extraction, parameter aggregation, model solving, and target output, with each step providing accurate and stable input for subsequent steps. Basic preprocessing (S2.1): As the starting point of the process, the three-dimensional RGB data of the color image is reduced to one-dimensional grayscale information by grayscale conversion, and redundant data is removed to simplify the calculation. At the same time, Gaussian filtering is used to smooth the noise based on normal distribution weighting to avoid random noise from interfering with subsequent gradient calculation, providing a cleaner image input for extracting reliable features.

[0046] Spatial feature extraction (S2.2): Based on the preprocessed image in S2.1, the Sobel operator is used to perform convolution operations on the 3×3 neighborhood of the pixels to obtain spatial features. x, y The spatial gradient matrix along the axis. The spatial gradient reflects the degree of gray-level change of a pixel in a plane. It is the core of recognizing spatial structural features such as target edges and textures, and provides key parameters for associating spatial changes with motion states.

[0047] Temporal feature extraction (S2.3): By calculating the first... i Frame and the i The grayscale difference of the corresponding pixels in frame -1 is used to obtain the temporal gradient, which directly captures the trend of pixel grayscale change over time. The temporal gradient is an intuitive basis for judging whether a pixel is moving, and together with the spatial gradient, it constitutes the "spatiotemporal dual-dimensional feature" of optical flow analysis.

[0048] Feature aggregation enhancement (S2.4): To address the issue of single-pixel gradients being susceptible to noise interference, spatial gradients are enhanced within a 3×3 neighborhood. Gxx, Gxy, Gyy ) and spatiotemporal correlation gradient ( Gxt, Gyt This process involves summing and aggregating the data. This step uses spatial redundancy information to cancel out local noise, transforming discrete single-pixel features into stable regional features, and providing robustness coefficients for constructing a solvable mathematical model.

[0049] Mathematical Model Construction (S2.5): Based on the optical flow constraint equation, the gradient parameters aggregated in S2.4 are substituted to construct a system of two linear equations. Essentially, this system integrates the optical flow constraint relationships of all pixels in the neighborhood, achieving the transformation from features to a model.

[0050] Target output (S2.6): Solve the system of equations to obtain the optical flow velocity of each pixel. Moving pixels are filtered by calculating the magnitude of the optical flow vector, and then adjacent moving pixels are aggregated into complete regions through connected component analysis. Finally, the set of moving targets in the i-th frame image is output. .

[0051] Preferably, step S3 includes the following process: S3.1: Point Cloud Perform filtering, using Gaussian filtering to remove noise points; The core of this step is to suppress point cloud noise while preserving the structural integrity of the target, providing reliable raw data for subsequent difference point detection. Point clouds acquired by LiDAR (such as equipment and ground point clouds in substation scenarios) are susceptible to environmental interference (electromagnetic radiation, dust scattering) and sensor errors (fluctuations in ranging accuracy), generating discrete noise points. These points usually deviate significantly from the spatial positions of the surrounding normal point clouds. If not removed, they will form false difference points in subsequent frame difference calculations, interfering with moving target recognition. Gaussian filtering is based on the principle of weighted averaging of a normal distribution. Assuming that the point cloud noise follows a Gaussian distribution, it assigns different weights to all points in the local neighborhood of each point according to their distance from the center (the closer the distance, the higher the weight), and corrects the center point position through weighted summation. This method can suppress high-frequency noise while preserving the continuous structure of the point cloud (such as equipment edges and target contours) to the maximum extent, avoiding the loss of target features due to excessive smoothing.

[0052] S3.2: Calculating Point Clouds With point clouds The Euclidean distance of each point in space, if the distance is greater than a threshold Then this point is considered a point of difference; In this step, the continuous frames acquired by the lidar ( and In this context, the point cloud of static objects (such as transformers and switchgear) remains almost unchanged in 3D space, with its Euclidean distance approaching 0; while the point cloud of moving targets (such as inspection robots and operating tools) will shift as the target moves, and its distance from the corresponding point in the previous frame will exceed a set threshold. By setting reasonable thresholds It can quickly filter out the differences in points from massive point clouds, that is, the point cloud of suspected moving targets.

[0053] S3.3: After obtaining the difference points, perform connected component analysis on them, merging adjacent difference points into a single connected component; then cluster the difference points, grouping spatially similar difference points into one cluster, forming multiple clusters. Each cluster may correspond to a moving target. Therefore, the set of moving targets in the point cloud of the i-th frame is: .

[0054] In this step, the discrepancies are mostly isolated, discrete points. However, the point cloud of an actual moving target (such as an inspection robot) has spatial continuity; that is, the point clouds on the target surface are adjacent to each other in three-dimensional space, forming continuous point cloud clusters. This step uses two operations—connected component analysis and clustering—to aggregate discrete discrepancies into complete clusters based on spatial proximity. Connected component analysis focuses on physical proximity, directly merging discrepancies with a distance less than a set threshold. Clustering focuses on density or distance similarity, further grouping spatially close connected components into one category. Ultimately, each cluster corresponds to an actual moving target.

[0055] Through the steps outlined above, this algorithm addresses the needs of extracting moving targets from point clouds, offering advantages such as strong noise resistance, high efficiency, and accurate target aggregation. First, Gaussian filtering is used to weighted smooth the local neighborhood of each point, effectively eliminating discrete noise points. Simultaneously, distance-weighted filtering preserves the continuous spatial structure of the target (such as equipment edges and target contours), preventing noise interference in subsequent difference detection. Second, a frame difference method combined with Euclidean distance is employed to filter difference points. This approach is simple and intuitive, requiring no complex feature extraction or modeling, resulting in low computational overhead. It can quickly locate suspected moving points from massive point clouds, meeting the real-time requirements of industrial scenarios. Finally, through a combination of connected component analysis and clustering, discrete difference points are aggregated into complete clusters based on the spatial continuity of the moving target, while small noise clusters are eliminated. This accurately distinguishes between real moving targets and false interference, improving target recognition accuracy.

[0056] As can be seen from the above technical solutions, the embodiments of the present invention have the following advantages: The present invention extracts moving targets from images and point clouds using optical flow and differential methods, and uses the point cloud and image features of the moving targets to iteratively optimize the extrinsic parameter matrix, eliminating moving targets with large errors, and realizing the extrinsic parameter calibration of cameras and lidar based on moving targets in any scene. This effectively solves the problem that calibration depends on calibration boards and that extrinsic parameter correction cannot be performed during field use, and can well meet the needs of measuring the distance between energized bodies and surrounding objects in large substation scenarios.

Claims

1. A method for optimizing camera and radar extrinsic parameters based on moving targets at a work site, characterized in that, The method includes the following steps: S1: The camera and lidar to be calibrated simultaneously acquire continuous frames of image and point cloud data, let the image set be... Point cloud set as The specific expression is: ； ； The image data acquired in the nth acquisition is: Point cloud data is Each acquired image and point cloud is stored sequentially. and middle; S2: Extraction of the middle part using optical flow method i Images acquired in this session Moving targets in ; S3: Extracting the first frame using the frame difference method i Point cloud collected in the second time Moving targets in ; S4: Using the device's existing calibration matrix, project the point cloud of the moving target set onto the image. The projected point cloud is... ; S5: Matching using rigid body transformation method and And store the matched moving targets in ,Right now ; For all matched moving targets in the k acquisitions, i.e., moving targets in the image and point cloud. ,in , For the k-th group, match the moving targets in the image and the moving targets in the point cloud; S6: If The number of moving targets is less than the threshold If the condition is met, return to step S2; otherwise, iterate through the previous steps. Extracting using the Canny operator Image edge features are extracted using a normal estimation method. The point cloud edge features were analyzed, and the image was registered with the point cloud edge features using the Hungarian matching algorithm to obtain... The set of image and point cloud features of all moving targets in the dataset, i.e.: ; S7: Employs the PNP algorithm, utilizing... Perform extrinsic parameter calibration of the camera and lidar to obtain the extrinsic parameter calibration matrix. ; S8: Using the external parameter calibration matrix Will All moving targets in the point cloud are projected onto the image, and the Euclidean distance between them and the corresponding image features is calculated as the projection error. S9: Calculate the root mean square error of the projection error. If the root mean square error is less than the threshold... Then the extrinsic calibration matrix is ​​considered to be... For the extrinsic parameter calibration matrix of the camera and lidar, otherwise in Remove the moving target with the largest projection error and return to S6.

2. The method for optimizing camera and radar extrinsic parameters based on moving targets at the work site according to claim 1, characterized in that, In step S2, Includes the following processes: S2.1: Preprocess the image by converting it into a grayscale image and using a Gaussian filtering algorithm to smooth out noise in the image; S2.2: Use the Sobel operator to calculate the grayscale value of each pixel in the grayscale image. x shaft and y The rate of change of grayscale along the axial direction yields the image. x shaft and y Spatial gradient matrix along the axis; S2.3: Using the first i The pixel value of the second captured image minus the first i- The pixel values ​​at corresponding locations in a single image capture are used to obtain the temporal gradient of each pixel. S2.4: Within a 3×3 neighborhood window of each pixel, calculate x Sum of squares of spatial gradients of orientation Gxx The specific expression is: ； in For the image The grayscale value of the location; The product of the spatial gradients in the x-direction and y-direction, Gxy, is calculated using the following expression: ； The sum of squares of the spatial gradients in the y-direction, Gyy, is calculated using the following expression: ； Within the neighborhood window, calculate x The sum of the product of the spatial gradient and the temporal gradient Gxt and the sum of the spatial gradient and temporal gradient in the y-direction Gyt ,Right now: ； ； in, To capture the time difference between two images; S2.5: Construct a system of linear equations, the specific expression of which is: ； S2.6: Solve the system of linear equations to obtain the optical flow velocity of each pixel, thereby obtaining the moving target in the image. Then the... i The set of moving targets in the frame image is .

3. The method for optimizing camera and radar extrinsic parameters based on moving targets at the work site according to claim 1, characterized in that, In step S3, Includes the following processes: S3.1: Point Cloud Perform filtering, using Gaussian filtering to remove noise points; S3.2: Calculating Point Clouds With point clouds The Euclidean distance of each point in space, if the distance is greater than a threshold Then this point is considered a point of difference; S3.3: After obtaining the difference points, perform connected component analysis on them, merging adjacent difference points into a single connected component; then cluster the difference points, grouping spatially similar difference points into one cluster, forming multiple clusters. Each cluster may correspond to a moving target. Therefore, the set of moving targets in the point cloud of the i-th frame is: .

Images