Camera-lidar joint calibration method and device based on natural scene features

Abstract

This application provides a camera-LiDAR joint calibration method and apparatus based on natural scene features. The method involves generating a dense point cloud based on a discrete point cloud sequence acquired by a robot in a natural scene, and encapsulating it with a synchronous visual image acquired by the robot in the natural scene to generate multimodal collaborative data. The multimodal collaborative data is instantiated into visual pixels and LiDAR point clouds to determine the target region at the image end and the target clusters in the point cloud. The target region at the image end is then mapped and smoothed to construct a boundary metric mask. Based on the boundary metric mask, the three-dimensional physical coordinate group of the target clusters in the point cloud and the two-dimensional coordinate group of the target region at the image end are located, and the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group are calculated. Based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, a dual relationship is established between the three-dimensional physical coordinate group and the two-dimensional coordinate group, and spatial calculations are performed to generate scene spatial registration joint parameters.

Description

Technical Field

[0001] This application relates to the field of robot multi-source environmental perception technology, and more specifically, to a camera-LiDAR joint calibration method and device based on natural scene features. Background Technology

[0002] With the acceleration of urbanization and the development of intelligent manufacturing, the demand for security protection in key areas has surged. Inspection robots based on autonomous driving systems are widely used in unmanned inspections in various industrial and public settings. To achieve autonomous navigation and multi-source perception, inspection robots are typically equipped with multiple sensors, such as high-precision LiDAR and visual cameras, to construct environments and identify various abnormal behavioral characteristics. In this process, data fusion between multimodal sensors relies on accurate joint calibration parameters of the camera and LiDAR.

[0003] In existing multi-sensor fusion solutions for inspection robots, a camera-LiDAR joint calibration method based on artificially structured targets is typically adopted. This method first involves manually arranging calibration boards with specific sizes and reflective patterns in an open and structured environment; then, controlling the inspection robot to acquire laser point cloud and visual image data containing the calibration board at different angles and distances, extracting the corner points or edge features of the calibration board; finally, based on the extracted target geometric features, the spatial transformation extrinsic parameter matrix between the sensors is calculated using a reprojection error minimization algorithm.

[0004] However, this calibration scheme has obvious technical drawbacks. Since inspection robots often need to operate 24 / 7 in complex natural scenes and adapt to harsh environments, manually setting up calibration boards is cumbersome and makes on-site dynamic calibration difficult. Furthermore, traditional methods, if used to extract features directly from the target in natural scenes, are highly susceptible to environmental noise and interference, making it difficult to accurately establish the spatial duality between scattered 3D point clouds and 2D pixels. This results in insufficient reliability of the spatial joint calibration parameters, failing to meet the inspection robot's need for accurate collaborative registration of multimodal perception data in complex scenarios. Summary of the Invention

[0005] This application provides a camera-LiDAR joint calibration method and apparatus based on natural scene features, so as to at least alleviate the above-mentioned technical problems.

[0006] A camera-LiDAR joint calibration method based on natural scene features includes: Step 1: Generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. Step 2: Instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively to determine the target region at the image end and the target cluster in the point cloud, and perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map; Step 3: Based on the boundary metric mask map, locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image, and calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. Step 4: Based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group and perform spatial calculations on them to generate scene spatial registration joint parameters.

[0007] Optionally, the step of performing mapping smoothing processing based on the inverse distance gradient on the target region of the image to construct a boundary metric mask map includes: extracting the effective target pixel set and the remaining background pixel set within the target region of the image; assigning binarized difference labels to the effective target pixel set and the remaining background pixel set to generate an initial binary segmentation mask layer; calculating the Euclidean distance offset from each pixel in the initial binary segmentation mask layer to the target segmentation boundary; and performing exponential distance decay weighting processing on the internal weights of the initial binary segmentation mask layer according to the Euclidean distance offset to construct the boundary metric mask map.

[0008] Optionally, the step of calculating the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group includes: determining all three-dimensional discrete point coordinate vectors belonging to the same target cluster instance in the three-dimensional physical coordinate group, and performing spatial vector accumulation and mean-based division operations on them to obtain the three-dimensional geometric centroid coordinates; determining all two-dimensional pixel coordinate vectors belonging to the same target cluster instance in the two-dimensional coordinate group, and performing planar vector accumulation and mean-based division operations on them to obtain the two-dimensional semantic centroid coordinates.

[0009] Optionally, the step of establishing a dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and performing spatial calculations on them to generate scene space registration joint parameters, includes: constructing a centroid mapping relationship between space and plane based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates; performing perspective point-related projection algebraic solution on the centroid mapping relationship to obtain an initial extrinsic parameter transformation matrix; and inputting the initial extrinsic parameter transformation matrix as a search starting point into a preset particle swarm optimization space for iterative optimization processing to generate the scene space registration joint parameters.

[0010] Optionally, the step of inputting the initial extrinsic transformation matrix as the search starting point into a preset particle swarm optimization algorithm space for iterative optimization to generate the scene space registration joint parameters includes: applying a spatial perturbation variable to the initial extrinsic transformation matrix to initialize the generation of a biomimetic search particle swarm; mapping the three-dimensional physical coordinate group to the plane where the boundary metric mask map is located through the state matrix of each particle in the biomimetic search particle swarm to determine the feature projection landing point set; constructing a target panoramic information matching degree based on the consistency response value between the feature projection landing point set and the boundary metric mask map; and performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene space registration joint parameters.

[0011] Optionally, the step of performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene spatial registration joint parameters includes: evaluating the fitness evaluation value of each particle in the biomimetic search particle swarm through the target panoramic information matching degree; extracting the individual optimal exploration position of each particle and the global optimal exploration position of the entire swarm based on the fitness evaluation value; performing linear decay calculation according to the optimization evolution process to generate dynamic inertia adjustment weights; using the individual optimal exploration position, the global optimal exploration position, and the dynamic inertia adjustment weights to update the current movement speed and distribution position of each particle to output the evolved state particle swarm, and determining the scene spatial registration joint parameters based on the evolved state particle swarm.

[0012] Optionally, determining the scene space registration joint parameters based on the evolved state particle swarm includes: performing fitness evaluation processing on the evolved state particle swarm again to obtain a verification evaluation result; determining whether the verification evaluation result triggers a preset optimization convergence condition; in response to not triggering the preset optimization convergence condition, adjusting the optimization iteration number and returning to the linear decay calculation performed according to the optimization evolution process to generate dynamic inertia adjustment weights; in response to triggering the preset optimization convergence condition, converting the global optimal exploration position corresponding to the verification evaluation result into a calibration extrinsic parameter matrix and using it as the scene space registration joint parameters.

[0013] Optionally, the step of generating a scene-dense point cloud based on a discrete point cloud sequence collected by the robot in a natural scene includes: separating the current observation frame point cloud and the historical observation frame point cloud from the discrete point cloud sequence; performing voxel downsampling on the current observation frame point cloud and the historical observation frame point cloud to obtain a spatially sparse point cloud; performing local geometric difference filtering on the spatially sparse point cloud to extract a spatially distributed sparse feature point set; performing local geometric morphology encoding on the spatially distributed sparse feature point set to generate a structural feature descriptor; and performing pose transformation operation on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene-dense point cloud.

[0014] Optionally, the step of performing attitude transformation operations on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene dense point cloud includes: selecting initial homonymous feature point pairs from the spatially distributed sparse feature point sets corresponding to the current observation frame point cloud and the historical observation frame point cloud based on the nearest neighbor metric relationship of the structural feature descriptor; generating an initial coarse registration transformation matrix based on the initial homonymous feature point pairs; mapping the historical observation frame point cloud to the reference coordinate system of the current observation frame point cloud using the initial coarse registration transformation matrix to establish a nearest neighbor point pair distance error space; and performing nearest point iterative minimization optimization on the nearest neighbor point pair distance error space to generate the scene dense point cloud.

[0015] A camera-LiDAR joint calibration device based on natural scene features, comprising: The first program unit is used to generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. The second program unit is used to instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively, so as to determine the target region at the image end and the target cluster in the point cloud, and to perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map. The third program unit is used to locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image based on the boundary metric mask map, and to calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. The fourth program unit is used to establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and to perform spatial calculation on them to generate scene spatial registration joint parameters.

[0016] The technical advantages of the technical solution provided in this application are: This application presents a camera-LiDAR joint calibration method and apparatus based on natural scene features. Addressing the shortcomings of traditional centralized or target-dependent calibration schemes, such as cumbersome manual setup and implementation, difficulty in on-site calibration, and severe interference with extraction accuracy in natural scenes, this method generates a dense point cloud from a sequence of discrete point clouds acquired by a robot in a natural scene. This dense point cloud is then encapsulated with synchronous visual images acquired by the robot in the same natural scene to generate multimodal collaborative data. This solves the problem of traditional schemes' over-reliance on external targets and weak adaptability. Compared to the traditional passive acquisition mode that requires setting up a calibration board in a specific environment, this application directly utilizes the sequence features naturally acquired by the robot in real inspection scenarios to encapsulate multimodal data, significantly improving the autonomy and environmental adaptability of on-site dynamic calibration.

[0017] Based on the generated multimodal collaborative data, visual pixels and laser point clouds are instantiated separately to determine the target region at the image end and the target clusters in the point cloud. The target region at the image end is then mapped and smoothed to construct a boundary metric mask. This solves the problem that traditional methods are easily affected by environmental noise when directly extracting features in natural scenes. Compared to traditional methods that easily produce mismatches when directly extracting corners or edges from the natural environment, this application constrains the target region through instantiation technology and combines it with smoothing processing to construct a boundary metric mask. This effectively suppresses interference from complex backgrounds, significantly improving the stability of multimodal feature region extraction and edge recognition.

[0018] Finally, based on the boundary metric mask map, the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image are located, and the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group are calculated. Based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group is established and spatially solved to generate scene spatial registration joint parameters, solving the problem of difficulty in establishing an accurate mapping relationship between scattered point clouds and two-dimensional pixels. Compared with the traditional strategy of relying on local sparse feature point pairs to directly solve, which is prone to large errors, this application transforms the registration core to the more macroscopic and stable structural features of geometric centroid and semantic centroid. By performing spatial solution through the dual relationship between the centroid and centroid, the matching error caused by occlusion and parallax is effectively reduced, making the reliability of the finally generated scene spatial registration joint parameters higher and better meeting the needs of accurate collaborative registration of multimodal perception data in complex scenes. Attached Figure Description

[0019] Figure 1 This is a schematic flowchart of a camera-LiDAR joint calibration method based on natural scene features according to an embodiment of this application; Figure 2 This is a structural block diagram of a camera-LiDAR joint calibration device based on natural scene features, according to an embodiment of this application. Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0020] like Figure 1 As shown, this embodiment of the present application provides a camera-LiDAR joint calibration method based on natural scene features, which includes: Step 1: Generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. Step 2: Instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively to determine the target region at the image end and the target cluster in the point cloud, and perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map; Step 3: Based on the boundary metric mask map, locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image, and calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. Step 4: Based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group and perform spatial calculations on them to generate scene spatial registration joint parameters.

[0021] Optionally, the step of performing mapping smoothing processing based on the inverse distance gradient on the target region of the image to construct a boundary metric mask map includes: extracting the effective target pixel set and the remaining background pixel set within the target region of the image; assigning binarized difference labels to the effective target pixel set and the remaining background pixel set to generate an initial binary segmentation mask layer; calculating the Euclidean distance offset from each pixel in the initial binary segmentation mask layer to the target segmentation boundary; and performing exponential distance decay weighting processing on the internal weights of the initial binary segmentation mask layer according to the Euclidean distance offset to construct the boundary metric mask map.

[0022] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, the specific technical implementation of performing mapping smoothing processing based on the inverse distance gradient on the target area of ​​the image to construct a boundary metric mask map first requires visual noise reduction and region decoupling processing on the visual information in the synchronized visual image. Because natural scenes contain environmental interference factors such as tree occlusion and uneven lighting, directly extracting calibration features from the entire image can easily introduce a lot of mismatch noise. Therefore, it is necessary to extract the effective target pixel set and the remaining background pixel set within the target area of ​​the image. Specifically, the visual semantic morphological features within the target area of ​​the image are clustered and categorized. Pixels belonging to the natural main structures that need to be jointly calibrated (such as streetlights, building edges, etc. in a park scene) are extracted and aggregated to form the effective target pixel set; simultaneously, interference objects and distant pixels that do not belong to the natural main structures are summarized and aggregated to form the remaining background pixel set. This visual noise reduction and region decoupling process eliminates the interference of complex background visual features, providing a relatively clean data processing foundation for subsequent mapping smoothing based on the inverse distance gradient.

[0023] Preferably, after obtaining the effective target pixel set and the remaining background pixel set, in order to establish a transformation relationship from the visual semantic space to the numerical computation space, binary difference labels are assigned to the effective target pixel set and the remaining background pixel set to generate an initial binary segmentation mask layer. In the specific underlying data allocation process, a two-dimensional empty matrix space with the same spatial resolution as the target region at the image end is constructed. The rows and columns of this two-dimensional empty matrix space correspond to the horizontal and vertical pixel coordinates of the image captured by the camera that acquired the synchronous visual image, respectively, and the elements at the intersection of the rows and columns correspond to specific pixels. Subsequently, the elements at the intersections belonging to the effective target pixel set are filled with a first numerical label representing the foreground activation state (e.g., a higher fixed value), and the elements at the intersections belonging to the remaining background pixel set are filled with a second numerical label representing the background suppression state (e.g., a lower fixed value). These first and second numerical labels together constitute the binary difference labels. The two-dimensional empty matrix space after numerical filling has completed the state transformation, and the output is the initial binary segmentation mask layer. Using the initial binary segmentation mask layer, the rough physical outline of the target to be calibrated in the natural scene can be initially defined in the form of a discretized numerical matrix.

[0024] Preferably, although the initial binary segmentation mask layer distinguishes the target from the background at the visual semantic numerical level, its edges exhibit a step-like numerical abrupt change. When performing cross-modal dual feature association from the three-dimensional physical coordinate group to the two-dimensional plane containing the two-dimensional coordinate group, this step-like numerical abrupt change leads to an overly narrow matching response domain, making the spatial solution process prone to getting trapped in a locally suboptimal matching state. To expand the convergence range of effective calibration matching, it is necessary to extract the target segmentation boundary in the initial binary segmentation mask layer. Specifically, by detecting the switching transition positions of the first and second numerical labels between adjacent pixels in the initial binary segmentation mask layer, the closed or semi-closed contour lines formed by connecting these numerical transition points are defined as the target segmentation boundary. This target segmentation boundary objectively reflects the real physical entity edge projection position of the physical entity in the natural scene on the visual image, and it will serve as the core distance metric baseline for subsequent mapping smoothing processing based on the inverse distance gradient.

[0025] Preferably, after determining the location reference of the target segmentation boundary, the Euclidean distance offset from each pixel in the initial binary segmentation mask layer to the target segmentation boundary is calculated. Specifically, each pixel in the initial binary segmentation mask layer is traversed. Regardless of whether the pixel currently carries the first or second numerical label, the shortest straight-line physical distance from the pixel to the nearest pixel on the target segmentation boundary is determined using spatial geometric measurements. The measured shortest straight-line physical distance is recorded as the Euclidean distance offset specific to that pixel. The Euclidean distance offset essentially characterizes the spatial span of each pixel from the projected position of the actual physical entity edge. A lower offset value indicates a higher reference guidance value for multimodal feature alignment at the pixel's location.

[0026] Preferably, after obtaining the Euclidean distance offset of all pixels in the initial binary segmentation mask layer, an exponential distance decay weighting process is performed on the internal weights of the initial binary segmentation mask layer based on the Euclidean distance offset to construct the boundary metric mask map. In this technical step, the original binarized difference labels in the initial binary segmentation mask layer are used as the internal weights to be updated. Using the negative exponential operation logic of the natural logarithm base, the Euclidean distance offset of each pixel is used as a variable factor of the nonlinear exponent for exponential distance decay calculation. This nonlinear operation mode ensures that when the Euclidean distance offset of a pixel is close to zero (i.e., close to the target segmentation boundary), the updated weight obtained after exponential distance decay calculation approaches the upper limit of the extreme value, maintaining a high feature responsivity; while as the Euclidean distance offset gradually increases, the corresponding updated weight will show a sharp nonlinear decreasing trend until it approaches the background suppression state of the lower limit of the extreme value. By remapping the latest updated weights of all pixels back into the aforementioned constructed two-dimensional empty matrix space, the boundary metric mask map with gradient transition characteristics is obtained.

[0027] Preferably, the boundary metric mask constructed through the above multi-step technical processing transforms the initial binary segmentation mask layer, which originally only had discrete features, into a smooth potential field map with continuous gradient changes in objective data form. In the boundary metric mask map, the region near the projection position of the real physical entity edge has a higher update weight, while the update weight of the radiating region gradually decreases smoothly. Compared with the high spatial error sensitivity caused by traditional joint calibration methods that directly use corner points or binary edges for hard matching, the mapping smoothing processing based on the inverse distance gradient implemented in this application can provide a broad search space with a clear numerical gradient guidance when performing spatial calculations from the three-dimensional physical coordinate group of the target cluster in the point cloud to the two-dimensional plane where the two-dimensional coordinate group is located. This processing method effectively improves the feature fault tolerance rate of the inspection robot when facing slight posture jitter or deviations in the initial calibration point in complex natural scenes, so that the spatial calculation process of the finally generated scene spatial registration joint parameters has high convergence stability, which better meets the needs of unmanned inspection equipment for dynamic and accurate collaborative registration in natural environments.

[0028] Optionally, the step of calculating the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group includes: determining all three-dimensional discrete point coordinate vectors belonging to the same target cluster instance in the three-dimensional physical coordinate group, and performing spatial vector accumulation and mean-based division operations on them to obtain the three-dimensional geometric centroid coordinates; determining all two-dimensional pixel coordinate vectors belonging to the same target cluster instance in the two-dimensional coordinate group, and performing planar vector accumulation and mean-based division operations on them to obtain the two-dimensional semantic centroid coordinates.

[0029] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, the specific technical implementation for calculating the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group requires first establishing a macroscopic statistical benchmark for feature alignment between the dense point cloud of the scene and the synchronized visual image. Considering the complex surface morphology of natural structures such as streetlights and buildings in natural scenes, directly using local discrete features for cross-modal comparison can easily lead to matching bias. Therefore, instance association analysis based on spatial connectivity and visual semantic consistency is performed on the three-dimensional physical coordinate group and the two-dimensional coordinate group to determine the data ownership boundary of instances belonging to the same target cluster in the three-dimensional physical coordinate group and the two-dimensional coordinate group. This same target cluster instance objectively represents the common referent of an isolated and continuous real physical entity in the natural scene in the three-dimensional perception space of the dense point cloud of the scene and the two-dimensional projection space of the synchronized visual image. By establishing this instance association relationship, the data ownership boundary of the same target cluster instance is clarified, providing a clear basis for the subsequent calculation of the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates.

[0030] Preferably, after establishing the data ownership boundary of the same target cluster instance, for the three-dimensional perception space of the dense point cloud of the scene, the coordinate vectors of all three-dimensional discrete points belonging to the same target cluster instance in the three-dimensional physical coordinate group are determined. Specifically, each three-dimensional discrete point contained in the three-dimensional physical coordinate group and located within the data ownership boundary is traversed, and the lateral spatial position data, longitudinal spatial position data, and depth spatial position data (e.g., corresponding to forward distance, lateral distance, and height data, respectively) of these three-dimensional discrete points in the three-dimensional reference coordinate system of the lidar (i.e., the three-dimensional physical coordinate system of the sensor itself that acquired the discrete point cloud sequence) are extracted. The extracted lateral spatial position data, longitudinal spatial position data, and depth spatial position data are then subjected to matrix dimension concatenation to construct the three-dimensional discrete point coordinate vector corresponding one-to-one with each three-dimensional discrete point. The three-dimensional discrete point coordinate vector, in the form of an algebraic expression, objectively records the geometric orientation distribution information of each reflection point on the surface of a real physical entity in the natural scene in three-dimensional physical space.

[0031] Preferably, after generating all the three-dimensional discrete point coordinate vectors in the three-dimensional physical coordinate group, spatial vector accumulation and mean-based division operations are performed on all three-dimensional discrete point coordinate vectors belonging to the same target cluster instance to obtain the three-dimensional geometric centroid coordinates. In the actual algebraic numerical calculation step, firstly, all three-dimensional discrete point coordinate vectors included in the same target cluster instance are traversed and summed, and the values ​​of all three-dimensional discrete point coordinate vectors are accumulated in each independent three-dimensional spatial dimension to obtain a three-dimensional spatial accumulation vector representing the overall spatial distribution trend; then, the total number of all three-dimensional discrete point coordinate vectors participating in the accumulation calculation is counted, and the values ​​of each dimension of the three-dimensional spatial accumulation vector are divided by the total number to obtain the three-dimensional geometric centroid coordinates. The three-dimensional geometric centroid coordinates physically represent the overall geometric center of mass of the three-dimensional physical coordinate group in spatial distribution, which can effectively offset the positional offset caused by the sparseness of local three-dimensional discrete points or unstructured surfaces.

[0032] Preferably, while extracting the geometric center in the three-dimensional perception space, for the two-dimensional projection space of the synchronized visual image, all two-dimensional pixel coordinate vectors belonging to the same target cluster instance in the two-dimensional coordinate group are determined. Specific technical processing actions include: extracting each valid pixel belonging to the same target cluster instance from the two-dimensional coordinate group located from the synchronized visual image based on the data attribution boundary; subsequently, acquiring the horizontal and vertical pixel position data (e.g., row and column coordinates) of these valid pixels on the two-dimensional projection plane of the synchronized visual image. Performing two-dimensional matrix merging processing on the acquired horizontal and vertical pixel position data generates the unique two-dimensional pixel coordinate vector for each valid pixel. The two-dimensional pixel coordinate vector objectively reflects the semantic projection position set state of real physical entities in the natural scene on the photosensitive plane of the visual sensor.

[0033] Preferably, after extracting all the two-dimensional pixel coordinate vectors in the two-dimensional coordinate group, planar vector accumulation and mean-based division operations are performed on all two-dimensional pixel coordinate vectors belonging to the same target cluster instance to obtain the two-dimensional semantic centroid coordinates. The specific algebraic processing logic is as follows: All two-dimensional pixel coordinate vectors belonging to the same target cluster instance are accumulated along both the horizontal and vertical dimensions to generate a two-dimensional planar accumulation vector; then, the total number of all two-dimensional pixel coordinate vectors participating in this accumulation calculation is obtained, and algebraic division is performed on the two-dimensional planar accumulation vector using this total number to obtain the two-dimensional semantic centroid coordinates. The two-dimensional semantic centroid coordinates eliminate the dependence on single visual edge pixels or isolated corner points, objectively presenting the visual distribution center viewpoint position of the target semantic coverage area in the synchronized visual image.

[0034] Preferably, through the independent vector operations performed on the three-dimensional perception space of the dense point cloud of the scene and the two-dimensional projection space of the synchronized visual image, the originally large and scattered three-dimensional physical coordinate group and the two-dimensional coordinate group are successfully abstracted into the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates with a simple structure and stable form. Compared with the spatial error sensitivity problem caused by directly relying on edge contour points or corner points for matching in traditional joint calibration methods, the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates extracted in this application belong to the algebraic feature representation at the macroscopic statistical level. Since the spatial vector accumulation and mean division operation and the planar vector accumulation and mean division operation can effectively suppress local edge jitter noise caused by sensor resolution limitations, environmental occlusion or sudden changes in illumination in natural scenes, the finally obtained three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates have high anti-interference stability. This provides a relatively pure and stable denoising feature base for establishing the dual relationship between cross-modal features and performing spatial calculations, which better meets the technical requirements for high fault tolerance in multimodal perception data collaborative registration in complex natural scenes.

[0035] Optionally, the step of establishing a dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and performing spatial calculations on them to generate scene space registration joint parameters, includes: constructing a centroid mapping relationship between space and plane based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates; performing perspective point-related projection algebraic solution on the centroid mapping relationship to obtain an initial extrinsic parameter transformation matrix; and inputting the initial extrinsic parameter transformation matrix as a search starting point into a preset particle swarm optimization space for iterative optimization processing to generate the scene space registration joint parameters.

[0036] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, in the specific technical implementation of constructing the centroid mapping relationship between space and plane based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, since the aforementioned technical processing steps have already highly abstracted the discrete features in the natural scene, it is necessary to establish a mathematical projection dual link between cross-modal data. Specifically, the horizontal, vertical, and depth spatial position data of the three-dimensional geometric centroid coordinates belonging to the same real physical entity, as well as the horizontal and vertical pixel position data of the two-dimensional semantic centroid coordinates, are extracted. Each spatial position data of the three-dimensional geometric centroid coordinates is used as the mapping input in the three-dimensional perception space coordinate system, and the corresponding pixel position data of the two-dimensional semantic centroid coordinates is used as the mapping output in the two-dimensional projection plane coordinate system. Through data alignment and combination processing, the centroid mapping relationship between space and plane is constructed. This centroid mapping relationship objectively characterizes the macroscopic geometric correspondence law when the real physical entity in the natural scene performs cross-dimensional mapping from the three-dimensional perception space of the dense point cloud of the scene to the two-dimensional projection plane of the synchronized visual image.

[0037] Preferably, after successfully constructing the centroid mapping relationship, perspective point-related projection algebra is performed on the centroid mapping relationship to obtain the initial extrinsic transformation matrix. In this technical processing step, a pre-calibrated intrinsic parameter matrix of the camera acquiring the synchronous visual image is first introduced (this intrinsic parameter matrix is ​​a pre-configured data set containing inherent physical optical properties such as the camera's focal length and optical center pixel position), and a multi-view perspective projection algebra equation system is established in conjunction with the centroid mapping relationship. This perspective projection algebra equation system parameterizes the relative rotation and relative translation transformations between the three-dimensional perception space where the lidar acquiring the discrete point cloud sequence is located and the two-dimensional projection plane where the camera acquiring the synchronous visual image is located, forming a set of nonlinear equations containing multiple spatial transformation rotation position variables and spatial transformation translation unknown variables. By establishing this set of nonlinear equations, this application transforms the physical spatial pose difference between the camera and the lidar into a purely algebraic solution process, providing a rigorous basic computational framework for the subsequent derivation of the initial extrinsic transformation matrix.

[0038] Preferably, for the aforementioned set of nonlinear equations containing multiple unknown variables, it is necessary to perform matrix singular value decomposition and least squares error minimization algebraic operations to complete the closed loop of the perspective point-related projection algebraic solution. Specifically, the nonlinear equation set is expanded by matrix singular value decomposition to reduce its dimensionality, and a set of algebraic solutions is found that minimizes the sum of the spatial Euclidean distance errors between the three-dimensional geometric centroid coordinates reprojected onto the two-dimensional projection plane according to the equation set and the actual two-dimensional semantic centroid coordinates. This set of spatial transformation rotation and spatial transformation translation solutions that minimize the reprojection error is then matrix-recombined to obtain the initial extrinsic parameter transformation matrix. This initial extrinsic parameter transformation matrix initially completes the global pose coarse alignment between the dense point cloud of the scene and the synchronized visual image. Compared to directly and randomly selecting parameters as the search starting point in a complex natural scene, it greatly reduces the blind zone of subsequent spatial calculations and avoids the extrinsic parameter solution algorithm getting trapped in completely irrelevant local error response regions in the early stages.

[0039] Preferably, considering the uneven surface of real physical entities such as streetlights and buildings in natural scenes, the initial extrinsic transformation matrix obtained solely by relying on the centroid and centroid constraints at the macroscopic statistical level may still have slight pose residuals in the alignment of microscopic edge pixels. Therefore, the initial extrinsic transformation matrix needs to be used as the starting point for iterative optimization in a preset particle swarm optimization space. In this process, the rotation and translation parameters contained within the initial extrinsic transformation matrix are used as the geometric center origin of the multidimensional optimization space. Based on a pre-defined allowable range for parameter fluctuations, spatial perturbation variables are applied to the initial extrinsic transformation matrix to initialize and generate a biomimetic search particle swarm. This biomimetic search particle swarm contains multiple search state particles carrying different small pose perturbation offsets. These search state particles jointly initialize and construct the preset particle swarm optimization space. Each search state particle in this preset particle swarm optimization space objectively represents a state matrix formed by superimposing subtle spatial pose changes on the initial extrinsic transformation matrix, i.e., a candidate registration fine-tuning scheme.

[0040] Preferably, during the iterative optimization process in the preset particle swarm optimization space, it is necessary to introduce the three-dimensional physical coordinate group and the boundary metric mask generated in the aforementioned technical processing steps to verify and evaluate the duality of spatial solutions. Specifically, a large number of three-dimensional discrete points contained in the three-dimensional physical coordinate group belonging to the same target cluster instance are mapped to the plane where the boundary metric mask is located by performing cross-space perspective reprojection calculation through the state matrix of each particle in the biomimetic search particle swarm, thereby determining the feature projection landing point set. Subsequently, the update weights of these feature projection landing point sets at the corresponding coordinate positions in the boundary metric mask (i.e., the weights obtained by exponential distance decay weighting in the preprocessing) are read, and all the extracted update weights are algebraically accumulated to generate the consistency response value between the feature projection landing point set and the boundary metric mask. Based on this consistent response value, a target panoramic information matching degree is constructed. This target panoramic information matching degree objectively reflects the degree of fit between the reprojected three-dimensional discrete points and the edges of real physical entities. Then, the fitness evaluation value of each particle in the biomimetic search particle swarm is evaluated through the target panoramic information matching degree.

[0041] Preferably, the optimal exploration position of each particle and the global optimal exploration position of the entire swarm are extracted based on the fitness evaluation value. Simultaneously, linear decay calculations are performed according to the optimization evolution process to generate dynamic inertia adjustment weights. The current movement speed and distribution position of each particle are updated using the optimal exploration position, the global optimal exploration position, and the dynamic inertia adjustment weights to output the evolved state particle swarm, thereby driving the preset particle swarm optimization space to perform iterative updates of the swarm spatial position towards higher fitness regions. The evolved state particle swarm is then subjected to fitness evaluation processing again to obtain a verification evaluation result, and it is determined whether the verification evaluation result triggers a preset optimization convergence condition (e.g., the change in the highest fitness evaluation value in the verification evaluation result is lower than a preset threshold). In response to the failure to trigger the preset optimization convergence condition, the optimization iteration algebra is adjusted and the step of performing linear decay calculation based on the optimization evolution process is returned to continue iteration; in response to triggering the preset optimization convergence condition, the iterative optimization process is stopped, the global optimal exploration position corresponding to the verification evaluation result is converted into a calibration extrinsic parameter matrix, and it is output as the final scene space registration joint parameter. Through the above-mentioned hierarchical spatial solution design guided by macroscopic centroid constraints for coarse alignment and guided by microscopic mask numerical gradients for fine matching, this application effectively overcomes the calibration bottleneck of scattered multimodal features and severe illumination interference in natural scenes, so that the final output scene space registration joint parameter has high environmental robustness and accurately meets the requirements of accurate collaborative registration of multi-source data of automated inspection equipment in dynamic scenes.

[0042] Optionally, the step of inputting the initial extrinsic transformation matrix as the search starting point into a preset particle swarm optimization algorithm space for iterative optimization to generate the scene space registration joint parameters includes: applying a spatial perturbation variable to the initial extrinsic transformation matrix to initialize the generation of a biomimetic search particle swarm; mapping the three-dimensional physical coordinate group to the plane where the boundary metric mask map is located through the state matrix of each particle in the biomimetic search particle swarm to determine the feature projection landing point set; constructing a target panoramic information matching degree based on the consistency response value between the feature projection landing point set and the boundary metric mask map; and performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene space registration joint parameters.

[0043] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, in the specific technical implementation of applying spatial perturbation variables to the initial extrinsic parameter transformation matrix to initialize and generate a biomimetic search particle swarm, in order to overcome the microscopic attitude residuals caused by the unevenness of the surfaces of real physical entities such as streetlights and buildings in natural scenes, this application introduces a swarm intelligence optimization mechanism. Specifically, the initial extrinsic parameter transformation matrix derived in the preprocessing stage is used as the geometric center origin of the preset particle swarm optimization space. Around this geometric center origin, according to the pre-set allowable distribution range of parameter fluctuations (e.g., a rotational deviation range of ± a few degrees and a translational deviation range of ± a few centimeters), multiple spatial perturbation variables representing multidimensional spatial offsets are generated. Subsequently, these spatial perturbation variables are superimposed on the spatial transformation rotation solution and spatial transformation translation solution contained in the initial extrinsic parameter transformation matrix to construct multiple state matrices carrying different small attitude perturbation offsets, and the biomimetic search particle swarm is generated by each particle carrying these state matrices. Each particle (i.e., an independent individual) in this biomimetic search particle swarm objectively represents a candidate registration fine-tuning scheme based on its own state matrix.

[0044] Preferably, after successfully initializing the biomimetic search particle swarm, the three-dimensional physical coordinate group is mapped to the plane containing the boundary metric mask map through the state matrix of each particle in the biomimetic search particle swarm, to determine the feature projection landing point set. In this technical processing step, for each particle contained in the biomimetic search particle swarm, its corresponding state matrix is ​​extracted. At the same time, all three-dimensional discrete points contained in the three-dimensional physical coordinate group belonging to the same target cluster instance are obtained. Using the extracted state matrix, cross-space perspective reprojection calculation (including spatial rotation transformation and spatial translation transformation operations) is performed on all these three-dimensional discrete points one by one, thereby transforming these three-dimensional discrete points from the three-dimensional perception space and perspective reprojecting them onto the two-dimensional projection plane where the boundary metric mask map is located. All the two-dimensional projection landing point coordinate data that fall on the two-dimensional projection plane after the cross-space perspective reprojection calculation are aggregated and integrated to form the feature projection landing point set that corresponds one-to-one with the state matrix.

[0045] Preferably, after obtaining the set of feature projection landing points corresponding to each particle, a target panoramic information matching degree is constructed based on the consistency response value between the set of feature projection landing points and the boundary metric mask. Specifically, the operation involves: traversing the coordinates of each two-dimensional projection landing point in the set of feature projection landing points; and in the pre-constructed boundary metric mask with continuous gradient change characteristics, searching for pixels whose spatial positions coincide with those of the two-dimensional projection landing point coordinates. Then, the update weights stored within these pixels (i.e., the internal weights obtained through exponential distance decay weighting in the preprocessing, with higher update weights closer to the edge of the real physical entity) are read. All update weights extracted for the particle are then algebraically summed to generate the consistency response value, which measures the degree of fit between the current perspective reprojection result and the target segmentation boundary (i.e., the projection position of the edge of the real physical entity).

[0046] Preferably, in order to establish a unified evaluation system for the entire biomimetic search particle swarm, it is necessary to construct the target panoramic information matching degree based on the obtained consistency response value. Specifically, in the conversion processing mechanism, the consistency response value is divided by the total number of landing points contained in the feature projection landing point set, and a normalized algebra operation is performed to eliminate the influence of target size differences on the evaluation benchmark. The algebraic value output after the normalized algebra operation is defined as the target panoramic information matching degree. This target panoramic information matching degree objectively characterizes the cross-modal alignment accuracy between the three-dimensional discrete points in the three-dimensional perception space and the semantic edge features in the two-dimensional projection plane under the influence of the current particle's state matrix. The higher the target panoramic information matching degree value, the tighter the multimodal feature fit.

[0047] Preferably, after evaluating all particles, the fitness evaluation value of each particle in the biomimetic search particle swarm is assessed using the target panoramic information matching degree, and iterative optimization is performed on the biomimetic search particle swarm based on the target panoramic information matching degree. Specifically, the target panoramic information matching degree is directly mapped to the fitness evaluation value used to measure the quality of particles. Based on the fitness evaluation value, the exploration position corresponding to the highest fitness evaluation value reached by each particle in the optimization history is back-recorded and extracted as the individual optimal exploration position for each particle; simultaneously, the exploration position corresponding to the highest fitness evaluation value reached by the entire biomimetic search particle swarm in all current iterations is extracted and used as the global optimal exploration position for the entire swarm. Subsequently, a linear decay calculation is performed based on the time span of the optimization evolution process to generate dynamic inertia adjustment weights for balancing global search and local mining capabilities. By utilizing the guiding direction of the individual optimal exploration position and the guiding direction of the global optimal exploration position, and combining the dynamic inertia adjustment weight, the current movement speed and distribution position of each particle in the biomimetic search particle swarm are updated to output the evolutionary state particle swarm, thereby driving the entire swarm to migrate spatially to a high fitness region.

[0048] Preferably, as the above-mentioned projection mapping, fitness evaluation, and position update are executed repeatedly, the particles in the preset particle swarm optimization space will gradually converge. At the end of this iterative optimization process, the scene space registration joint parameters are determined based on the evolved state particle swarm. Specifically, the judgment mechanism is as follows: after outputting the evolved state particle swarm in each round, the fitness evaluation process is performed again on the evolved state particle swarm (i.e., the latest fitness evaluation value of the evolved particles is recalculated) to obtain a verification evaluation result. Subsequently, it is determined whether the verification evaluation result triggers the preset optimization convergence condition. The specific judgment logic is as follows: Monitor the change in the highest fitness evaluation value in the review evaluation results. When the increase in the highest fitness evaluation value is lower than the preset algorithm convergence tolerance threshold within multiple consecutive iterations, it is determined that the preset optimization convergence condition has been triggered. In response to not triggering the preset optimization convergence condition, adjust the optimization iteration number and return to the step of performing linear decay calculation based on the optimization evolution process to generate dynamic inertia adjustment weights, continuing the next round of iterative iteration. In response to triggering the preset optimization convergence condition, stop the iterative optimization process, convert the global optimal exploration position corresponding to the review evaluation result into a calibration extrinsic parameter matrix, and output it as the final scene space registration joint parameter. This hierarchical fine-tuning optimization design effectively avoids matching errors caused by noise interference from natural scene environments, resulting in high environmental robustness of the final generated scene space registration joint parameters, which better meets the needs of automated inspection equipment for multi-source data collaborative registration in dynamic scenes.

[0049] Optionally, the step of performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene spatial registration joint parameters includes: evaluating the fitness evaluation value of each particle in the biomimetic search particle swarm through the target panoramic information matching degree; extracting the individual optimal exploration position of each particle and the global optimal exploration position of the entire swarm based on the fitness evaluation value; performing linear decay calculation according to the optimization evolution process to generate dynamic inertia adjustment weights; using the individual optimal exploration position, the global optimal exploration position, and the dynamic inertia adjustment weights to update the current movement speed and distribution position of each particle to output the evolved state particle swarm, and determining the scene spatial registration joint parameters based on the evolved state particle swarm.

[0050] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, the specific technical implementation of evaluating the fitness evaluation value of each particle in the biomimetic search particle swarm through the target panoramic information matching degree requires transforming the target panoramic information matching degree generated in the preprocessing stage into a metric standard driving the evolution of the swarm intelligence algorithm. Specifically, for each particle in the biomimetic search particle swarm (i.e., an independent individual representing a candidate registration fine-tuning scheme), the target panoramic information matching degree calculated in the previous technical processing stage based on the feature projection landing point set and the boundary metric mask map is extracted. This target panoramic information matching degree, in a numerical form after normalized algebraic operations, objectively characterizes the cross-modal alignment accuracy between the three-dimensional discrete point and the target segmentation boundary (i.e., the projection position of the real physical entity edge) under the action of the current particle's state matrix. The extracted target panoramic information matching degree is directly assigned to the corresponding particle, serving as the fitness evaluation value for measuring the particle's current spatial solution performance. The higher the fitness evaluation value, the closer the state matrix represented by the particle is to the cross-modal mapping relationship of the real physical entity.

[0051] Preferably, after assigning fitness evaluation values ​​to all particles, the individual optimal exploration position of each particle and the global optimal exploration position of the entire swarm are extracted based on the fitness evaluation values. In this process, firstly, a longitudinal historical backtracking is performed on each particle in the biomimetic search particle swarm, comparing all fitness evaluation values ​​obtained in each iteration since initialization; the highest fitness evaluation value is selected, and the spatial distribution position of the particle when it obtained this highest fitness evaluation value is recorded, extracted as the individual optimal exploration position of that particle. Simultaneously, in the horizontal dimension of the entire biomimetic search particle swarm, all fitness evaluation values ​​generated by all particles in the current iteration and historical iterations are collected; through global extreme value search, the largest fitness evaluation value is located, and the spatial distribution position of the particle that generated this largest fitness evaluation value is recorded, extracted as the global optimal exploration position of the entire swarm. The individual optimal exploration position and the global optimal exploration position represent the technical guidance of local optimization memory and global optimization collaboration, respectively.

[0052] Preferably, after obtaining the reference benchmark for guiding position updates (i.e., the individual optimal exploration position and the global optimal exploration position), a linear decay calculation is performed according to the optimization evolution process to generate a dynamic inertia adjustment weight. Considering that in the early stages of iterative optimization, particles need a larger exploration range to avoid getting trapped in local convergence dead zones, while in the later stages of iterative optimization, the particle activity range needs to be reduced for fine-tuning, a pre-set maximum total number of optimization iterations and the current optimization iteration algebra being executed are obtained. Using the ratio of the maximum total number of optimization iterations to the current optimization iteration algebra as a time evolution variable, combined with a pre-configured initial inertia upper limit and a final inertia lower limit, a linear decreasing algebra operation is performed. As the optimization iteration algebra increases, the value output by this linear decreasing algebra operation gradually decreases, and this real-time changing output value is defined as the dynamic inertia adjustment weight. This dynamic inertia adjustment weight objectively regulates the strength at which each particle maintains its original current movement speed at the algorithm level, achieving a smooth transition from global coarse search to local accurate mining.

[0053] Preferably, after preparing all the parameter variables required for the update, the current movement speed and current spatial distribution position of each particle are updated using the individual optimal exploration position, the global optimal exploration position, and the dynamic inertia adjustment weight to output the evolved state particle swarm. The specific algebraic update logic is as follows: For each particle in the biomimetic search particle swarm, firstly, the spatial coordinate difference between its current spatial distribution position and the individual optimal exploration position is calculated to generate a local position deviation vector, and then the spatial coordinate difference between its current spatial distribution position and the global optimal exploration position is calculated to generate a global position deviation vector; subsequently, the particle's current movement speed is multiplied by the dynamic inertia adjustment weight, and the result of the multiplication is algebraically added to the local position deviation vector and the global position deviation vector to obtain the particle's evolved movement speed; next, the particle's current spatial distribution position is algebraically accumulated with its unique evolved movement speed to generate the particle's evolved spatial distribution position. Once all particles in the biomimetic search particle swarm have completed the iterative changes in their current spatial distribution position and current movement speed, the swarm containing the evolved spatial distribution positions of all particles is aggregated and output as the evolved state particle swarm, thereby driving the entire swarm to make a substantial spatial migration to a high fitness region.

[0054] Preferably, after outputting the evolved state particle swarm, the scene spatial registration joint parameters are determined based on the evolved state particle swarm. To achieve this technical objective, the fitness evaluation process needs to be performed on the evolved state particle swarm again. Specifically, the updated state matrix of each particle in the evolved state particle swarm, obtained based on the spatial distribution position transformation after evolution, is extracted. Following the computational logic recorded in the previous technical processing step, the updated state matrix is ​​reused to perform cross-spatial perspective reprojection calculations on all the three-dimensional discrete points to regenerate the feature projection landing point set. The corresponding updated weights are read from the boundary metric mask map and algebraically accumulated to generate the consistency response value. Then, normalized algebraic operations are used to generate the latest target panoramic information matching degree, which is then used as the latest fitness evaluation value for each evolved particle. This set of all fitness evaluation values ​​calculated for the evolved swarm is summarized and extracted to obtain the verification evaluation result. The review and evaluation results objectively reflect the latest detection status of the particle swarm's accuracy in fitting multimodal features of natural scenes after undergoing spatial location migration and evolution.

[0055] Preferably, after obtaining the verification evaluation result, it is determined whether the verification evaluation result triggers the preset optimization convergence condition. The specific determination logic is as follows: extract the value of the current highest fitness evaluation value from the verification evaluation result, and calculate the increase in value between this value and the highest fitness evaluation value recorded in the previous iteration round; when the increase in value is lower than the preset algorithm convergence tolerance threshold in multiple consecutive iteration rounds, it is determined that the preset optimization convergence condition has been triggered, indicating that the optimization process of the entire particle swarm has reached a stable state. At this time, in response to triggering the preset optimization convergence condition, the subsequent iterative optimization process is stopped; subsequently, the global optimal exploration position corresponding to the verification evaluation result (i.e., the evolved spatial distribution position with the current highest fitness evaluation value in the final convergence state) is extracted, and it is reverse-analyzed into a combination of spatial rotation parameter matrix and spatial translation parameter matrix, that is, converted into a calibration extrinsic parameter matrix, and output as the final scene spatial registration joint parameter. This hierarchical fine spatial calculation mechanism, which combines swarm intelligence with the matching degree of the target panoramic information, effectively overcomes the matching error caused by environmental noise interference in natural scenes. As a result, the final generated scene spatial registration joint parameters have high environmental robustness and spatial calculation accuracy, which better meets the accurate collaborative calibration requirements of automated inspection equipment in dynamic multi-source environmental perception scenarios.

[0056] Optionally, determining the scene space registration joint parameters based on the evolved state particle swarm includes: performing fitness evaluation processing on the evolved state particle swarm again to obtain a verification evaluation result; determining whether the verification evaluation result triggers a preset optimization convergence condition; in response to not triggering the preset optimization convergence condition, adjusting the optimization iteration number and returning to the linear decay calculation performed according to the optimization evolution process to generate dynamic inertia adjustment weights; in response to triggering the preset optimization convergence condition, converting the global optimal exploration position corresponding to the verification evaluation result into a calibration extrinsic parameter matrix and using it as the scene space registration joint parameters.

[0057] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, the specific technical implementation of performing fitness evaluation processing on the evolved state particle swarm again to obtain the verification evaluation results requires verifying whether each particle, after undergoing spatial position migration and evolution, truly improves the cross-modal alignment accuracy of multimodal data. Specifically, for each particle in the evolved state particle swarm, the evolved spatial distribution position generated after updating its current movement speed and current spatial distribution position in the current iteration is extracted. This evolved spatial distribution position objectively represents, on the algebraic data surface, a candidate state parameter combination containing spatial rotation and translation parameters recently explored by the current particle. This newly explored candidate state parameter combination is matrix-formatted and reassigned to the particle as its updated state matrix.

[0058] Preferably, after assigning the updated state matrix to each particle in the evolved particle swarm, according to the operational logic recorded in the pre-processing stage, the updated state matrix is ​​used to re-perform cross-spatial perspective reprojection calculations for all the three-dimensional discrete points belonging to the same target cluster instance. This cross-spatial perspective reprojection calculation process reprojects the three-dimensional discrete points in the three-dimensional perception space onto the two-dimensional projection plane based on the spatial attitude relationships contained in the updated state matrix, thereby generating an updated feature projection landing point set for each particle in the current iteration stage. Subsequently, the coordinates of each two-dimensional projection landing point in the updated feature projection landing point set are traversed, and the pixels with overlapping spatial positions are located in the boundary metric mask map, and the corresponding updated weights are read.

[0059] Preferably, an algebraic summation operation is performed on all the updated weights extracted for the evolved particle to generate a consistency response value. Subsequently, based on the consistency response value, the influence of target size differences is eliminated through normalized algebraic operations, thereby calculating the latest target panoramic information matching degree of the particle in the current iteration round. This latest target panoramic information matching degree is directly assigned to the particle as the latest fitness evaluation value to measure the quality of the particle's current evolutionary state. After recalculating the latest fitness evaluation values ​​of all particles in the evolved state particle swarm, this set of fitness evaluation values ​​containing the latest exploration state of the entire swarm is summarized and extracted to obtain the verification evaluation result. This verification evaluation result objectively quantifies the degree of fit between the candidate registration fine-tuning scheme represented by the updated state matrix and the target segmentation boundary (i.e., the projection position of the real physical entity edge) of each particle after the current round of swarm evolution.

[0060] Preferably, after obtaining the verification evaluation result, it is determined whether the verification evaluation result triggers a preset optimization convergence condition. The specific determination logic is as follows: The latest fitness evaluation value with the largest value is selected from the verification evaluation results and marked as the current highest fitness evaluation value. Simultaneously, the highest fitness evaluation value recorded at the end of the previous iteration round is retrieved from the entire particle swarm. An algebraic subtraction operation is performed between the current highest fitness evaluation value and the highest fitness evaluation value recorded in the previous iteration round to calculate the increase in the highest fitness evaluation value between two adjacent iteration rounds. Subsequently, this increase in value is compared and verified with a preset algorithm convergence tolerance threshold. This preset algorithm convergence tolerance threshold is a small numerical constant used to objectively define the tolerance boundary between when the swarm intelligence optimization algorithm stagnates or has found the global optimum.

[0061] Preferably, in determining whether the review evaluation result triggers the preset optimization convergence condition, there are two possible branch paths. If the preset optimization convergence condition is not triggered (i.e., the increase in the highest fitness evaluation value is greater than or equal to the preset algorithm convergence tolerance threshold, indicating that the optimization process of the entire particle swarm is still in an active exploratory state and has not yet stabilized), then the optimization iteration number is adjusted. Specifically, the current optimization iteration number is increased by a fixed step, and carrying the evolutionary state of the particle swarm, it returns to the step of performing linear decay calculations based on the optimization evolution process to generate dynamic inertia adjustment weights, in order to enter the next round of updating the current movement speed and current spatial distribution position of each particle.

[0062] Preferably, in response to triggering the preset optimization convergence condition (i.e., determining that in multiple consecutive iterations, the increase in the highest fitness evaluation value is lower than the preset algorithm convergence tolerance threshold, indicating that the entire particle swarm has converged to the global optimum or near-optimal state), the iterative optimization process is stopped. At this time, the evolved spatial distribution position of the particle that generated the current highest fitness evaluation value is extracted from the review evaluation results. This evolved spatial distribution position is the globally optimal exploration position determined in the final convergence state. Subsequently, the globally optimal exploration position is reverse-analyzed into a combination of a spatial rotation parameter matrix and a spatial translation parameter matrix, i.e., converted into a calibration extrinsic parameter matrix. This calibration extrinsic parameter matrix specifically includes two parameters: a three-dimensional rotation matrix component and a three-dimensional translation vector component. Specifically, the three-dimensional rotation matrix component is used to correct the angular deviations in pitch, yaw, and roll between the lidar acquiring the discrete point cloud sequence and the camera acquiring the synchronous visual image. The three-dimensional translation vector component is used to compensate for the physical distance deviations between the lidar's coordinate origin and the camera's coordinate origin in the horizontal, vertical, and depth spatial axes. Finally, this calibration extrinsic parameter matrix is ​​output as the scene spatial registration joint parameter. This scene spatial registration joint parameter can be directly applied to the multi-source sensor data fusion system of the subsequent inspection robot to accurately achieve online collaborative alignment between the dense point cloud of the scene and the synchronous visual image.

[0063] Preferably, in the application scenario where the inspection robot performs multi-source environmental perception in complex natural scenes, the process of extracting and applying the specific parameters within the calibration extrinsic parameter matrix is ​​a key step in implementing the abstract scene space registration joint parameters into specific physical space correction. In specific technical implementation, the calibration extrinsic parameter matrix output from the preceding technical steps undergoes algebraic decoupling and decomposition processing to separate the three-dimensional rotation matrix component representing the spatial angle transformation relationship, and the three-dimensional translation vector component representing the spatial position movement relationship. The three-dimensional rotation matrix component and the three-dimensional translation vector component together constitute the core mathematical carrier of the aforementioned scene space registration joint parameters. Through this algebraic decoupling and decomposition processing, the complex six-degree-of-freedom spatial transformation relationship is decoupled into mutually independent angle rotation mapping rules and physical distance translation rules, providing clear transformation rules for subsequent step-by-step spatial coordinate system alignment of the three-dimensional discrete points belonging to the same target cluster instance.

[0064] Preferably, the decomposed three-dimensional rotation matrix components are subjected to matrix element reading processing to extract pitch angle rotation parameters characterizing the vertical pitch angle transformation law. In the physical environment where the inspection robot travels in a natural scene, due to road bumps or installation tolerances of the sensor bracket, there is often a relative angular difference in vertical tilt between the lidar that collects the discrete point cloud sequence and the camera that collects the synchronous visual image (e.g., the camera lens is tilted upwards by two degrees relative to the lidar's emitting surface). The extracted pitch angle rotation parameters are applied to each of the three-dimensional discrete points under the same target cluster instance, and a first-level rotation transformation process is performed to generate pitch-corrected discrete points. These pitch-corrected discrete points objectively eliminate the tilt deviation between the lidar and the camera in the vertical field of view direction, so that the three-dimensional discrete points maintain a high geometric parallelism with the two-dimensional projection plane in the vertical field of view dimension.

[0065] Preferably, while extracting the pitch angle rotation parameters, the components of the three-dimensional rotation matrix are processed by matrix element reading to extract the yaw angle rotation parameters used to characterize the left and right yaw angle conversion law. In a physical installation scenario, the scanning center plane of the lidar and the optical center plane of the camera may have a left-right deflection in the horizontal direction (for example, the camera lens is deflected to the left by one degree relative to the center of the lidar). For all the pitch-corrected discrete points generated in the previous step, a second-level rotation transformation process is performed using the yaw angle rotation parameters to generate yaw-pitch dual-corrected discrete points. All the yaw-pitch dual-corrected discrete points successfully compensate for the left-right orientation misalignment of the lidar and the camera in the horizontal field of view direction, ensuring that the three-dimensional coordinates of the discrete points have a relatively accurate orientation consistency with the imaging coordinate system of the camera in the horizontal unfolding direction.

[0066] Preferably, the components of the three-dimensional rotation matrix are further processed by matrix element reading to extract the roll angle rotation parameter, which characterizes the rotation law around the line of sight axis. Due to the stress deformation of the mechanical structure of the inspection robot, the horizontal scanning reference plane of the lidar and the horizontal imaging reference plane of the camera may undergo clockwise or counterclockwise relative twisting (for example, the camera is slightly tilted clockwise relative to the lidar). The roll angle rotation parameter is used to perform a third-level rotation transformation process on all the yaw and pitch dual-correction discrete points generated in the previous step to generate full-attitude rotation correction discrete points. After these three progressive angle correction transformations, all the full-attitude rotation correction discrete points objectively eliminate the angular deviation in pitch, yaw, and roll angles between the lidar that acquires the discrete point cloud sequence and the camera that acquires the synchronous visual image, thus achieving pure alignment of spatial attitude orientation.

[0067] Preferably, after aligning the attitude angles, the decomposed three-dimensional translation vector components are subjected to vector dimension splitting to extract the horizontal and vertical translation parameters. In the multi-layer sensor layout of the inspection robot, the LiDAR and the camera are usually not installed at the same physical absolute center point, and there must be a baseline distance (for example, the camera is installed 10 cm directly above the LiDAR and 2 cm to the right). Using the horizontal and vertical translation parameters, planar displacement compensation processing is performed on all the full attitude rotation correction discrete points to generate coplanar displacement correction discrete points. All the coplanar displacement correction discrete points effectively offset the physical distance deviation between the coordinate origin of the LiDAR and the coordinate origin of the camera in the horizontal and vertical spatial axes, so that the discrete point data forms a more accurate correspondence with the visual pixels on the two-dimensional projection plane.

[0068] Preferably, the three-dimensional translation vector component is then subjected to vector dimension splitting to extract depth translation parameters. These depth translation parameters objectively reflect the physical installation hierarchy differences between the lidar and the camera in terms of forward and backward observation distances (e.g., the center of the camera's sensor is recessed by five centimeters compared to the lidar's emission center). The depth translation parameters are then used to perform forward and backward depth-of-field displacement compensation processing on all the coplanar displacement correction discrete points to generate the final three-dimensional full-scale correction discrete points. All the three-dimensional full-scale correction discrete points offset the physical distance deviation between the lidar's coordinate origin and the camera's coordinate origin along the depth space axis. Through the independent decoupling and progressive application of the parameters within the three-dimensional rotation matrix component and the three-dimensional translation vector component, this application constructs a logically rigorous cross-modal spatial transformation link, greatly reducing the spatial displacement error during joint calibration of the camera and lidar in complex natural scenes, and providing a stable and reliable multimodal collaborative data foundation for inspection robots to achieve accurate target ranging and environmental semantic understanding.

[0069] Optionally, the step of generating a scene-dense point cloud based on a discrete point cloud sequence collected by the robot in a natural scene includes: separating the current observation frame point cloud and the historical observation frame point cloud from the discrete point cloud sequence; performing voxel downsampling on the current observation frame point cloud and the historical observation frame point cloud to obtain a spatially sparse point cloud; performing local geometric difference filtering on the spatially sparse point cloud to extract a spatially distributed sparse feature point set; performing local geometric morphology encoding on the spatially distributed sparse feature point set to generate a structural feature descriptor; and performing pose transformation operation on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene-dense point cloud.

[0070] Preferably, in applications where inspection robots perform multi-source environmental perception in complex natural scenes, the discrete point cloud of a single frame is often sparse due to viewpoint occlusion and uneven beam distribution, which is insufficient to support high-precision multimodal feature extraction. Therefore, a spatiotemporal fusion three-dimensional representation needs to be constructed. In the specific technical implementation of separating the current observation frame point cloud and historical observation frame point clouds from the discrete point cloud sequence, firstly, according to the scanning timestamp order of the LiDAR, the latest acquired frame of discrete point cloud data is extracted from the collected discrete point cloud sequence and defined as the current observation frame point cloud. This current observation frame point cloud objectively records the instantaneous three-dimensional reflection sampling results of the inspection robot on the geometric shape of the surrounding environment surface at the current spatial position. Simultaneously, based on a pre-set time backtracking window (e.g., backtracking data within the past ten seconds), the previous frame or multiple frames of discrete point cloud data that are temporally adjacent to the current observation frame point cloud are extracted from the discrete point cloud sequence, and the set of these discrete point cloud data is defined as the historical observation frame point cloud. The historical observation frame point cloud preserves the past three-dimensional reflection sampling results of the inspection robot from different perspectives of the same physical scene.

[0071] Preferably, after acquiring the current observation frame point cloud and the historical observation frame point cloud with differences in time span, in order to reduce the computational redundancy of subsequent feature matching, a voxel downsampling operation is performed on the current observation frame point cloud and the historical observation frame point cloud to obtain a spatially sparse point cloud. Specifically, the three-dimensional perception space is divided into several uniformly sized voxel grids, and then all discrete points in the current observation frame point cloud and the historical observation frame point cloud are mapped to these voxel grids. For each voxel grid containing at least one discrete point, the algebraic average of the three-dimensional spatial coordinates of all discrete points within that voxel grid is calculated, and this algebraic average is used to generate a new representative point coordinate, which replaces all the original discrete points within that voxel grid. This voxel downsampling operation significantly reduces the amount of point cloud data while preserving the macroscopic geometric topology of the scene, thereby converting the current observation frame point cloud and the historical observation frame point cloud into the dimensionality-reduced spatially sparse point cloud.

[0072] Preferably, in order to extract stable anchor points for inter-frame registration from massive point cloud data, local geometric difference filtering processing is performed on the spatially sparse point cloud to extract a spatially distributed sparse feature point set. In this processing step, for the spatially sparse point cloud corresponding to the current observation frame point cloud and the historical observation frame point cloud respectively, each representative point coordinate is traversed, and several nearest representative point coordinates in three-dimensional space are searched for. The rate of curvature change of the local surface normal vector formed by the representative point coordinate and the nearest representative point coordinate is calculated. This rate of curvature change objectively reflects the degree of unevenness of the local point cloud surface. Subsequently, a rate of curvature change threshold is set, and representative point coordinates with a rate of curvature change greater than the rate of curvature change threshold (usually corresponding to structurally abrupt regions such as building edges and road sign corners in natural scenes) are filtered out and combined to form the spatially distributed sparse feature point set. The spatially distributed sparse feature point set filters out redundant and representative point coordinates that lack distinctiveness, such as flat road surfaces or smooth walls, and extracts spatially distributed sparse feature points with high geometric distinguishability in natural scenes.

[0073] Preferably, after extracting spatially distributed sparse feature points with high geometric discriminative power, it is necessary to establish a rotation- and translation-resistant mathematical description for these spatially distributed sparse feature points. Therefore, local geometric morphology encoding is performed on the set of spatially distributed sparse feature points to generate a structural feature descriptor. Specifically, for each spatially distributed sparse feature point in the set, a local sensing sphere with a specific radius is constructed with that sparse feature point as its center. Inside this local sensing sphere, a relative spatial distribution histogram of the coordinates of its nearest representative points relative to the spatially distributed sparse feature point at the sphere's center is statistically analyzed. For example, the distribution of the coordinates of the nearest representative points in different azimuth and pitch angle intervals, or the distribution of the angle between normal vectors, is statistically analyzed. The data from these relative spatial distribution histograms are concatenated according to a fixed vector dimension to form a multidimensional numerical vector, which is the structural feature descriptor specific to that spatially distributed sparse feature point. The structural feature descriptor transforms the isolated spatially distributed sparse feature points into an algebraic vector representation containing rich local surface topological information, providing a high-dimensional metric for subsequent feature point matching.

[0074] Preferably, after extracting the spatially distributed sparse feature point set and generating the structural feature descriptor, it is necessary to address the spatial pose misalignment problem between the current observation frame point cloud and the historical observation frame point cloud. The technical logic for performing pose transformation operations on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene dense point cloud is as follows: First, based on the nearest neighbor metric relationship of the structural feature descriptor (i.e., calculating the Euclidean distance similarity between the structural feature descriptors), highly similar pairs of spatially distributed sparse feature points are searched and selected from the spatially distributed sparse feature point set corresponding to the current observation frame point cloud and the historical observation frame point cloud. These pairs of spatially distributed sparse feature point points are used as initial homonymous feature point pairs. Subsequently, based on the spatial coordinate correspondence between these initial homonymous feature point pairs, an initial coarse registration transformation matrix containing a spatial rotation matrix and a translation vector is calculated using a singular value decomposition algorithm. The initial coarse registration transformation matrix can perform preliminary rotation and translation of the historical observation frame point cloud in the three-dimensional perception space, so that its overall posture is closer to the reference coordinate system of the current observation frame point cloud.

[0075] Preferably, the initial coarse registration transformation matrix is ​​used to perform an attitude transformation operation on the historical observation frame point cloud, mapping the historical observation frame point cloud to the reference coordinate system of the current observation frame point cloud. Due to errors in the initial attitude transformation operation, a small distance deviation still exists between the transformed historical observation frame point cloud and the current observation frame point cloud, thus establishing a nearest neighbor point pair distance error space between them. Within this nearest neighbor point pair distance error space, the nearest point iterative minimization solution is performed on the transformed historical observation frame point cloud and the current observation frame point cloud. Specifically, the closest discrete point pair between the transformed historical observation frame point cloud and the current observation frame point cloud is repeatedly searched iteratively, and a spatial transformation matrix for fine-tuning is continuously calculated and updated based on this closest discrete point pair. This spatial transformation matrix is ​​used to continuously correct the pose of the historical observation frame point cloud until the sum of squared overall distance errors between the transformed historical observation frame point cloud and the current observation frame point cloud reaches a minimum value. Through a series of attitude transformation operations and meticulous nearest-point iterative minimization optimization, the historical observation frame point clouds are accurately stitched into the current observation frame point cloud, filling in the blank areas caused by occlusion or sparse line bundles in the single-frame current observation frame point cloud, and finally fusing to generate the scene dense point cloud. This scene dense point cloud has significantly improved spatial resolution and structural integrity compared to the single-frame discrete point cloud sequence, providing a high-quality 3D geometric information base for subsequent accurate alignment of cross-modal multi-sensor features.

[0076] Optionally, the step of performing attitude transformation operations on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene dense point cloud includes: selecting initial homonymous feature point pairs from the spatially distributed sparse feature point sets corresponding to the current observation frame point cloud and the historical observation frame point cloud based on the nearest neighbor metric relationship of the structural feature descriptor; generating an initial coarse registration transformation matrix based on the initial homonymous feature point pairs; mapping the historical observation frame point cloud to the reference coordinate system of the current observation frame point cloud using the initial coarse registration transformation matrix to establish a nearest neighbor point pair distance error space; and performing nearest point iterative minimization optimization on the nearest neighbor point pair distance error space to generate the scene dense point cloud.

[0077] Preferably, in the application scenario where inspection robots perform multi-source environmental perception in complex natural scenes, the specific technical implementation of selecting initial pairs of identically named feature points from the spatially distributed sparse feature points corresponding to the current observation frame point cloud and the historical observation frame point cloud based on the nearest neighbor metric relationship of the structural feature descriptors needs to solve the identity matching problem of spatially distributed sparse feature points in the cross-frame time domain. Specifically, all structural feature descriptors corresponding to the current observation frame point cloud are extracted, and the set of these structural feature descriptors is used as a matching benchmark library; simultaneously, all structural feature descriptors corresponding to the historical observation frame point cloud are extracted, and the set of these structural feature descriptors is used as a query library to be matched. Each structural feature descriptor in the query library to be matched is traversed, and the candidate structural feature descriptor with the closest Euclidean distance is searched in the matching benchmark library. This nearest neighbor metric relationship can be quantified by calculating the Euclidean distance between the multidimensional numerical vectors (i.e., the structural feature descriptors themselves) of the query library to be matched and the matching benchmark library. When the calculated Euclidean distance is less than a pre-set descriptor similarity threshold, the feature is considered to be successfully matched. The two successfully matched structural feature descriptors, representing the sparsely distributed spatial feature points belonging to the current observation frame point cloud and the historical observation frame point cloud respectively, are combined to form an initial pair of feature points with the same name that corresponds across frames.

[0078] Preferably, after completing the cross-frame feature point association, an initial coarse registration transformation matrix is ​​generated based on all the initial identical feature point pairs. In this technical processing step, since all the initial identical feature point pairs are obtained by independent matching based on the structural feature descriptors, they are inevitably subject to interference from similar structures in the natural scene (e.g., multiple street lamp poles with identical appearances), resulting in mismatches. Therefore, a random sampling consensus algorithm mechanism is introduced for algebraic screening. Specifically, a very small number (e.g., three) of the initial identical feature point pairs are randomly selected from all the initial identical feature point pairs. Based on the spatial coordinate mapping relationship of this initial identical feature point pair subset, a temporary spatial transformation parameter combination (which includes temporary spatial rotation and translation parameters) is solved through matrix singular value decomposition. Subsequently, this temporary spatial transformation parameter combination is applied to all the remaining initial identical feature point pairs, and the number of interior identical feature point pairs that can satisfy the projection error tolerance is counted. After multiple cyclic samplings and statistical calculations of the inlier feature point pairs, the temporary spatial transformation parameter combination corresponding to the sampling result with the largest number of inlier feature point pairs is extracted. This optimal set of temporary spatial transformation parameter combinations is then matrix-fixed to generate the initial coarse registration transformation matrix used for global attitude coarse alignment.

[0079] Preferably, after obtaining the initial coarse registration transformation matrix that can characterize the large-scale spatial relative displacement relationship between frames, the initial coarse registration transformation matrix is ​​used to map the historical observation frame point cloud to the reference coordinate system of the current observation frame point cloud to establish a nearest neighbor point pair distance error space. Specifically, the initial coarse registration transformation matrix is ​​used as a spatial transformation multiplier, and algebraic multiplication and addition operations are performed with the three-dimensional coordinate vector of each discrete point contained in the historical observation frame point cloud. This performs a rigid body attitude transformation operation on the entire historical observation frame point cloud in the three-dimensional perception space. After this attitude transformation operation, the historical observation frame point cloud and the current observation frame point cloud are uniformly incorporated into the same reference coordinate system (i.e., the three-dimensional physical coordinate system of the inspection robot when collecting the current observation frame point cloud). At this point, for each discrete point in the transformed historical observation frame point cloud, the corresponding discrete point with the closest physical spatial Euclidean distance is searched in the current observation frame point cloud, thereby forming a massive number of nearest neighbor discrete point pairings. The set of spatial distance differences between these nearest neighbor discrete point pairings objectively constructs the nearest neighbor point pair distance error space.

[0080] Preferably, to further eliminate the local pose residuals introduced by coarse registration, a nearest-neighbor point pair distance error space is optimized by performing a nearest-point iterative minimization solution to generate the scene's dense point cloud. Within the nearest-neighbor point pair distance error space, a global objective loss function is defined, consisting of the sum of squared Euclidean distances between all the nearest discrete point pairs. Subsequently, a nonlinear optimization algorithm is used to continuously calculate and update a spatial correction matrix (containing small increments in rotation angle and translation distance) for fine-tuning along the gradient direction of the global objective loss function. After each round of fine-tuning, the updated spatial correction matrix is ​​used to fine-tune the overall pose of the historical observation frame point cloud again, and new nearest discrete point pairs are re-established in the nearest-neighbor point pair distance error space.

[0081] Preferably, during the nearest-point iterative minimization optimization process, the steps of establishing the nearest-neighbor discrete point pairing group, updating the spatial correction matrix, and fine-tuning the overall pose of the historical observation frame point cloud are executed cyclically. To prevent the algorithm from getting stuck in an infinite loop, an iteration termination threshold is set (e.g., the change in the global objective loss function is lower than a certain minimum constant, or the maximum number of iterations is reached). When the optimization process triggers this iteration termination threshold, it indicates that the historical observation frame point cloud has reached the optimal geometric fit with the current observation frame point cloud at the microscopic level, and the iteration stops at this point.

[0082] Preferably, after stopping the nearest point iterative minimization optimization, the historical observation frame point cloud, after undergoing multiple attitude transformation operations and fine-tuning alignment, is directly fused and stitched with the current observation frame point cloud at the 3D data structure level. During the stitching process, redundant discrete points with extremely close spatial coordinates are subjected to a second filtering and culling process based on voxel meshes to maintain the uniformity of the fused point cloud data density. The 3D point cloud data set output after the above stitching and second filtering and culling processes is the final scene dense point cloud. Through this hierarchical spatiotemporal fusion mechanism guided by the structural feature descriptor for coarse registration, and then supplemented by iterative optimization for fine alignment, this application effectively overcomes the technical bottleneck of sparse and severely occluded single-frame point clouds in the dynamic movement of inspection robots. The final generated scene dense point cloud has significantly improved spatial resolution and target structure integrity, providing a 3D geometric information base with high information abundance and a relatively stable structure for the subsequent instantiation of visual pixels and laser point clouds.

[0083] like Figure 2 As shown, this embodiment of the present application provides a camera-LiDAR joint calibration device based on natural scene features, which includes: The first program unit is used to generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. The second program unit is used to instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively, so as to determine the target region at the image end and the target cluster in the point cloud, and to perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map. The third program unit is used to locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image based on the boundary metric mask map, and to calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. The fourth program unit is used to establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and to perform spatial calculation on them to generate scene spatial registration joint parameters.

[0084] like Figure 3 As shown, an electronic device according to an embodiment of this application includes a memory and a processor. The memory stores a computer program, and the processor is used to execute the computer program to implement the method of any one of the embodiments of this application.

[0085] Figures 2-3 For an exemplary description, please refer to the above. Figure 1 This will not be elaborated upon here.

Claims

1. A camera-LiDAR joint calibration method based on natural scene features, characterized in that, include: Step 1: Generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. Step 2: Instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively to determine the target region at the image end and the target cluster in the point cloud, and perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map; Step 3: Based on the boundary metric mask map, locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image, and calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. Step 4: Based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group and perform spatial calculations on them to generate scene spatial registration joint parameters.

2. The camera-LiDAR joint calibration method based on natural scene features according to claim 1, characterized in that, The step of performing mapping smoothing processing based on the inverse distance gradient on the target region of the image to construct a boundary metric mask map includes: extracting the effective target pixel set and the remaining background pixel set within the target region of the image; assigning binarized difference labels to the effective target pixel set and the remaining background pixel set to generate an initial binary segmentation mask layer; calculating the Euclidean distance offset from each pixel in the initial binary segmentation mask layer to the target segmentation boundary; and performing exponential distance decay weighting processing on the internal weights of the initial binary segmentation mask layer according to the Euclidean distance offset to construct the boundary metric mask map.

3. The camera-LiDAR joint calibration method based on natural scene features according to claim 1, characterized in that, The steps of calculating the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group include: determining all three-dimensional discrete point coordinate vectors belonging to the same target cluster instance in the three-dimensional physical coordinate group, and performing spatial vector accumulation and mean-based division operations on them to obtain the three-dimensional geometric centroid coordinates; determining all two-dimensional pixel coordinate vectors belonging to the same target cluster instance in the two-dimensional coordinate group, and performing planar vector accumulation and mean-based division operations on them to obtain the two-dimensional semantic centroid coordinates.

4. The camera-LiDAR joint calibration method based on natural scene features according to claim 1, characterized in that, The step of establishing the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and performing spatial calculations on them to generate scene space registration joint parameters, includes: constructing a centroid mapping relationship between space and plane based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates; performing perspective point-related projection algebra solution on the centroid mapping relationship to obtain an initial extrinsic parameter transformation matrix; and inputting the initial extrinsic parameter transformation matrix as the search starting point into a preset particle swarm optimization space for iterative optimization processing to generate the scene space registration joint parameters.

5. The camera-LiDAR joint calibration method based on natural scene features according to claim 4, characterized in that, The step of inputting the initial extrinsic transformation matrix as the search starting point into a preset particle swarm optimization algorithm space for iterative optimization to generate the scene space registration joint parameters includes: applying spatial perturbation variables to the initial extrinsic transformation matrix to initialize the generation of a biomimetic search particle swarm; mapping the three-dimensional physical coordinate group to the plane where the boundary metric mask map is located through the state matrix of each particle in the biomimetic search particle swarm to determine the feature projection landing point set; constructing the target panoramic information matching degree based on the consistency response values ​​of the feature projection landing point set and the boundary metric mask map; and performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene space registration joint parameters.

6. The camera-LiDAR joint calibration method based on natural scene features according to claim 5, characterized in that, The step of performing iterative optimization on the biomimetic search particle swarm based on the target panoramic information matching degree to generate the scene spatial registration joint parameters includes: evaluating the fitness evaluation value of each particle in the biomimetic search particle swarm through the target panoramic information matching degree; extracting the individual optimal exploration position of each particle and the global optimal exploration position of the entire swarm based on the fitness evaluation value; performing linear decay calculation according to the optimization evolution process to generate dynamic inertia adjustment weights; using the individual optimal exploration position, the global optimal exploration position, and the dynamic inertia adjustment weights to update the current movement speed and distribution position of each particle to output the evolved state particle swarm, and determining the scene spatial registration joint parameters based on the evolved state particle swarm.

7. The camera-LiDAR joint calibration method based on natural scene features according to claim 6, characterized in that, The step of determining the scene space registration joint parameters based on the evolved state particle swarm includes: performing fitness evaluation processing on the evolved state particle swarm again to obtain a verification evaluation result; determining whether the verification evaluation result triggers a preset optimization convergence condition; in response to not triggering the preset optimization convergence condition, adjusting the optimization iteration algebra and returning to the linear decay calculation performed according to the optimization evolution process to generate dynamic inertia adjustment weights; in response to triggering the preset optimization convergence condition, converting the global optimal exploration position corresponding to the verification evaluation result into a calibration extrinsic parameter matrix and using it as the scene space registration joint parameters.

8. The camera-LiDAR joint calibration method based on natural scene features according to claim 1, characterized in that, The method for generating a dense point cloud based on a discrete point cloud sequence collected by a robot in a natural scene includes: separating the current observation frame point cloud and the historical observation frame point cloud from the discrete point cloud sequence; performing voxel downsampling on the current observation frame point cloud and the historical observation frame point cloud to obtain a spatially sparse point cloud; performing local geometric difference filtering on the spatially sparse point cloud to extract a spatially distributed sparse feature point set; performing local geometric morphology encoding on the spatially distributed sparse feature point set to generate a structural feature descriptor; and performing pose transformation operation on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the dense point cloud of the scene.

9. The camera-LiDAR joint calibration method based on natural scene features according to claim 8, characterized in that, The step of performing attitude transformation operations on the current observation frame point cloud and the historical observation frame point cloud based on the structural feature descriptor to generate the scene dense point cloud includes: selecting initial homonymous feature point pairs from the spatially distributed sparse feature point sets corresponding to the current observation frame point cloud and the historical observation frame point cloud based on the nearest neighbor metric relationship of the structural feature descriptor; generating an initial coarse registration transformation matrix based on the initial homonymous feature point pairs; mapping the historical observation frame point cloud to the reference coordinate system of the current observation frame point cloud using the initial coarse registration transformation matrix to establish a nearest neighbor point pair distance error space; and performing nearest point iterative minimization optimization on the nearest neighbor point pair distance error space to generate the scene dense point cloud.

10. A camera-LiDAR joint calibration device based on natural scene features, characterized in that, include: The first program unit is used to generate a dense point cloud of the scene based on the discrete point cloud sequence collected by the robot in the natural scene, and encapsulate it with the synchronous visual image collected by the robot in the natural scene to generate multimodal collaborative data. The second program unit is used to instantiate the multimodal collaborative data into visual pixels and laser point clouds respectively, so as to determine the target region at the image end and the target cluster in the point cloud, and to perform mapping smoothing processing on the target region at the image end to construct a boundary metric mask map. The third program unit is used to locate the three-dimensional physical coordinate group of the target cluster in the point cloud and the two-dimensional coordinate group of the target region in the image based on the boundary metric mask map, and to calculate the three-dimensional geometric centroid coordinates of the three-dimensional physical coordinate group and the two-dimensional semantic centroid coordinates of the two-dimensional coordinate group. The fourth program unit is used to establish the dual relationship between the three-dimensional physical coordinate group and the two-dimensional coordinate group based on the three-dimensional geometric centroid coordinates and the two-dimensional semantic centroid coordinates, and to perform spatial calculation on them to generate scene spatial registration joint parameters.

Images