Camera external parameter calibration method and device based on image and point cloud matching

Abstract

The invention provides a camera external parameter calibration method and device based on image and point cloud matching, and the technical scheme of the invention is that a point cloud rendering view corresponding to an original point cloud is rotated, so that the point cloud rendering view and a camera image are superposed visually, and initial space association is established; and directly learning and solving accurate camera external parameters from the local relevance between the original point cloud intensity information and the image RGB information in an end-to-end manner by using a deep learning model. According to the scheme, the strict camera-radar orientation consistency requirement in a traditional method is not needed through rough matching, manual participation in the whole process of traditional calibration is replaced, transition from visual alignment to geometric alignment is achieved, and the calibration efficiency and scene adaptability are remarkably improved. Meanwhile, the scheme of the invention initiates a process of visual field cone cutting-perspective projection rasterization-intensity-RGB end-to-end matching, and compared with a traditional feature point method, the calculation complexity is greatly reduced, and the memory occupation is reduced by 80%.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of camera parameter calibration, and in particular to a camera extrinsic parameter calibration method and device based on image and point cloud matching. BACKGROUND

[0002] Camera extrinsic parameter calibration is to determine the position and orientation of the camera in the three-dimensional space (i.e. the rotation and translation matrix), and to establish the mapping relationship between the world coordinate system and the camera coordinate system. The current extrinsic parameter calibration has the following defects:

[0003] 1) There are requirements for the scene and the radar orientation, specifically, the camera image and the radar point cloud must be strictly consistent in orientation.

[0004] 2) Strong artificial dependence, specifically, the calibration board based method needs to be strictly deployed and accurately detect the physical calibration board corner points, and cannot be used in open or no calibration object scenes (such as mine area, power transmission corridor, open road). The natural feature matching based method (such as lane line, pole tower) needs to manually select corresponding feature points on the image and point cloud or manually adjust the virtual camera pose (such as dragging the rotation point cloud in the Potree visualization tool) until the projection coincides, which is tedious, subjective and cannot realize automatic calibration.

[0005] 3) Low computational efficiency, specifically, feature extraction is time-consuming, and the current scheme usually involves complex preprocessing steps (such as point cloud road segmentation, filtering, image histogram equalization, edge detection) to obtain matchable features. The optimization process is lengthy, and the current scheme relies on iterative optimization algorithms (such as EPnP, PnP variants, particle swarm optimization, genetic algorithm) to solve the extrinsic parameters, especially in high-dimensional space or poor initial value, the convergence speed is slow, and the single calibration time-consuming can reach several minutes.

[0006] 4) High dependence on scene feature objects and point cloud density, specifically, sparse point cloud matching is difficult, low-beam laser radar (such as mine 16-line radar) produces sparse point cloud, it is difficult to reliably extract enough geometric features (such as edges, corners), and often needs to accumulate multiple frames to increase the density, further increasing the computational burden and delay. The scene universality is limited, specifically, the current scheme highly depends on the existence of specific, easily identifiable and stable feature objects (such as clear lane lines, regular poles, traffic signs) in the scene. In the environment with few features or dynamic changes (such as construction area, vegetation covered area), the performance is poor.

[0007] 5) Cross-modal matching error and robustness problem, specifically, modal gap, image texture information (RGB) and point cloud geometry information (3D coordinates / intensity) have essential differences, which leads to direct matching difficulty (such as two-dimensional image edge and three-dimensional point cloud structure do not completely correspond). Compensation algorithm is complex, in order to overcome the modal difference, it is often necessary to introduce additional algorithm (such as generating depth map / intensity map of point cloud projection, using inverse distance transformation to generate mask, designing complex similarity measure), which increases the system complexity and potential error source. Dynamic scene adaptability is poor, the method based on static calibration board cannot update the external parameter online; the generalization ability of the model based on deep learning is limited by the distribution of training data, and the robustness to unobserved scenes or severe environmental changes is insufficient. SUMMARY

[0008] The present disclosure at least provides a camera extrinsic parameter calibration method and device based on image and point cloud matching, to solve the problem that in the current camera extrinsic parameter calibration method, the automatic scheme has poor robustness and low efficiency due to high feature matching complexity; the scheme with human participation is tedious to operate, and both of them are strongly dependent on specific calibration targets (such as chessboard and calibration board), which restricts their applicability in complex scenes.

[0009] According to an aspect of the present disclosure, a camera extrinsic parameter calibration method based on image and point cloud matching is provided, comprising:

[0010] Obtaining the original point cloud of the camera extrinsic parameter calibration scene and the camera image; rotating the point cloud rendering view corresponding to the original point cloud to make the point cloud rendering view visually coincide with the camera image; determining the initial extrinsic parameter information of the camera in the visually coinciding state;

[0011] Determining the position of the virtual camera and the intersection point closest to the surface of the point cloud between the position of the virtual camera and the line-of-sight principal axis of the virtual camera in the visually coinciding state;

[0012] Based on the position of the virtual camera and the intersection point, a temporary camera coordinate system is constructed; according to the horizontal field of view angle of the camera and the position of the virtual camera, the conical field of view region of the camera is determined; the original point cloud located in the conical field of view region is selected from the rotated original point cloud, and the coordinates of the selected original point cloud are converted into the temporary camera coordinate system to obtain the target point cloud;

[0013] A virtual imaging plane is constructed in the temporary camera coordinate system, the target point cloud is projected onto the virtual imaging plane to obtain a plurality of projection points; wherein the information of each projection point includes point cloud intensity and original three-dimensional coordinates; the original three-dimensional coordinates are the coordinates of the original point cloud in the visually coinciding state;

[0014] discretize the projection points to two-dimensional grids, and respectively weight and fuse point cloud intensities of points in each two-dimensional grid to obtain an intensity of each two-dimensional grid; and generate a point cloud intensity feature map based on the intensity of each two-dimensional grid; wherein a resolution of the two-dimensional grid is the same as a resolution of the camera image;

[0015] According to the intersection, an image block is intercepted from the projection points, and an intensity feature block of the same region as the image block is intercepted from the point cloud intensity feature map;

[0016] The intensity feature block and the image block are input into a trained deep learning model, so that the deep learning model physically associates and fuses image texture and point cloud intensity, and outputs a fusion feature; the fusion feature is globally aggregated, and pose residual prediction is performed based on an aggregation result to obtain pose residual prediction information;

[0017] Based on the pose residual prediction information and initial extrinsic parameter information of the camera, an extrinsic parameter calibration result of the camera is determined.

[0018] In a possible implementation, the deep learning model includes the following steps in processing the image block:

[0019] Feature encoding is performed on the image block;

[0020] Spatial line pooling processing is performed on a result of the feature encoding;

[0021] Attention weights are generated based on a result of the spatial line pooling processing;

[0022] Feature enhancement processing is performed on the feature encoding by using the attention weights to obtain an enhanced feature tensor of the image.

[0023] In a possible implementation, the deep learning model includes the following steps in processing the intensity feature block:

[0024] Spherical coordinate conversion and periodic encoding are performed on the point cloud in the intensity feature block;

[0025] Variable convolution is used to perform offset learning and adaptive feature extraction on a result of the periodic encoding to obtain geometric enhanced features of the point cloud.

[0026] In a possible implementation, after the enhanced feature tensor of the image and the geometric enhanced features of the point cloud are obtained, the deep learning model is further used to:

[0027] Feature projection is performed on the enhanced feature tensor;

[0028] projecting the geometric enhanced feature twice using two learnable weights to obtain a first geometric feature projection and a second geometric feature projection;

[0029] determining an attention weight based on the projected enhanced feature tensor and the first geometric feature projection;

[0030] generating a gating signal based on the enhanced feature tensor and the geometric enhanced feature;

[0031] generating the fusion feature based on the gating signal, the attention weight, and the second geometric feature projection.

[0032] In a possible implementation, the constructing a temporary camera coordinate system based on the position of the virtual camera and the intersection point comprises:

[0033] taking the position of the virtual camera as an origin of the temporary camera coordinate system, taking a direction from the position of the virtual camera to the intersection point as a positive direction of an X-axis, and automatically generating a Y-axis and a Z-axis according to a right-hand rule; wherein a positive direction of the Y-axis is a direction of a plane formed by the X-axis and the Z-axis, and a positive direction of the Z-axis is a zenith direction.

[0034] In a possible implementation, the determining a conical field of view region of the camera according to the horizontal field of view angle of the camera and the position of the virtual camera comprises:

[0035] constructing the conical field of view region with the position of the virtual camera as an apex, an X-axis as a central axis, and the horizontal field of view angle as an apex angle.

[0036] In a possible implementation, the position of the virtual camera is a position coordinate of a viewpoint of the camera in a projection coordinate system in a state of visual coincidence, and the intersection point of the virtual camera is a position coordinate of an intersection point of the camera in the projection coordinate system in the state of visual coincidence.

[0037] In a possible implementation, the pose residual prediction is performed using the following formula:

[0038]

[0039] wherein, is a ReLU activation function, is pose residual prediction information, is an aggregation result, and W1, W2, and W3 are learnable weights.

[0040] In a possible implementation, the pose residual prediction information comprises prediction information corresponding to a rotation matrix and prediction information corresponding to a translation vector.

[0041] According to another aspect of the present disclosure, a camera extrinsic parameter calibration device based on image and point cloud matching is provided, comprising:

[0042] A coarse matching module is configured to obtain a raw point cloud of a camera extrinsic parameter calibration scene and a camera image; rotate a point cloud rendering view corresponding to the raw point cloud so that the point cloud rendering view and the camera image visually coincide; and determine initial extrinsic parameter information of the camera in the visually coinciding state;

[0043] A positioning module is configured to determine a position of a virtual camera and a closest intersection point between a line-of-sight principal axis of the virtual camera and a point cloud surface in the visually coinciding state;

[0044] A coordinate system construction and point cloud clipping module is configured to construct a temporary camera coordinate system based on the position of the virtual camera and the intersection point; determine a conical field of view region of the camera according to a horizontal field of view angle of the camera and the position of the virtual camera; screen the raw point cloud located in the conical field of view region from the rotated raw point cloud; and convert coordinates of the screened raw point cloud to the temporary camera coordinate system to obtain a target point cloud;

[0045] A point cloud compression module is configured to construct a virtual imaging plane in the temporary camera coordinate system, project the target point cloud onto the virtual imaging plane to obtain a plurality of projection points, wherein information of each projection point includes point cloud intensity and original three-dimensional coordinates; the original three-dimensional coordinates are coordinates of the raw point cloud in the visually coinciding state; and discretize the projection points to a two-dimensional grid and respectively perform weighted fusion on point cloud intensity of points in each two-dimensional grid to obtain intensity of each two-dimensional grid; and generate a point cloud intensity feature map based on the intensity of each two-dimensional grid; wherein a resolution of the two-dimensional grid is the same as a resolution of the camera image;

[0046] A cross-modal feature matching module is configured to intercept an image block from the projection points and an intensity feature block of the same region as the image block from the point cloud intensity feature map according to the intersection point; input the intensity feature block and the image block into a trained deep learning model to enable the deep learning model to physically associate and fuse image texture and point cloud intensity, and output a fusion feature; perform global feature aggregation on the fusion feature, and perform pose residual prediction based on an aggregation result to obtain pose residual prediction information;

[0047] An extrinsic parameter calibration module is configured to determine an extrinsic parameter calibration result of the camera based on the pose residual prediction information and the initial extrinsic parameter information of the camera.

[0048] The camera extrinsic parameter calibration method and device based on image and point cloud matching of the present disclosure has no requirement for the orientation of the image and the point cloud, and can still achieve accurate calibration of the extrinsic parameter through the coarse matching stage and subsequent steps even in the case of inconsistent initial poses. The scheme of the present disclosure greatly reduces the dependence on manual operation, and only needs to rotate the point cloud to make the point cloud and the image visually coincide, without the need for accurate selection of feature points or adjustment of the virtual camera pose. The present disclosure only provides a rough initial visual angle, greatly simplifying the manual operation. The core matching process in the present disclosure is automatically completed, and the coordinates of the intersection of the position of the virtual camera in the coarse matching state and the line of sight and the point cloud are automatically captured, combined with the corresponding image, and the fine matching is automatically completed through the algorithm, completely eliminating the dependence on manual selection or fine adjustment in the traditional method. In addition, the technical scheme of the present disclosure can significantly improve the calculation efficiency and meet the online / near real-time demand. The scheme of the present disclosure bypasses complex feature engineering, and directly learns the cross-modal correspondence between the local structure of the point cloud (including geometric and intensity information) and the image block (RGB texture) by using an end-to-end deep learning model, thereby saving time-consuming feature extraction (such as point cloud segmentation and edge detection) and complex iterative optimization algorithms. The scheme of the present disclosure reduces the dependence on specific features and point cloud density, improves the scene universality, and uses the original point cloud and context information. The deep learning model learns the correlation pattern between the point cloud intensity and the image RGB pixel, rather than relying on specific artificial defined features (such as lane lines and towers), which makes it effective in scenes lacking clear structured features (such as mining areas, natural landscapes, and power transmission corridors). The deep learning model can use the local intensity information of the point cloud and the image context for matching, which has better adaptability to sparse point clouds generated by low-beam radars compared to methods relying on clear geometric features. The scheme of the present disclosure solves the cross-modal matching through an end-to-end method, improves the accuracy and robustness, and the deep learning model can implicitly learn how to align the image pixel space and the point cloud three-dimensional space, effectively bridging the modal gap between RGB and 3D point cloud (geometry + intensity), and avoiding the need to design complex manual compensation algorithms. The deep learning model can learn more generalized matching patterns through large-scale data training, and has stronger robustness to changes in light and angle, partial occlusion, and dynamic environmental interference (within the coverage of the training data).

[0049] In summary, the scheme of the present disclosure is a highly efficient, automated, and robust camera extrinsic parameter calibration method.

[0050] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0052] Figure 1 is one of the flowcharts of the camera extrinsic parameter calibration method based on image and point cloud matching in the embodiments of the present disclosure;

[0053] Figure 2 is the second flowchart of the camera extrinsic parameter calibration method based on image and point cloud matching in the embodiments of the present disclosure;

[0054] Figure 3A is a schematic diagram of a camera image in the embodiments of the present disclosure;

[0055] Figure 3B is an image after coarse matching in the embodiments of the present disclosure;

[0056] Figure 4A is a processing flowchart of an image branch in the embodiments of the present disclosure;

[0057] Figure 4B is a processing flowchart of a cross-modal feature fusion branch in the embodiments of the present disclosure;

[0058] Figure 5 is a structural schematic diagram of a deep learning model in the embodiments of the present disclosure;

[0059] Figure 6 is a structural schematic diagram of a camera extrinsic parameter calibration device based on image and point cloud matching in the embodiments of the present disclosure. DETAILED DESCRIPTION

[0060] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to assist in understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0061] The present disclosure is for the current camera extrinsic parameter calibration method, the automatic scheme is caused by high complexity of feature matching, which leads to poor robustness and low efficiency; the operation of the scheme with human participation is tedious, and both are strongly dependent on specific calibration targets (such as checkerboard, calibration board), which restricts their applicability in complex scenes, and a camera extrinsic parameter calibration method based on image and point cloud matching is proposed. The scheme of the present disclosure uses single coarse matching, without the strict camera-radar orientation consistency requirement in traditional methods, replacing the traditional full-process manual participation (such as feature selection, calibration board placement), realizing the transition from "visual alignment" to "geometric alignment", and significantly improving the calibration efficiency and scene adaptability. The process of "view cone clipping → perspective projection gridding → intensity-RGB end-to-end matching" is created, and the computational complexity is reduced from Down to ( The memory usage is reduced by 80% (only point clouds within the field of view are processed), and the solution disclosed in this paper utilizes point cloud intensity information as a physical bridge connecting 3D geometry and 2D textures, overcoming the failure of pure geometry matching in scenes with missing textures (such as white walls and skies).

[0062] The technical solution of this disclosure will be described below through specific embodiments.

[0063] like Figure 1 The diagram shown is a flowchart of the camera extrinsic parameter calibration method based on image and point cloud matching in this embodiment. The execution subject of this embodiment is a computing device or component with data processing capabilities. Specifically, the method of this embodiment may include the following steps:

[0064] S110. Obtain the original point cloud of the camera extrinsic calibration scene and the camera image; rotate the point cloud rendering view corresponding to the original point cloud so that the point cloud rendering view and the camera image visually overlap; determine the initial extrinsic parameter information of the camera in the visually overlapping state.

[0065] In the Potree point cloud visualization tool, by interactively rotating the original point cloud, the rendered point cloud view is made to roughly overlap with the corresponding camera image in terms of object outlines or scene structure. This state is called visual overlap between the rendered point cloud view and the camera image. Figure 2 As shown, this step performs the initialization of the human-machine collaboration space, thus completing the coarse matching operation. For example... Figure 3A The image shown is a camera image, and 3B is the image after coarse matching and the image under the same condition.

[0066] S120. Determine the point of closest intersection between the position of the virtual camera and the distance between the main axis of the virtual camera's line of sight and the point cloud surface under the visually overlapping state.

[0067] The position of the virtual camera is the position coordinate of the camera's viewpoint in the projection coordinate system when they are visually aligned. The intersection point of the virtual cameras is the position coordinate of the intersection point of the cameras in the projection coordinate system when they are visually aligned.

[0068] like Figure 2 As shown, in coarse matching mode, the position and intersection point of the virtual camera are automatically recorded. The position of the virtual camera (C_rough): the coordinates of the Potree viewpoint (i.e., the viewpoint that the user considers to be matched) in 3D space. The line-of-sight intersection point (P_intersect): the 3D coordinates of the point where the virtual camera intersects the point cloud surface along its main line-of-sight axis from its position. This step completes the capture of key geometric reference data.

[0069] In the specific implementation process, the position and intersection point of the virtual camera can be converted to the projected coordinate system using the following steps.

[0070] ① Transformation from local coordinates to geocentric Cartesian coordinates

[0071] Given local point coordinates and affine transformation matrix (4×4 homogeneous transformation matrix):

[0072]

[0073] Transformed geocentric Cartesian coordinates The calculation is as follows:

[0074]

[0075] Take the first three components:

[0076]

[0077] This is a transformation between 3D coordinates, converting the 3D coordinates from the web interface (for ease of display) to the 3D coordinates of a point cloud.

[0078] ② Transformation from geocentric Cartesian coordinates to projected coordinates

[0079] Given a projection definition proj(UTM), perform an inverse projection transformation. geocentric coordinates Convert to target projected coordinates :

[0080]

[0081] in It is the forward transformation from the projected coordinate system to the geocentric coordinate system, and the inverse transformation. The general formula is:

[0082]

[0083] ③ Complete conversion formula

[0084] Combining the above two steps, click To the final projected coordinates The complete conversion is defined as follows:

[0085]

[0086] in It is the affine transformation matrix from the local coordinate system to the geocentric coordinate system. It is an inverse projection transformation from the geocentric coordinate system to the target projection coordinate system.

[0087] S130. Based on the position of the virtual camera and the intersection point, construct a temporary camera coordinate system; determine the conical field of view area of ​​the camera according to the horizontal field of view angle of the camera and the position of the virtual camera; filter the original point cloud located in the conical field of view area from the rotated original point cloud, and transform the coordinates of the filtered original point cloud to the temporary camera coordinate system to obtain the target point cloud.

[0088] The temporary camera coordinate system constructed above based on the position of the virtual camera and the intersection point may specifically include:

[0089] The location of the virtual camera is taken as the origin of the temporary camera coordinate system, and the intersection point of the virtual camera is taken as the positive direction of the X-axis. The Y-axis and Z-axis are automatically generated according to the right-hand rule. The positive direction of the Y-axis is the direction perpendicular to the plane formed by the X-axis and Z-axis, and the positive direction of the Z-axis is the zenith direction.

[0090] like Figure 2 As shown, the above steps completed the coordinate system reconstruction.

[0091] The above-mentioned method of determining the camera's conical field of view based on the camera's horizontal field of view and the position of the virtual camera can specifically include:

[0092] The conical field of view is constructed with the position of the virtual camera as the vertex, the x-axis as the central axis, and the horizontal field of view angle as the apex.

[0093] The temporary camera coordinate system described above can be represented as Frame_temp. This step transforms the original point cloud to the Frame_temp coordinate system, retaining only the point cloud data within the cone and eliminating redundant points outside the field of view. This step achieves point cloud spatial transformation and field-of-view constraint processing. By dynamically constructing a temporary camera coordinate system, data normalization is achieved, and physical optical constraints are used to filter the target point cloud, which can significantly improve computational efficiency and overcome the redundant computational defects of traditional methods in processing global point clouds.

[0094] like Figure 2 As shown, this step completes the point cloud trimming and obtains the target point cloud.

[0095] S140. Construct a virtual imaging plane in the temporary camera coordinate system, and project the target point cloud onto the virtual imaging plane to obtain multiple projection points; wherein, the information of each projection point includes the point cloud intensity and the original three-dimensional coordinates; the original three-dimensional coordinates are the coordinates of the original point cloud in the visually overlapping state.

[0096] This step implements perspective projection transformation. In the Frame_temp coordinate system, a virtual imaging plane is set along the X-axis (distance d from the origin, where d is adaptively calculated by the point cloud distribution density). The point cloud within the conical field of view is projected onto this plane, preserving the intensity value and original 3D coordinates of each projection point.

[0097] S150. Discretize the projection points into a two-dimensional grid, and perform weighted fusion of the point cloud intensity of each point in the two-dimensional grid to obtain the intensity of each two-dimensional grid; and generate a point cloud intensity feature map based on the intensity of each two-dimensional grid; wherein the resolution of the two-dimensional grid is the same as the resolution of the camera image.

[0098] This step achieves point cloud intensity map rasterization by discretizing the projected points into a two-dimensional grid according to the (u,v) coordinates. The grid resolution is consistent with the camera image resolution. Gaussian weighted fusion is performed on the point cloud intensity values ​​in each grid cell to generate a point cloud intensity feature map M_intensity with the same size as the image.

[0099] Steps 140-150 compress the 3D point cloud intensity information into a 2D structured representation, constructing a topological correspondence with image pixels. For example... Figure 2 As shown, steps 140-150 achieve point cloud compression, or feature compression, by generating a point cloud intensity feature map.

[0100] S160. Based on the intersection point, extract an image block from the projection point, and extract an intensity feature block from the point cloud intensity feature map that is in the same region as the image block.

[0101] The input to a deep learning model is a bimodal input: an image patch (image input) and an intensity feature patch (point cloud input).

[0102] Image input: Crop an image patch I_patch (RGB three channels) centered on the projection point of P_intersect; Point cloud input: Extract the intensity feature block P_patch (single-channel intensity + position encoding) of the corresponding region in M_intensity; Concatenate (I_patch, P_patch) to form a two-channel input tensor.

[0103] S170. Input the intensity feature block and image block into the trained deep learning model so that the deep learning model can physically associate and fuse the image texture with the point cloud intensity and output the fused features.

[0104] Inference is achieved using a deep learning model, and fused features are output.

[0105] Steps 160-170 use an end-to-end deep learning model to directly learn the intensity-RGB modal correlation, replacing the traditional multi-stage optimization algorithm, such as... Figure 2 As shown, cross-modal feature matching was achieved.

[0106] like Figure 4A As shown, the deep learning model includes an image branch that extracts RGB texture features, extracting robust texture features strongly correlated with spatial location to overcome interference from illumination changes and viewpoint shifts. Specifically, the image branch performs the following steps during the processing of the image patch:

[0107] 1) For image patches (i.e. Figure 4A Feature encoding is performed on the original image of the Central Earth.

[0108] Input definition: image patch

[0109] in, (Height): 224 pixels; (Width): 224 pixels

[0110] The input specification based on the classic CNN architecture meets the following conditions. (Meets the alignment requirements for multiple downsampling steps), where the number of downsampling steps is... (ResNet-18 includes 4 stages with a total of 5 downsampling steps of size 2), ensuring that the final feature map size is...

[0111]

[0112] Feature encoding:

[0113] Through multi-layer convolution kernels To learn local texture patterns (edges / corners / regions), an improved ResNet-18 network is used. Layer convolution operation:

[0114]

[0115] in, This is a convolution operation; It is the ReLU activation function; for Convolution kernel weights s; This represents the output feature of the (j-1)th layer, which is the input of the current layer; it can be understood as "the image feature representation after processing by the previous layer", which contains more abstract texture information. This represents the bias term of the j-th layer, which helps adjust the convolution result and allows the model to fit the data more flexibly.

[0116] 2) Spatial loop pooling is performed on the feature encoding results.

[0117] The formula for spatial casting pooling is as follows:

[0118]

[0119]

[0120] h and w represent the height and width of the feature map.

[0121] 3) Attention weights are generated based on the results of spatial line pooling.

[0122] Attention weight generation: ,in

[0123] 2) and 3) implement a coordinate attention mechanism to enhance key region features related to the intersection of the line of sight and suppress background noise.

[0124] 4) Use attention weights to perform feature enhancement processing on the feature encoding to obtain the enhanced feature tensor of the image.

[0125] Feature enhancement:

[0126] Output augmented feature tensor:

[0127] The 'c' above represents a channel.

[0128] The deep learning model includes a point cloud branch for intensity-location feature extraction, fusing intensity reflection characteristics with geometric structure to address the problem of missing features in sparse point clouds. The point cloud branch may include the following steps in processing the intensity feature blocks:

[0129] 1) Perform spherical coordinate transformation and periodic encoding on the point cloud in the intensity feature block.

[0130] Input definition: Intensity feature block (Channel 1: Intensity; Channels 2-4: Location Encoding)

[0131] Spherical coordinate transformation in spherical coordinate system position encoding:

[0132]

[0133] Periodic coding:

[0134]

[0135] 2) Variable convolution is used to perform offset learning and adaptive feature extraction on the results of the periodic encoding to obtain the geometric enhancement features of the point cloud.

[0136] Offset learning for n = 1,...,9 (3x3 cores):

[0137]

[0138] Adaptive feature extraction:

[0139]

[0140] Where R is the receptive field, W offset Represents the learnable weights (offset prediction convolutional kernels), Φ density (p0): Local features of the point cloud after adaptive extraction; P0: Sampling position of the center of the convolution kernel; W(P n P(.): Samples unknown weights; P(.): Input feature map; P n The nth sampling point has a fixed offset relative to P0; ΔP n : The nth sampling point, dynamically offset relative to P0.

[0141] Output: Geometric Enhancement Features

[0142] like Figure 4B As shown, the deep learning model includes a cross-modal feature fusion branch, used to establish a physical correlation between RGB texture and point cloud intensity-geometry, bridging the modal gap. Specifically, after obtaining the enhanced feature tensor of the image and the geometric enhanced features of the point cloud, the cross-modal feature fusion branch is further used for:

[0143] 1) Perform feature projection on the enhanced feature tensor.

[0144] The aforementioned enhanced feature tensor is Figure 4B The RGB features in [the text].

[0145] The specific projection formula is as follows:

[0146]

[0147] 2) The geometric enhancement features are projected twice using two learnable weights to obtain the first geometric feature projection and the second geometric feature projection.

[0148] Geometric enhancement features are Figure 4B The intensity characteristics in.

[0149] The specific projection formula is as follows:

[0150]

[0151]

[0152] in These are learnable weights.

[0153] 3) Determine the attention weights based on the projected enhanced feature tensor and the first geometric feature projection.

[0154] The attention weight calculation in this step is used to quantify the correlation strength between image pixels and point cloud regions, and the formula is as follows:

[0155]

[0156] Energy(i,j): Cross-modal feature, essentially a similarity measure between image features and point cloud features. It measures the degree of correlation between the i-th image feature and the j-th point cloud feature; a larger value indicates a higher degree of matching. i and j are the feature indices of different modalities; d k : Feature projection dimension; Q i ·K j T : Quantify the similarity of features between two modalities.

[0157] 4) Generate a gated signal based on the enhanced feature tensor and the geometric enhancement features.

[0158] Gating signal generation:

[0159]

[0160] W g This represents the weight of the gating signal, which is a learnable weight.

[0161] 5) Generate the fused features based on the gating signal, attention weights, and the second geometric feature projection.

[0162] Weighted output:

[0163]

[0164] Output fusion features:

[0165] The fusion mechanism is to gating cross-modal attention, dynamically balance the contributions of both modes, and resist interference.

[0166] S180. Perform global feature aggregation on the fused features, and perform pose residual prediction based on the aggregation result to obtain pose residual prediction information; determine the camera's extrinsic calibration result based on the pose residual prediction information and the camera's initial extrinsic information.

[0167] Global feature aggregation is performed using the following formula:

[0168]

[0169] The pose residual is predicted using the following formula:

[0170]

[0171] In the formula, It is the ReLU activation function. For pose residual prediction information, For the aggregation result, W1, W2, and W3 are learnable weights.

[0172] The pose residual prediction information includes prediction information corresponding to the rotation matrix and prediction information corresponding to the translation vector.

[0173] Rotation matrix update:

[0174] in For Lie algebra, For exponential mapping, R rough This represents the coarse matching rotation matrix, which is included in the initial extrinsic information of the camera in the visual coincidence state.

[0175] Translation vector update:

[0176] t rough This represents the initial translation amount for coarse matching, which is included in the initial extrinsic information of the camera in the visual coincidence state.

[0177] The loss function is designed as follows:

[0178]

[0179] in , For Lie algebraic-logarithmic mappings, R gt Let tgt represent the true rotation matrix and tgt represent the true translation vector.

[0181] Step S180 directly outputs 6DoF pose residual prediction information from the fused features, avoiding iterative optimization. The decoder then outputs the pose residual prediction information. The residual values ​​are mapped to extrinsic parameters, and the pose obtained from coarse matching (i.e., the initial extrinsic parameter information of the camera) are superimposed to output the final camera extrinsic parameters.

[0182] In some embodiments, deep learning models such as Figure 5 As shown, it includes:

[0183] Images: The input can be images and / or point cloud data, which are the raw objects processed by the model and provide the basis for subsequent operations such as feature extraction.

[0184] Local features: Input can be layout features of images and / or point clouds, location information, intensity information.

[0185] Self-attention: It processes local features, explores the dependencies between features within the same set of data, highlights important features within the group, enhances feature representation, and allows the model to focus on its own valuable information.

[0186] Cross attention: Establishes connections between features, enables information exchange, helps the model build relationships between features, and prepares for subsequent matching and other operations.

[0187] Confidence (C): Used to measure the reliability of features or related results during processing. It can help determine whether a feature is reliable and affect subsequent operations such as pruning, for example, selecting more reliable features for further processing.

[0188] Pruning: Based on confidence level and other information, remove unreliable or less useful feature points (marked with "×" in the diagram), simplify calculations, and retain more valuable features to continue to be passed to subsequent layers (Layer #N, etc.).

[0189] Layer #N: The subsequent layer in the multi-layer structure of the model. The features that have been processed and pruned by the previous Layer #1 are processed here again to continuously optimize the feature representation until the exit condition is met.

[0190] Matching (similarity, matchability, assignment related):

[0191] Similarity: Calculates the degree of similarity between two sets of features, constructs a similarity matrix, and reflects the probability of different image features matching each other.

[0192] Matchability: A helpful measure of whether a feature is suitable for matching. Combined with similarity, it is used to calculate the assignment (assignment, matching result), which determines the corresponding matching relationship between features.

[0193] Exit (Exit Judgment): Determines whether to stop the current layer (e.g., Layer #1) or the entire feature processing flow. If the condition is met, it enters the matching stage; otherwise, it continues to process features in the current layer or subsequent layers (e.g., the flow before continuing from Layer #N).

[0194] Based on the same inventive concept, this disclosure provides a camera extrinsic parameter calibration device based on image and point cloud matching. The steps performed by the components of this device are the same as or similar to those described above, therefore, similar details will not be repeated. Figure 6 As shown, the camera extrinsic parameter calibration device based on image and point cloud matching in this embodiment includes:

[0195] The coarse matching module 610 is used to acquire the original point cloud of the camera extrinsic calibration scene and the camera image; rotate the point cloud rendering view corresponding to the original point cloud so that the point cloud rendering view and the camera image visually overlap; and determine the initial extrinsic parameter information of the camera in the visually overlapping state.

[0196] The positioning module 620 is used to determine the position of the virtual camera and the closest intersection point between the virtual camera's main line of sight and the point cloud surface in the visually overlapping state.

[0197] The coordinate system construction and point cloud clipping module 630 is used to construct a temporary camera coordinate system based on the position of the virtual camera and the intersection point; determine the conical field of view of the camera according to the horizontal field of view angle of the camera and the position of the virtual camera; filter the original point cloud located in the conical field of view from the rotated original point cloud, and transform the coordinates of the filtered original point cloud to the temporary camera coordinate system to obtain the target point cloud.

[0198] The point cloud compression module 640 is used to construct a virtual imaging plane in the temporary camera coordinate system, project the target point cloud onto the virtual imaging plane to obtain multiple projection points; wherein, the information of each projection point includes point cloud intensity and original three-dimensional coordinates; the original three-dimensional coordinates are the coordinates of the original point cloud in the visually overlapping state; and, to discretize the projection points into a two-dimensional grid, and to perform weighted fusion on the point cloud intensity of the points in each two-dimensional grid to obtain the intensity of each two-dimensional grid; and to generate a point cloud intensity feature map based on the intensity of each two-dimensional grid; wherein, the resolution of the two-dimensional grid is the same as the resolution of the camera image.

[0199] The cross-modal feature matching module 650 is used to extract an image patch from the projection point based on the intersection point, and extract an intensity feature patch from the point cloud intensity feature map that is the same region as the image patch; input the intensity feature patch and the image patch into a trained deep learning model, so that the deep learning model physically associates and fuses the image texture and the point cloud intensity, and outputs the fused features; perform global feature aggregation on the fused features, and perform pose residual prediction based on the aggregation result to obtain pose residual prediction information.

[0200] The extrinsic parameter calibration module 660 is used to determine the extrinsic parameter calibration result of the camera based on the pose residual prediction information and the initial extrinsic parameter information of the camera.

[0201] The various embodiments of the techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0202] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0203] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0204] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0205] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0206] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0207] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0208] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A camera extrinsic parameter calibration method based on image and point cloud matching, characterized in that, include: Acquire the original point cloud and camera images of the scene for camera extrinsic parameter calibration; Rotate the point cloud rendering view corresponding to the original point cloud so that the point cloud rendering view visually overlaps with the camera image; Determine the initial extrinsic parameters of the camera under visual overlap conditions; Determine the point of closest intersection between the position of the virtual camera and the distance between the virtual camera's main line of sight and the point cloud surface under the visually overlapping state; A temporary camera coordinate system is constructed based on the position of the virtual camera and the intersection point; the conical field of view of the camera is determined according to the horizontal field of view angle of the camera and the position of the virtual camera. The original point cloud located in the conical field of view is selected from the rotated original point cloud, and the coordinates of the selected original point cloud are transformed into the temporary camera coordinate system to obtain the target point cloud; A virtual imaging plane is constructed in the temporary camera coordinate system, and the target point cloud is projected onto the virtual imaging plane to obtain multiple projection points; wherein, the information of each projection point includes the point cloud intensity and the original three-dimensional coordinates; the original three-dimensional coordinates are the coordinates of the original point cloud in the visually overlapping state; The projected points are discretized into a two-dimensional grid, and the point cloud intensity of each point in the two-dimensional grid is weighted and fused to obtain the intensity of each two-dimensional grid; and a point cloud intensity feature map is generated based on the intensity of each two-dimensional grid; wherein the resolution of the two-dimensional grid is the same as the resolution of the camera image; Based on the intersection point, an image block is extracted from the projection point, and an intensity feature block with the same region as the image block is extracted from the point cloud intensity feature map; Intensity feature blocks and image blocks are input into a trained deep learning model, enabling the deep learning model to physically associate and fuse image texture with point cloud intensity, and output fused features; global feature aggregation is performed on the fused features, and pose residual prediction is performed based on the aggregation results to obtain pose residual prediction information; Based on the pose residual prediction information and the initial extrinsic parameter information of the camera, the extrinsic parameter calibration result of the camera is determined.

2. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 1, characterized in that, The deep learning model processes the image patch using the following steps: Perform feature encoding on image patches; The result of feature encoding is subjected to spatial loop pooling. Attention weights are generated based on the results of spatial wire pooling. By using attention weights to enhance the feature encoding, we obtain the enhanced feature tensor of the image.

3. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 2, characterized in that, The deep learning model includes the following steps in processing the intensity feature blocks: The point cloud in the intensity feature block is subjected to spherical coordinate transformation and periodic encoding; By using variable convolution to perform offset learning and adaptive feature extraction on the results of the periodic encoding, geometric enhancement features of the point cloud are obtained.

4. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 3, characterized in that, After obtaining the augmented feature tensor of the image and the geometric augmented features of the point cloud, the deep learning model is further used for: Perform feature projection on the enhanced feature tensor; The geometric enhancement features are projected twice using two learnable weights to obtain the first geometric feature projection and the second geometric feature projection. The attention weights are determined based on the projected enhanced feature tensor and the first geometric feature projection. A gated signal is generated based on the enhanced feature tensor and the geometric enhancement features; The fused features are generated based on the gating signal, attention weights, and the second geometric feature projection.

5. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 1, characterized in that, The construction of a temporary camera coordinate system based on the position of the virtual camera and the intersection point includes: The location of the virtual camera is taken as the origin of the temporary camera coordinate system, and the intersection point of the virtual camera is taken as the positive direction of the X-axis. The Y-axis and Z-axis are automatically generated according to the right-hand rule. The positive direction of the Y-axis is the direction perpendicular to the plane formed by the X-axis and Z-axis, and the positive direction of the Z-axis is the zenith direction.

6. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 5, characterized in that, The step of determining the conical field of view of the camera based on the horizontal field of view angle of the camera and the position of the virtual camera includes: The conical field of view is constructed with the position of the virtual camera as the vertex, the x-axis as the central axis, and the horizontal field of view angle as the apex.

7. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 1, characterized in that, The position of the virtual camera is the position coordinate of the camera's viewpoint in the projection coordinate system when they are visually overlapping; the intersection point of the virtual cameras is the position coordinate of the intersection point of the cameras in the projection coordinate system when they are visually overlapping.

8. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 1, characterized in that, The pose residual is predicted using the following formula: In the formula, It is the ReLU activation function. For pose residual prediction information, For the aggregation result, W1, W2, and W3 are learnable weights.

9. The camera extrinsic parameter calibration method based on image and point cloud matching according to claim 1, characterized in that, The pose residual prediction information includes prediction information corresponding to the rotation matrix and prediction information corresponding to the translation vector.

10. A camera extrinsic parameter calibration device based on image and point cloud matching, characterized in that, include: The coarse matching module is used to acquire the original point cloud and camera image of the scene for camera extrinsic calibration. Rotate the point cloud rendering view corresponding to the original point cloud so that the point cloud rendering view visually overlaps with the camera image; Determine the initial extrinsic parameters of the camera under visual overlap conditions; The positioning module is used to determine the position of the virtual camera and the closest intersection point between the virtual camera's main line of sight and the point cloud surface in the visually overlapping state. The coordinate system construction and point cloud clipping module is used to construct a temporary camera coordinate system based on the position of the virtual camera and the intersection point; and to determine the conical field of view of the camera based on the horizontal field of view angle of the camera and the position of the virtual camera. The original point cloud located in the conical field of view is selected from the rotated original point cloud, and the coordinates of the selected original point cloud are transformed into the temporary camera coordinate system to obtain the target point cloud; A point cloud compression module is used to construct a virtual imaging plane in the temporary camera coordinate system, project the target point cloud onto the virtual imaging plane to obtain multiple projection points; wherein, the information of each projection point includes point cloud intensity and original three-dimensional coordinates; the original three-dimensional coordinates are the coordinates of the original point cloud in the visually overlapping state; and, the module discretizes the projection points into a two-dimensional grid, and performs weighted fusion on the point cloud intensity of each point in the two-dimensional grid to obtain the intensity of each two-dimensional grid; and, the module generates a point cloud intensity feature map based on the intensity of each two-dimensional grid; wherein, the resolution of the two-dimensional grid is the same as the resolution of the camera image; A cross-modal feature matching module is used to extract image patches from the projection points based on the intersection points, and to extract intensity feature patches from the point cloud intensity feature map that are in the same region as the image patches; input the intensity feature patches and image patches into a trained deep learning model, so that the deep learning model can physically associate and fuse image texture with point cloud intensity, and output fused features; perform global feature aggregation on the fused features, and perform pose residual prediction based on the aggregation results to obtain pose residual prediction information; The extrinsic parameter calibration module is used to determine the extrinsic parameter calibration result of the camera based on the pose residual prediction information and the initial extrinsic parameter information of the camera.

Images