An end-to-end binocular stereo vision system calibration method

Abstract

The application relates to an end-to-end binocular stereo vision system calibration method, which comprises the following steps: pre-processing a checkerboard target image collected by a binocular camera to obtain an enhanced image; extracting an image deep feature map of the enhanced image through a shared feature extraction network; extracting geometric correlation information of the enhanced image, and constructing a corner point topology graph; encoding the corner point topology graph to obtain a global geometric feature vector; fusing the image deep feature map and the geometric feature vector to generate a fusion feature vector; and inputting the fusion feature vector into a parameter regression network to predict calibration parameters of the binocular camera. The application has the beneficial effect that the entire calibration process is automatically completed through a deep learning network, camera intrinsic parameters, extrinsic parameters and distortion parameters are directly predicted from an input image pair, corner points do not need to be manually extracted, matching pairs do not need to be constructed, and tedious step-by-step optimization does not need to be performed, so that the labor cost is greatly reduced, and the calibration efficiency is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of stereo vision calibration, and particularly relates to an end-to-end binocular stereo vision system calibration method. BACKGROUND

[0002] With the rapid development of computer vision and artificial intelligence, binocular stereo vision technology is widely used in automatic driving, robot navigation, three-dimensional reconstruction, industrial detection and other fields. This technology obtains left and right view images of the same scene through two cameras, and realizes depth perception in three-dimensional space by using the disparity information between the images. In order to realize high-precision three-dimensional reconstruction, the binocular camera system must be accurately calibrated, including internal parameters (focal length, principal point coordinates, distortion parameters), external parameters (rotation and translation relationship) and the like. This process is called binocular stereo vision system calibration, which directly determines the accuracy and stability of the system to obtain three-dimensional information.

[0003] At present, the mainstream binocular calibration method is mainly based on the traditional geometric model derivation, such as Zhang's calibration method (based on checkerboard images) and structured light + matching feature method. The typical process of this kind of method includes: manually collecting calibration board images; extracting image corner points; estimating camera internal parameters by using multiple images; calculating the relative pose (external parameters) between cameras; and optimizing the re-projection error by using least squares. Although these methods have certain effect in practical application, the traditional method relies on manual acquisition of calibration board images and auxiliary corner point detection. And it is sensitive to image quality, image blur, occlusion or angle deviation may cause corner point extraction failure, affecting the calibration accuracy. The robustness is insufficient, and the traditional method is easy to fail in low-texture or special scene.

[0004] In order to solve the above problems, in recent years, researchers have begun to try to introduce deep learning technology into binocular vision system calibration, and have proposed an end-to-end calibration method. For example, an end-to-end method is used to complete the calibration of multi-sensor fusion of laser radar and camera. This kind of method constructs a learnable neural network model, and directly regresses all the calibration parameters of the binocular camera from the input image pair, but this kind of method requires high accuracy of prior model for the learning of input image feature points. The existing traditional calibration technology has obvious geometric features but complex manual corner point extraction and poor robustness, and the end-to-end deep learning technology has low image feature and high accuracy requirement for prior model, but high efficiency and strong robustness. SUMMARY

[0005] The purpose of the application is to overcome the deficiencies in the prior art, and provide an end-to-end binocular stereo vision system calibration method.

[0006] In a first aspect, an end-to-end binocular stereo vision system calibration method is provided, comprising:

[0007] Step 1: Preprocess the checkerboard target image acquired by the binocular camera to obtain an enhanced image;

[0008] Step 2: Extract the deep feature map of the enhanced image using a shared feature extraction network;

[0009] Step 3: Extract the geometric information of the enhanced image and construct a corner topology map; encode the corner topology map to obtain the global geometric feature vector;

[0010] Step 4: Fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector;

[0011] Step 5: Input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

[0012] Preferably, S1 includes:

[0013] Step 1.1: Perform grayscale processing on the left and right checkerboard target images acquired by the binocular camera;

[0014] Step 1.2: Normalize the grayscale image;

[0015] Step 1.3: Enhance the image using Gaussian blur, contrast stretching, and edge enhancement.

[0016] Preferably, in step 2, the shared feature extraction network is ResNet-18, and the deep feature map of the image contains spatial structure information, including corner positions, edges, and textures.

[0017] Preferably, step 3 includes:

[0018] Step 3.1: Extract the geometrically relevant information of the enhanced image, including corner points, structural textures, and symmetrical meshes;

[0019] Step 3.2: Use the Harris algorithm to obtain the coarse location of the corner points of the chessboard and perform sub-pixel level optimization. Obtain the corner point coordinates by iteratively calculating the gradient direction of the region near the corner point.

[0020] Step 3.3: Embed corner points into the graph neural network using edge connection information, construct an adjacency matrix and encode it, preserve the geometric arrangement of corner points, and output the corner point topology graph;

[0021] Step 3.4: Based on the corner topology map, perform disparity field encoding, corner response intensity encoding, chessboard physical size encoding, and corner index and grid position encoding to obtain the global geometric feature vector.

[0022] Preferably, in step 4, the deep feature map of the image and the geometric feature vector are fused using an attention cross-connection mechanism.

[0023] Preferably, in step 5, the calibration parameters include focal length, principal point coordinates, rotation matrix parameters, and translation vector.

[0024] As a preferred option, it also includes:

[0025] Step 6: Construct a loss function and optimize the model based on the loss function.

[0026] In a second aspect, an end-to-end binocular stereo vision system calibration apparatus is provided for performing any of the methods described in the first aspect, including:

[0027] The preprocessing module is used to preprocess the checkerboard target images acquired by the binocular camera to obtain enhanced images;

[0028] The first extraction module is used to extract the deep feature map of the enhanced image through a shared feature extraction network;

[0029] The second extraction module is used to extract geometrically relevant information and corner coordinates of the enhanced image, and construct a corner topology map; and encode the corner topology map to obtain a global geometric feature vector;

[0030] The fusion module is used to fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector;

[0031] The input module is used to input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

[0032] Thirdly, a computer storage medium is provided, wherein a computer program is stored therein; when the computer program is run on a computer, the computer causes the computer to perform any of the methods described in the first aspect.

[0033] Fourthly, an electronic device is provided, comprising:

[0034] Memory, used to store computer programs;

[0035] A processor for executing the computer program to implement the method as described in any of the first aspects.

[0036] The beneficial effects of this invention are:

[0037] 1. The calibration process of this invention is completed automatically end-to-end, significantly reducing manual intervention. Specifically, this invention uses a deep learning network to automatically complete the entire calibration process, directly predicting camera intrinsic parameters (focal length, principal point), extrinsic parameters (rotation matrix, translation vector), and distortion parameters from input image pairs. This eliminates the need for manual corner point extraction, matching pair construction, and tedious step-by-step optimization, thereby greatly reducing labor costs and improving calibration efficiency.

[0038] 2. This invention utilizes the geometric features of checkerboard corner points for structured modeling, improving the accuracy of parameter estimation. Specifically, this invention extracts checkerboard corner points from images, constructs them into a graph structure, and models the topological relationships between corner points using a graph neural network (GNN). By combining the coordinates, response intensity, number, and scale information of the corner points, a geometric feature flow with physical meaning is formed, which helps the network understand the spatial structure of the image and improves the accuracy and robustness of calibration parameter prediction.

[0039] 3. This invention enhances the spatial modeling capability of a system by fusing image and geometric information. Specifically, this invention employs a dual-path information flow design, performing cross-attention fusion between image features and geometric feature flows composed of corner points. This allows the model to incorporate structured spatial priors while encoding image information, thereby enhancing the network's ability to perceive and express solid geometric constraints.

[0040] 4. This invention eliminates the risks of manual corner point selection and mismatches in traditional methods. Traditional methods for extracting checkerboard corner points are often affected by factors such as image quality, occlusion, and lighting, leading to risks of mismatches and missing points. This invention enhances the system's robustness to noise and adverse conditions by introducing deep feature learning and geometric coding mechanisms.

[0041] 5. This invention is applicable to calibration tasks under non-ideal setups and distortion conditions. It is not only applicable to standard binocular parallel optical axis settings, but can also be extended to scenarios where the optical axes are at an angle. By introducing geometric consistency loss and parallax direction constraints into the network, camera parameters can still be accurately estimated even with some distortion or imperfect optical axis alignment.

[0042] 6. This invention supports automatic batch data calibration and rapid online inference. The network structure of this invention is lightweight and can be deployed end-to-end, making it suitable for automatic calibration tasks involving large-scale camera arrays and batch image data. During the inference phase, input image pairs directly output complete calibration parameters without additional optimization iterations, demonstrating excellent real-time performance. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the binocular vision system structure constructed according to the present invention;

[0044] Figure 2 This is a technical roadmap diagram of the end-to-end calibration technology provided by the present invention;

[0045] Figure 3 This is a diagram of a chessboard-shaped target. Detailed Implementation

[0046] The present invention will be further described below with reference to embodiments. The description of the embodiments below is only for the purpose of helping to understand the present invention. It should be noted that those skilled in the art can make several modifications to the present invention without departing from the principle of the present invention, and these improvements and modifications also fall within the protection scope of the claims of the present invention.

[0047] Example 1:

[0048] To address the problems of existing technologies, Embodiment 1 of this application provides an end-to-end binocular stereo vision system calibration method. This method constructs a cross-attention fusion mechanism by fusing image feature streams and geometric feature streams, and realizes regression prediction of camera parameters based on multilayer perceptual networks (MLP). This method breaks through the dependence of traditional calibration methods on manual corner point extraction and prior models, and has a higher degree of automation, calibration accuracy and higher system robustness.

[0049] Specifically, such as Figure 2 As shown, the method includes:

[0050] Step 1: Preprocess the checkerboard target image acquired by the binocular camera to obtain an enhanced image.

[0051] First, such as Figure 1 As shown, two monocular cameras are rigidly connected to an operating table in an appropriate manner. The camera spacing and mounting height are adjusted according to factors such as object distance and field of view, using a checkerboard target. A checkerboard target is a widely used calibration tool in computer vision, suitable for camera calibration, 3D reconstruction, and stereo vision. It is a planar grid composed of a series of alternating black and white squares. Traditional methods calculate the camera's internal parameters by identifying the corner positions on the checkerboard target. The position of the checkerboard target is adjusted so that it is clearly displayed in the binocular camera's field of view.

[0052] In addition, this application selects a checkerboard target of appropriate size and places it in the center of the field of view. The angle and position of the target are adjusted to simultaneously acquire images from the left and right cameras as a set of image pairs. Fifty pairs of image pairs are captured as the initial image pairs. Among them, high-quality images are first screened, and finally 18 pairs of image pairs with clear corners and obvious contrast are used as input information. , enter.

[0053] Specifically, step 1 includes:

[0054] Step 1.1: Perform grayscale processing on the left and right checkerboard target images acquired by the binocular camera.

[0055] The initial image pairs are processed to grayscale, that is, the acquired color images are converted from RGB three-channel to single-channel images to retain their structural edges and texture information, eliminate color interference, and preserve the structure of the image.

[0056] The formula for grayscale conversion is as follows:

[0057]

[0058] Step 1.2: Normalize the grayscale image.

[0059] For example, the single-channel image obtained in step 1.1 is normalized to scale the pixel values ​​of the image from the original [0,255] to the range of [0,1].

[0060] The formula for normalization is as follows:

[0061]

[0062] in These are the original image pixel values. The maximum pixel value of the image. This represents the minimum pixel value of the image. Normalizing the input improves the stability of subsequent network training and prevents excessive scale differences in the input data from negatively impacting the model's training performance. Normalization ensures that images under different lighting conditions have a consistent numerical range.

[0063] Step 1.3: Enhance the image using Gaussian blur, contrast stretching, and edge enhancement.

[0064] Specifically, the image is first subjected to Gaussian blur to remove noise and smooth high-frequency interference. Then, contrast stretching is performed using linear contrast stretching and histogram equalization to enhance the contrast between bright and dark areas and improve edge texture. Next, edge enhancement is performed using the Sobel operator to enhance the edge features of the checkerboard grid corners. The corners of the checkerboard grid refer to points in the image where the grayscale value changes significantly; that is, the intersections of black and white areas on the checkerboard. These points have characteristics such as high contrast, regular distribution, and strong stability, making them easy for detection algorithms to identify. After processing, the enhanced image is obtained. , Output.

[0065] The formula for convolution using a Gaussian kernel is as follows:

[0066]

[0067] Use a Gaussian filter (kernel size 5×5). ), where (x,y) represents the coordinates of a pixel in the neighborhood relative to the center point. The standard deviation is represented, controlling the degree of blurring. G(x,y) represents the weight coefficient at the corresponding position in the filter kernel. After applying Gaussian blur, the image becomes:

[0068]

[0069] Original image Convolution with a Gaussian filter yields a blurred result. For subsequent processing. Next, contrast stretching is performed, here using linear contrast stretching:

[0070]

[0071] in These represent the maximum and minimum grayscale values ​​in the image, respectively. Finally, the Soble operator is used for edge detection and enhancement. Horizontal edges are calculated. vertical edge :

[0072]

[0073] Image gradient calculation is as follows:

[0074]

[0075] Then, by combining the formulas, we obtain the edge response map:

[0076]

[0077] The final output is an enhanced image. , The left and right image information serves as input to the shared feature extraction network.

[0078] Step 2: Extract the deep feature map of the enhanced image using a shared feature extraction network.

[0079] In step 2, the shared feature extraction network is ResNet-18, and the deep feature map of the image contains spatial structure information, including corner positions, edges, and textures.

[0080] For example, the enhanced left image And the right image The input is a ResNet-18 shared feature extraction network, which extracts features from the left and right images respectively, and outputs deep feature maps of the left and right images containing spatial structure information such as corner positions, edges, and textures. .

[0081] Specifically, the shared feature extraction network structure includes convolutional layers for extracting low-level texture features, stacked residual connection modules to capture mid-to-high-level semantic information, batch normalization and ReLU activation to enhance nonlinear modeling, spatial downsampling to extract multi-scale spatial structures, and finally outputs deep feature maps of the left and right images, including feature map height, width and depth information.

[0082] Step 3: Extract the geometric information of the enhanced image and construct a corner topology map; encode the corner topology map to obtain the global geometric feature vector.

[0083] Step 3 includes:

[0084] Step 3.1: Extract the geometrically relevant information of the enhanced image, including corner points and symmetric grids.

[0085] Step 3.2: Use the Harris algorithm to obtain the coarse location of the corner points of the chessboard and perform sub-pixel level optimization. Obtain the corner point coordinates by iteratively calculating the gradient direction of the region near the corner point.

[0086] For example, the coordinates of the corner point are represented as , Output a sequence of two-dimensional corner coordinates. .

[0087] Step 3.3: Using edge connection information, embed corner points into a graph neural network, construct an adjacency matrix and encode it, preserve the geometric arrangement relationship of corner points, and output the corner point topology graph, which is represented as a sparse adjacency matrix or graph vector.

[0088] Edge connectivity information refers to structured data describing the topological relationships between corner points. It defines the connection methods and attributes between corner points. Based on the rules of a chessboard structure, it automatically constructs grid connections according to the known number of rows and columns of the chessboard, horizontally connecting adjacent corner points in the same row and vertically connecting adjacent corner points in the same column. Based on the dynamics of spatial relationships, it calculates the Euclidean distance between corner points, connects points with a distance less than a threshold (e.g., 50 pixels), and records the direction vector of the connection. Based on image content enhancement, it extracts image gradients along the connection lines. For example, the larger the Sobel edge strength gradient, the higher the connection weight. The connection strength reflects the edge continuity in the actual image.

[0089] Step 3.4: Based on the corner topology map, perform disparity field encoding, corner response intensity encoding, chessboard physical size encoding, and corner index and grid position encoding to obtain the global geometric feature vector.

[0090] Specifically, disparity field coding includes: defining disparity as... Calculated only along the x-axis, the aligned image estimates initial disparity information by the lateral difference between the corner positions in the left and right images. The disparity vector is formed by the disparity values ​​of each corner.

[0091] For example, the corner points of the chessboard in the left and right images satisfy a one-to-one correspondence. Let the i-th corner point in the left image and the i-th corner point in the right image be... , These are the image points of the same physical point on different cameras. The parallax is calculated as the difference in horizontal coordinates between corresponding corner points. .

[0092] Corner response strength encoding includes: using Harris response, SIFT strength, or OpenCV corner response values ​​as weights to guide the network to focus on high-quality corners, and outputting the confidence score of each corner. .

[0093] For example, the response strength of each corner is calculated using the response value of the Harris corner detector as a weight, guiding the network to focus on corners of higher quality, and the confidence score of each corner is output as denoted as . It is used to reflect whether the local area of ​​a point is a stable corner point.

[0094] The physical size encoding of the chessboard includes: encoding the side length of the chessboard squares as l = 20.0 mm, which is added to the features as a global scale factor. Scene scale information is provided to recover the depth and camera parameters at the true scale. A scalar scale code s is output and propagated to each corner.

[0095] For example, combining the chessboard grid size information, the physical size of the chessboard is encoded, with the side length of the encoded chessboard grid being l = 20.0 mm, which is added to the features as a global scale factor. Scene scale information is provided to recover the depth and camera parameters at the true scale. A scalar scale code s is output and propagated to each corner point. The final result is composed of the coordinates of the left pixel corner point. Right pixel coordinates Corresponding strength at corner points checkerboard grid size information Corner number disparity vector .

[0096] Corner index and grid position encoding includes: adding a positional code for each corner point to its row i and column j, similar to Positional Encoding in Transformer. This describes the row and column indices of the corner point in the checkerboard, forming a structured input and outputting a grid position embedding vector.

[0097] For example, each corner point is numbered and denoted as follows: For each corner point, a positional encoding is added to its row i and column j, similar to Positional Encoding in Transformer. This encoding describes the row and column indices of the corner point within the checkerboard, forming a structured input. The output grid position embedding vector is positionally encoded according to a left-to-right, top-to-bottom order to represent the topological positional information within the checkerboard.

[0098] The final output node feature vector for each corner point is:

[0099]

[0100] Where D represents the total dimension (2 coordinates + 1 response value + 1 disparity + N-dimensional index).

[0101] Next, a graph structure is constructed where each corner point corresponds to a node. Each node includes normalized corner coordinates, and the corner points include four features: response intensity, corner number encoding, and disparity value. Nodes are connected by edges based on a checkerboard topology and adjacent corner points. The initial feature vector of each node is input through a neural network (GNN). The algorithm aggregates features from neighboring nodes using GraphSAGE, performs linear transformation, and then uses GAT (Graph Attention Network) to assign different attention weights for weighted fusion, outputting a low-dimensional embedding vector for each node. Global Mean Pooling is used.

[0102]

[0103] Obtain the global geometric eigenvectors , representing the global feature vector of the entire chessboard.

[0104] Step 4: Fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector.

[0105] Step 5: Input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

[0106] Example 2:

[0107] Based on Example 1, Example 2 of this application provides a more specific end-to-end binocular stereo vision system calibration method, including:

[0108] Step 1: Preprocess the checkerboard target image acquired by the binocular camera to obtain an enhanced image.

[0109] For example, two identical monocular cameras are rigidly connected in parallel at the same horizontal position using aluminum profiles, and the rest of the camera is also rigidly connected and fixed to an operating table. Figure 1 As shown. The distance between the two cameras and the mounting height of the cameras are adjusted according to factors such as the object distance to the object to be detected, the size of the field of view, and the light intensity, so that the object to be detected, i.e., the checkerboard target, can be clearly presented in the center of the binocular camera's field of view.

[0110] Select a checkerboard target with dimensions of 9×6 mm and a diameter of 18.5 mm based on the field of view. Figure 3 The target is positioned at the center of the field of view, ensuring the checkerboard pattern occupies more than 70% of the central image area. Simultaneously acquire checkerboard images from both the left and right cameras as image pairs, setting the resolution to 1280×960. Each image contains 9×6 complete interior corner points. Because there is a significant parallax (10-50 pixels) between the checkerboard positions in the left and right images, it is necessary to capture images from various angles and on different planes in a uniformly lit environment without reflections. Adjust the target's angle and position, simultaneously acquiring images from both the left and right cameras as a set of image pairs. A total of 50 image pairs are captured, and the 18 highest-quality pairs are recorded as [image pairs not specified in the original text]. , As initial image information.

[0111] Step 2: Extract the deep feature map of the enhanced image using a shared feature extraction network.

[0112] Step 3: Extract the geometric information and corner coordinates of the enhanced image, and construct a corner topology map; encode the corner topology map to obtain the global geometric feature vector.

[0113] Step 4: Fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector.

[0114] Specifically, image semantic features The geometric features are concatenated with the geometric feature vector g to form the final input vector. An attention cross-feedback mechanism is used to fuse the geometric features according to their semantic features, and the fused features are used as the input to the regression network.

[0115] The fusion formula is expressed as follows:

[0116]

[0117] An attention mechanism is introduced to weightedly fuse geometric and semantic features. Let the semantic features of the left and right images be denoted as follows: spliced Using the semantic features of the entire image as the query and the geometric features g as the key and value, a cross-attention mechanism is constructed as follows:

[0118]

[0119]

[0120] in For learnable linear transformation weights, This is the attention weighting coefficient. This represents the geometric features after attention enhancement. Finally, the attention-fused geometric vector is concatenated with the image semantic vector to form:

[0121]

[0122] Step 5: Input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

[0123] Specifically, the parameter regression network uses a fully connected network to regress the parameters. MLP stands for Multilayer Perceptron. It is a vector containing the following parameters: focal length Main point distortion coefficient The intrinsic parameters, along with the rotation vector r represented by the rotation matrix R and the translation vector represented by the translation matrix T, are collectively expressed as:

[0124]

[0125] Furthermore, the extracted feature maps are subjected to fully connected mapping and nonlinear activation to regress calibration parameters to form a vector. Specifically, this includes focal length, principal point coordinates, rotation matrix parameters, translation vector, etc.

[0126] Step 6: Construct a loss function and optimize the model based on the loss function.

[0127] Specifically, step 6 includes:

[0128] Step 6.1: Construct the projection error of the matching point pair in the reconstructed image. Using the predicted camera parameters, project the three-dimensional spatial points onto the image plane and calculate the error between the projected points and the detected corner points.

[0129] Specifically, to supervise the network in learning camera parameters that conform to the solid geometry, a geometric consistency loss based on reprojection error is constructed. For each detected checkerboard corner point in the image... Corresponding 3D points Given the camera's intrinsic parameters K, extrinsic parameters R, T, and distortion coefficients d obtained from network prediction, construct the camera projection matrix. Through the projection function:

[0130]

[0131] A point in the world coordinate system Projecting the image onto the image plane yields the predicted pixel coordinates. The reprojection error is defined as:

[0132]

[0133] These are the true pixel coordinates and the predicted reprojection coordinates of the i-th detected corner point, respectively.

[0134] A parallax direction consistency loss is constructed because a binocular vision system should satisfy the epipolar constraint, ensuring that the projections of the same corner point in the left and right images lie on the same scan line, thus satisfying parallax direction consistency. For spatial points... The predicted parameters from the left and right cameras are projected separately to obtain the following:

[0135]

[0136] Define disparity consistency loss as:

[0137]

[0138] In addition, to prevent abnormal prediction parameters, such as excessively large distortion coefficients or excessively large principal point drift, a regularization term is added:

[0139]

[0140] in The distortion coefficient is... These represent the currently predicted intrinsic parameter matrix and the initial estimated reference intrinsic parameters, respectively. This is the regularization weight coefficient.

[0141] Step 6.2: Taking into account the differences between the real pixels and the pixels reconstructed by the current prediction parameters, as well as the parameter regularization term, construct a loss function to measure the deviation between the model prediction result and the real value.

[0142] Specifically, the total loss function is expressed as:

[0143]

[0144] in, These are the weights of the disparity consistency loss and the regularization term, respectively. The total loss function enables the model output parameters to have good generalization ability.

[0145] Step 6.3, Training and Optimization: Use the Adam optimizer to jointly optimize the network weights and calibration parameters.

[0146] Specifically, the entire network image encoder, graph feature extraction, fusion module, and parameter regression network are all differentiable mechanisms, and the loss is differentiable for all network parameters and camera calibration parameters. Differentiate:

[0147]

[0148] The entire network is trained end-to-end to continuously adjust the parameters of the regression network model and improve its performance.

[0149] It should be noted that the parts in this embodiment that are the same as or similar to those in Embodiment 1 can be referred to each other, and will not be repeated in this application.

[0150] Example 3:

[0151] Based on Embodiment 1, Embodiment 3 of this application provides an end-to-end binocular stereo vision system calibration device, comprising:

[0152] The preprocessing module is used to preprocess the checkerboard target images acquired by the binocular camera to obtain enhanced images;

[0153] The first extraction module is used to extract the deep feature map of the enhanced image through a shared feature extraction network;

[0154] The second extraction module is used to extract geometrically relevant information and corner coordinates of the enhanced image, and construct a corner topology map; and encode the corner topology map to obtain a global geometric feature vector;

[0155] The fusion module is used to fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector;

[0156] The input module is used to input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

[0157] It should be noted that the system provided in this embodiment is the corresponding device for the method provided in embodiment 1. Therefore, the parts that are the same as or similar to those in embodiment 2 in this embodiment can be referred to each other, and will not be repeated in this application.

Claims

1. A calibration method for an end-to-end binocular stereo vision system, characterized in that, include: Step 1: Preprocess the checkerboard target image acquired by the binocular camera to obtain an enhanced image; Step 2: Extract the deep feature map of the enhanced image using a shared feature extraction network; Step 3: Extract the geometrically relevant information of the enhanced image and construct a corner topology map; The corner topology map is then encoded to obtain a global geometric feature vector; Step 4: Fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector; Step 3 includes: Step 3.1: Extract the geometrically relevant information of the enhanced image, including corner points, structural textures, and symmetrical meshes; Step 3.2: Use the Harris algorithm to obtain the coarse location of the corner points of the chessboard and perform sub-pixel level optimization. Obtain the corner point coordinates by iteratively calculating the gradient direction of the region near the corner point. Step 3.3: Embed corner points into the graph neural network using edge connection information, construct an adjacency matrix and encode it, preserve the geometric arrangement of corner points, and output the corner point topology graph; Step 3.4: Based on the corner topology map, perform disparity field encoding, corner response intensity encoding, chessboard physical size encoding, and corner index and grid position encoding to obtain the global geometric feature vector; Step 5: Input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

2. The end-to-end binocular stereo vision system calibration method according to claim 1, characterized in that, Step 1 includes: Step 1.1: Perform grayscale processing on the left and right checkerboard target images acquired by the binocular camera; Step 1.2: Normalize the grayscale image; Step 1.3: Enhance the image using Gaussian blur, contrast stretching, and edge enhancement.

3. The end-to-end binocular stereo vision system calibration method according to claim 2, characterized in that, In step 2, the shared feature extraction network is ResNet-18, and the deep feature map of the image contains spatial structure information, including corner positions, edges, and textures.

4. The end-to-end binocular stereo vision system calibration method according to claim 3, characterized in that, In step 4, the deep feature map of the image and the geometric feature vector are fused using an attention cross-connection mechanism.

5. The end-to-end binocular stereo vision system calibration method according to claim 4, characterized in that, In step 5, the calibration parameters include focal length, principal point coordinates, rotation matrix parameters, and translation vector.

6. The end-to-end binocular stereo vision system calibration method according to claim 5, characterized in that, Also includes: Step 6: Construct a loss function and optimize the model based on the loss function.

7. A calibration device for an end-to-end binocular stereo vision system, characterized in that, For performing the method according to any one of claims 1 to 6, comprising: The preprocessing module is used to preprocess the checkerboard target images acquired by the binocular camera to obtain enhanced images; The first extraction module is used to extract the deep feature map of the enhanced image through a shared feature extraction network; The second extraction module is used to extract geometrically relevant information and corner coordinates of the enhanced image, and construct a corner topology map; and encode the corner topology map to obtain a global geometric feature vector; The fusion module is used to fuse the deep feature map of the image with the geometric feature vector to generate a fused feature vector; The input module is used to input the fused feature vector into the parameter regression network to predict the calibration parameters of the stereo camera.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program; when the computer program is run on the computer, it causes the computer to perform the method described in any one of claims 1 to 6.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in any one of claims 1 to 6.

Images