Machine vision positioning system and method based on double cameras

Abstract

The invention relates to the technical field of machine vision, and discloses a machine vision positioning system and method based on double cameras, and the method comprises the steps: synchronously capturing a left image and a right image of a target object through the double cameras; carrying out online self-calibration on the double cameras, and obtaining an internal parameter matrix and an external parameter matrix of the cameras in real time; extracting feature points in the left and right images based on a deep learning model, and generating feature descriptors; carrying out three-dimensional reconstruction by utilizing a feature point matching and triangulation method, and calculating a three-dimensional space coordinate of the target object; outputting a positioning result of the target object according to the three-dimensional coordinates; and a positioning result is optimized by using a filtering algorithm, so that the stability and the precision are improved. According to the method, the internal and external parameters of the double cameras are updated in real time through an online self-calibration technology, the camera pose change or environmental interference is effectively dealt with, the adaptability and long-term operation stability of the system in a dynamic scene are improved, and the defect that frequent manual adjustment is needed in a traditional calibration method is avoided.

Description

Technical Field

[0001] This invention relates to the field of machine vision technology, and in particular to a machine vision positioning system and method based on dual cameras. Background Technology

[0002] In the current field of machine vision technology, dual-camera-based stereo localization systems have been widely used in 3D perception and target localization. However, existing technologies mostly rely on pre-performed offline calibration, which makes it difficult to adapt to parameter drift caused by dynamic changes in camera pose or environmental disturbances. At the same time, in the feature extraction stage, traditional manual feature extraction methods lack robustness in complex lighting and occlusion scenarios, leading to a decrease in matching accuracy.

[0003] In addition, existing systems often neglect real-time filtering and optimization of positioning results during 3D reconstruction, resulting in noise accumulation and jitter in the output data, which restricts the practical application effect and stability of the system in dynamic industrial scenarios. Summary of the Invention

[0004] This invention provides a machine vision positioning system and method based on dual cameras to solve existing technical problems, thereby addressing the issue of parameter drift caused by dynamic changes in camera pose or environmental disturbances.

[0005] To solve the above-mentioned technical problems, according to one aspect of the present invention, more specifically, a machine vision positioning method based on dual cameras, comprising the following steps:

[0006] S1. Use dual cameras to simultaneously capture left and right images of the target object;

[0007] S2. Perform online self-calibration on the dual cameras to obtain the camera's intrinsic and extrinsic parameter matrices in real time;

[0008] S3. Extract feature points from the left and right images based on a deep learning model and generate feature descriptors;

[0009] S4. Use feature point matching and triangulation methods to perform 3D reconstruction and calculate the 3D spatial coordinates of the target object.

[0010] S5. Output the positioning result of the target object based on the three-dimensional coordinates;

[0011] S6. Use filtering algorithms to optimize the positioning results to improve stability and accuracy.

[0012] Furthermore, the online self-calibration in capturing S2 specifically includes:

[0013] S201. Take multiple sets of image pairs using dual cameras and extract natural feature points from the images;

[0014] S202. Calculate the fundamental matrix and essential matrix of the left and right image pairs;

[0015] S203. Use the Random Sampling Consensus Algorithm (RANSAC) to eliminate mismatches and optimize the essential matrix;

[0016] S204. Decompose the essential matrix to obtain the camera's rotation matrix and translation vector, and combine it with the camera's intrinsic parameters to complete the calibration.

[0017] Furthermore, the fundamental matrix constructs the geometric constraints of corresponding points in the left and right camera images by satisfying the linear equation defined by the fundamental matrix through a pair of corresponding point coordinates in the left and right camera images, such that the product of the transpose of the homogeneous coordinates of a point in the left image, the fundamental matrix, and the homogeneous coordinates of the corresponding point in the right image is zero.

[0018] Furthermore, the essential matrix and the fundamental matrix are related through the intrinsic parameter matrices of the left and right cameras. Specifically, the essential matrix is ​​equal to the transpose of the intrinsic parameter matrix of the right camera, the product of the fundamental matrix and the intrinsic parameter matrix of the left camera. This mathematical relationship transforms the pixel-level constraints between image coordinates into geometric constraints in the normalized camera coordinate system.

[0019] Furthermore, the loss function used for training the deep learning model consists of two parts:

[0020] The first part is the mean square error loss between the predicted and actual locations of feature points;

[0021] The second part is the consistency loss of feature descriptors, which is used to constrain the feature descriptors extracted by the model to have high similarity between matching point pairs;

[0022] The total loss function is a weighted sum of the two aforementioned losses. The weight coefficients of the two losses are adjusted to balance the model's learning objectives of feature point localization accuracy and descriptor discrimination ability.

[0023] Furthermore, the triangulation method in S4 employs linear least squares, with the objective function being:

[0024] ;

[0025] In the above formula, X represents a point in three-dimensional space; Indicates the coordinates of the i-th image point; Let represent the projection matrix of the i-th camera.

[0026] Furthermore, the filtering algorithm in S6 is a Kalman filter, and its state equation and observation equation are as follows:

[0027] ;

[0028] ;

[0029] In the above formula, Represents the state vector; Represents the observation vector; A and H are transformation matrices; and This is the noise term.

[0030] Furthermore, the dual cameras employ a global shutter and synchronous triggering mechanism, with a frame rate of no less than 60fps, making them suitable for real-time positioning in dynamic scenes.

[0031] A machine vision positioning system based on dual cameras includes: an image acquisition module, a data processing module, and a control output module, wherein the data processing module includes: an online calibration unit, a feature extraction unit, a 3D reconstruction unit, and a filtering optimization unit.

[0032] The present invention provides a machine vision positioning system and method based on dual cameras. Compared with the prior art, the advantages achieved by this method are as follows:

[0033] 1. This invention updates the internal and external parameters of dual cameras in real time through online self-calibration technology, effectively responding to changes in camera pose or environmental interference, improving the system's adaptability in dynamic scenes and long-term operational stability, and avoiding the drawbacks of traditional calibration methods that require frequent manual adjustments.

[0034] 2. This invention uses a deep learning model to extract feature points and descriptors, which can adaptively learn key features in complex scenes, significantly improving the accuracy and robustness of feature matching, and can maintain high-precision positioning even in challenging environments such as changes in lighting and partial occlusion.

[0035] 3. This invention uses a three-dimensional reconstruction method that combines triangulation and bundle adjustment. While ensuring positioning accuracy, it optimizes computational efficiency and can quickly recover the three-dimensional spatial coordinates of the target object, providing a reliable spatial information foundation for real-time applications.

[0036] 4. This invention introduces a Kalman filter algorithm to optimize the positioning results, effectively suppressing measurement noise and jitter, outputting smooth and stable three-dimensional coordinates, and enhancing the reliability of the system in continuous tracking and dynamic control.

[0037] 5. This invention utilizes a dual-camera synchronous acquisition and global shutter mechanism to ensure the spatiotemporal consistency of image capture. Combined with a modular system design, it achieves full automation from image acquisition and processing to output, significantly improving the real-time performance and ease of integration of the overall system. Attached Figure Description

[0038] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0039] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operating procedures; however, the scope of protection of the present invention is not limited to the following embodiments.

[0040] Example 1:

[0041] like Figure 1 As shown, a machine vision localization method based on dual cameras includes:

[0042] Step S1: Simultaneously capture images with dual cameras.

[0043] The dual cameras simultaneously capture left and right images of the target object. To ensure image quality, it is recommended to conduct the operation in an environment with uniform lighting and no strong reflections. If the ambient light is unstable, an active light source (such as an LED ring light) can be used for supplemental lighting. During image acquisition, the dual cameras should maintain a fixed baseline distance (e.g., 100mm~500mm), and the shooting distance should be adjusted according to the size of the target object.

[0044] Step S2: Online self-calibration

[0045] Online self-calibration is a key feature of this invention. Its purpose is to update the intrinsic and extrinsic parameters of the dual cameras in real time during system operation to address issues such as camera pose changes or temperature drift. Specific implementation includes the following sub-steps:

[0046] 1. Acquire multiple image pairs: Take multiple image pairs containing natural feature points using dual cameras (e.g., no fewer than 10 pairs). Natural feature points can be prominent features in the scene such as corners and edges, or high-contrast markers can be pre-placed in the scene.

[0047] 2. Extracting Natural Feature Points: Feature points in the image are extracted using feature detection algorithms such as SIFT, ORB, or FAST. To improve robustness, the edge extraction method described in CN117990007A can be referenced, using the Canny operator for edge detection, combined with non-maximum suppression and double thresholding to ensure clear feature points and low noise.

[0048] 3. Calculate the fundamental matrix and the essential matrix:

[0049] The fundamental matrix (F) is calculated by satisfying the following constraint equations using the coordinates of a pair of corresponding points in the left and right images:

[0050] ;

[0051] In the above formula, and These are the homogeneous coordinates of corresponding points in the left and right images, respectively.

[0052] The essential matrix (E) and the fundamental matrix are related through the camera intrinsic matrix:

[0053] ;

[0054] In the above formula, and These are the intrinsic parameter matrices for the left and right cameras, respectively.

[0055] 4. False Match Removal and Optimization: False matches are removed using the Random Sampling Consensus Algorithm (RANSAC), and the essential matrix is ​​optimized. It is recommended that the RANSAC iterations be at least 1000, with a confidence level set to 0.99.

[0056] 5. Decompose the essential matrix to obtain extrinsic parameters: Perform singular value decomposition (SVD) on the essential matrix to obtain the camera's rotation matrix R and translation vector t. Combine this with the camera's intrinsic parameters to complete the calibration.

[0057] 6. Distortion Correction and Optimization: Radial distortion in the image is corrected. A radial distortion model is used:

[0058] ;

[0059] In the above formula, r is the distance from the image point to the optical center; , The distortion coefficients are denoted by . The Levenberg-Marquardt algorithm is used to optimize the distortion coefficients by minimizing the reprojection error.

[0060] Step S3: Extract feature points and descriptors based on deep learning model

[0061] This step uses a pre-trained deep learning model to extract feature points from the left and right images and generate feature descriptors. Model training details are as follows:

[0062] Network structure: An encoder-decoder structure is adopted. The encoder uses ResNet-50 or VGG16 backbone, and the decoder uses convolutional layers and upsampling layers to output feature point heatmaps and descriptor tensors.

[0063] Loss function: The total loss function consists of two parts:

[0064] Feature point location loss: The mean squared error (MSE) loss is used to calculate the error between the predicted feature point location and the true location.

[0065] Descriptor consistency loss: Uses contrastive loss or triplet loss to constrain the descriptor similarity between matching point pairs.

[0066] The total loss is the weighted sum:

[0067] ;

[0068] In the above formula, and These are weighting coefficients, typically set to 1.0 and 0.5.

[0069] Training data: Use synthetic datasets or real labeled datasets (such as COCO, ImageNet) for training, and perform data augmentation (rotation, scaling, brightness changes).

[0070] Step S4: Feature Point Matching and 3D Reconstruction

[0071] The three-dimensional spatial coordinates of the target object are calculated using feature point matching and triangulation methods.

[0072] 1. Feature point matching: Use nearest neighbor search (such as FLANN) or brute-force matching, combined with descriptor distance ratio test (such as Lowe's ratio test) to eliminate mismatches.

[0073] 2. Triangulation: Triangulation is performed using the linear least squares method. The objective function is:

[0074] ;

[0075] In the above formula, X represents a point in three-dimensional space; Indicates the coordinates of the i-th image point; Let represent the projection matrix of the i-th camera.

[0076] 3. Optimize 3D points: Use bundle adjustment to optimize 3D point coordinates and camera parameters to minimize reprojection errors.

[0077] Step S5: Output the positioning result

[0078] The system outputs the location result of the target object based on three-dimensional coordinates, in the format of a three-dimensional vector (X,Y,Z), in millimeters. It can be published via serial port, Ethernet, or ROS topic.

[0079] Step S6: Filter Optimization

[0080] The positioning results are optimized using a Kalman filter to improve stability and accuracy.

[0081] Its equation of state is:

[0082] ;

[0083] In the above formula, A represents the state vector (position, velocity); A is the state transition matrix; This is process noise.

[0084] Its observation equation is:

[0085] ;

[0086] In the above formula, H represents the observation vector; H is the observation matrix; To observe noise.

[0087] Example 2:

[0088] A machine vision positioning system based on dual cameras includes: an image acquisition module, a data processing module, and a control output module, wherein the data processing module includes: an online calibration unit, a feature extraction unit, a 3D reconstruction unit, and a filtering optimization unit.

[0089] The image acquisition module includes the dual cameras mentioned above, which are used to simultaneously capture left and right images of the target object;

[0090] The data processing module is communicatively connected to the image acquisition module and is used to receive the left and right images;

[0091] The online calibration unit is configured to perform online self-calibration to acquire the intrinsic and extrinsic parameter matrices of the two cameras in real time.

[0092] The feature extraction unit integrates a trained deep learning model and is configured to extract feature points from the left and right images and generate corresponding feature descriptors.

[0093] The three-dimensional reconstruction unit is configured to use the feature points for matching and to use a triangulation method to calculate the three-dimensional spatial coordinates of the target object.

[0094] The filtering optimization unit is configured to perform filtering processing on the three-dimensional spatial coordinates to optimize the stability and accuracy of the positioning results;

[0095] The control output module is communicatively connected to the data processing module and is used to receive and output the final three-dimensional positioning result.

[0096] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.

Claims

1. A machine vision localization method based on dual cameras, characterized in that, Includes the following steps: S1. Use dual cameras to simultaneously capture left and right images of the target object; S2. Perform online self-calibration on the dual cameras to obtain the intrinsic and extrinsic parameter matrices of the cameras in real time; S3. Extract feature points from the left and right images based on a deep learning model and generate feature descriptors; S4. Use feature point matching and triangulation methods to perform 3D reconstruction and calculate the 3D spatial coordinates of the target object. S5. Output the positioning result of the target object based on the three-dimensional coordinates; S6. Optimize the positioning results using a filtering algorithm.

2. The machine vision positioning system and method based on dual cameras according to claim 1, characterized in that: Online self-calibration in S2 specifically includes: S201. Take multiple sets of image pairs using dual cameras and extract natural feature points from the images; S202. Calculate the fundamental matrix and essential matrix of the left and right image pairs; S203. Use the random sampling consensus algorithm to eliminate mismatch points and optimize the essential matrix; S204. Decompose the essential matrix to obtain the camera's rotation matrix and translation vector, and combine it with the camera's intrinsic parameters to complete the calibration.

3. The machine vision positioning system and method based on dual cameras according to claim 2, characterized in that: The fundamental matrix constructs the geometric constraints of corresponding points in the left and right camera images by satisfying the linear equation defined by the fundamental matrix through a pair of corresponding point coordinates in the left and right camera images, such that the product of the transpose of the homogeneous coordinates of a point in the left image, the fundamental matrix, and the homogeneous coordinates of the corresponding point in the right image is zero.

4. The machine vision positioning system and method based on dual cameras according to claim 2, characterized in that: The essential matrix and the fundamental matrix are related through the intrinsic parameter matrices of the left and right cameras. Specifically, the essential matrix is ​​equal to the transpose of the intrinsic parameter matrix of the right camera, the product of the fundamental matrix and the intrinsic parameter matrix of the left camera.

5. The machine vision positioning system and method based on dual cameras according to claim 1, characterized in that: The loss function used for training the deep learning model consists of two parts: The first part is the mean square error loss between the predicted and actual locations of feature points; The second part is the consistency loss of feature descriptors, which is used to constrain the feature descriptors extracted by the model to have high similarity between matching point pairs; The total loss function is a weighted sum of the two aforementioned losses. The weight coefficients of the two losses are adjusted to balance the model's learning objectives of feature point localization accuracy and descriptor discrimination ability.

6. The machine vision positioning system and method based on dual cameras according to claim 1, characterized in that: The triangulation method in S4 uses linear least squares, and the objective function is: ； In the above formula, X represents a point in three-dimensional space; Indicates the coordinates of the i-th image point; Let represent the projection matrix of the i-th camera.

7. The machine vision positioning method based on dual cameras according to claim 1, characterized in that: The filtering algorithm in S6 is a Kalman filter, and its state equation and observation equation are as follows: ； ； In the above formula, Represents the state vector; Represents the observation vector; A and H are transformation matrices; and This is the noise term.

8. The machine vision positioning method based on dual cameras according to claim 1, characterized in that: The dual cameras employ a global shutter and synchronous triggering mechanism, with a frame rate of no less than 60fps, making them suitable for real-time positioning in dynamic scenes.

9. A machine vision positioning system based on dual cameras, characterized in that, The machine vision positioning system based on dual cameras, applicable to any one of claims 1-8, comprises: an image acquisition module, a data processing module, and a control output module, wherein the data processing module comprises: an online calibration unit, a feature extraction unit, a 3D reconstruction unit, and a filtering optimization unit.

Images