High-precision online measurement algorithm based on 3D camera

Abstract

The invention discloses a high-precision on-line measurement algorithm based on a 3D camera. The high-precision on-line measurement algorithm comprises the following on-line measurement steps: step 1, multi-modal data fusion; step 2, self-adaptive error compensation; 3, online calibration and feedback optimization are carried out; according to the method, the precision and the stability of online measurement are improved, measurement errors caused by factors such as camera parameter drift and motion blur in the prior art are overcome, the real-time performance of a measurement system is enhanced, the real-time measurement requirement on a high-speed production line is met, and the real-time measurement accuracy of the high-speed production line is improved. The problems of high calculation complexity and low processing speed in the prior art are solved, the adaptability of a measurement system is improved, the measurement system can stably work in a complex and changeable industrial environment, the problems that the measurement system is sensitive to environment change and needs frequent shutdown calibration in the prior art are solved, the potential of a 3D camera can be fully utilized, efficient and accurate online measurement is achieved, and the measurement accuracy is improved. And the development of industrial automation and intelligence is promoted.

Description

Technical Field

[0001] This invention relates to the field of online measurement technology, specifically a high-precision online measurement algorithm based on a 3D camera. Background Technology

[0002] In the process of industrial automation, online measurement technology is crucial. Traditional contact measurement is inefficient and susceptible to interference, making it difficult to meet the needs of modern industry. While non-contact measurement has certain advantages, it is significantly lacking in accuracy and stability, especially in real-time dynamic monitoring scenarios. Online measurement technology is an emerging and advanced measurement technology that allows processing and measurement to be performed on the same equipment, avoiding errors caused by secondary clamping. It also improves efficiency and allows for real-time inspection during product manufacturing, such as quality control on assembly lines, enabling real-time monitoring and assurance of product quality and accuracy. Existing technologies struggle to balance measurement accuracy, stability, and efficiency. Although 3D cameras are widely used, their measurement accuracy and stability are affected by various factors, such as camera parameter drift and motion blur caused by temperature changes, limiting their further application in high-precision measurement. In existing technologies, some measurement methods attempt to combine data from different sensors to improve measurement accuracy, such as using fusion measurement of 2D images and 3D point cloud data.

[0003] However, current measurement methods often have shortcomings in feature extraction and fusion, failing to fully leverage the advantages of multimodal data, resulting in limited measurement accuracy and robustness. Furthermore, for issues such as camera parameter drift and motion blur, existing compensation methods are mostly static or semi-static calibrations, unable to be dynamically adjusted in real time, making it difficult to adapt to complex and ever-changing industrial environments. In terms of real-time processing, existing algorithms have high computational complexity, making it difficult to meet the real-time measurement needs of high-speed production lines. Summary of the Invention

[0004] This invention provides a high-precision online measurement algorithm based on a 3D camera, which can effectively solve the problems mentioned in the background art. Current measurement methods often have shortcomings in feature extraction and fusion, failing to fully utilize the advantages of multimodal data, resulting in limited measurement accuracy and robustness. In addition, for the problems of camera parameter drift and motion blur, existing compensation methods are mostly static or semi-static calibrations, which cannot be dynamically adjusted in real time and are difficult to adapt to complex and ever-changing industrial environments. In terms of real-time processing, existing algorithms have high computational complexity and cannot meet the real-time measurement requirements of high-speed production lines.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a high-precision online measurement algorithm based on a 3D camera, comprising the following online measurement steps:

[0006] Step 1: Multimodal data fusion;

[0007] Step 2, adaptive error compensation;

[0008] Step 3: Online calibration and feedback optimization;

[0009] Step 4: Lightweight real-time processing;

[0010] In step one, a 3D camera is used to collect 3D point cloud data and 2D image texture information of the object under test, data preprocessing is performed, features of 3D point cloud and 2D image are extracted, and feature fusion is performed through a multimodal data fusion architecture to enhance the robustness of feature extraction.

[0011] In step two, an error model is established, and an adaptive error compensation mechanism is used to dynamically calibrate the measurement error, thereby improving measurement accuracy and stability.

[0012] In step three, the calibration parameters are automatically updated using a reference object, enabling continuous optimization of the measurement system without downtime.

[0013] Step four involves using GPU-accelerated point cloud processing and lightweight real-time processing algorithms to achieve efficient real-time processing of the measurement process.

[0014] According to the above technical solution, step one specifically includes data acquisition, data preprocessing, feature extraction and fusion;

[0015] The data acquisition includes 3D point cloud data acquisition and 2D image texture information acquisition;

[0016] The data preprocessing includes 3D point cloud data preprocessing and 2D image preprocessing;

[0017] The 3D point cloud data preprocessing includes filtering and denoising, downsampling, and normal estimation; the 2D image preprocessing includes grayscale correction and edge detection.

[0018] The feature extraction and fusion includes feature extraction and feature fusion;

[0019] The feature extraction includes 3D point cloud feature extraction and 2D image feature extraction, and the feature fusion includes feature-level fusion and attention-based fusion.

[0020] The 3D point cloud data acquisition method involves using a 3D camera to scan the object under test and obtain point cloud data containing the depth information of the object's surface. The camera's resolution and field of view parameters are configured according to the specific measurement scenario and accuracy requirements.

[0021] The frequency of point cloud data acquisition needs to meet real-time requirements, and is generally determined based on the moving speed of objects on the production line or the timeliness of the measurement task.

[0022] The 2D image texture information acquisition synchronously acquires 2D images corresponding to the 3D point cloud to provide texture details of the object. The resolution of the 2D image should match that of the 3D camera to ensure that the information of each pixel can be accurately matched during the fusion process.

[0023] Use appropriate lighting conditions to ensure the clarity of 2D images and the recognizability of textures, and avoid affecting the quality of texture information due to uneven lighting or overexposure.

[0024] According to the above technical solution, the filtering and denoising uses voxel grid filtering and statistical filtering methods to remove noise points in the point cloud;

[0025] The downsampling mentioned above is used to reduce the data size and improve processing efficiency when the point cloud data volume is too large. Random sampling and uniform sampling strategies can be used, and key features of the object need to be preserved.

[0026] The normal estimation involves calculating the normal direction at each point to provide geometric information for subsequent feature extraction and fusion. Commonly used methods include covariance analysis based on neighborhood points.

[0027] The grayscale correction refers to the grayscale correction of the image, specifically through histogram equalization, which enhances the contrast of the image and makes the texture features more obvious.

[0028] The edge detection uses Canny edge detection and Sobel operator to extract image edges, highlighting the contour information of objects and preparing for feature matching.

[0029] According to the above technical solution, the 3D point cloud feature extraction is based on the geometric shape of the point cloud, extracting curvature and principal component analysis (PCA) features;

[0030] The 2D image feature extraction uses SIFT and SURF algorithms to extract key points and descriptors in the image. These features are robust to changes in illumination and viewpoint. SIFT: Scale Invariant Feature Transform, SURF: Accelerated Robust Feature Transform.

[0031] The feature-level fusion method involves splicing and fusing 3D point cloud features and 2D image features to form a joint feature vector. Let the dimension of the 3D point cloud features be d1 and the dimension of the 2D image features be d2, then the dimension of the fused features is d = d1 + d2.

[0032] The attention-based fusion mechanism involves introducing an attention model to assign weights to features from different modalities, making the fusion process more focused on features that significantly contribute to the measurement task. The attention weights can be automatically adjusted based on training data using a learning algorithm, as shown in the following formula:

[0033] Let the 3D point cloud feature be F 3D2D image features are F 2D Then the fused feature F fused for:

[0034] F fused =αF 3D +(1-α)F 2D ;

[0035] Here, α is the attention weight, which is learned through training the neural network and reflects the importance of different modal features in the current measurement task.

[0036] According to the above technical solution, step two specifically includes error source analysis and modeling, and error compensation algorithm;

[0037] The error source analysis and modeling include temperature drift modeling and motion fuzz modeling;

[0038] The error compensation algorithm includes Kalman filter-based error compensation and neural network-based error compensation;

[0039] The temperature drift modeling refers to the shift in camera's internal parameters with temperature changes. These internal parameters include focal length and optical center coordinates. By experimentally measuring the camera's imaging error at different temperatures, a model is established to represent the relationship between temperature and camera parameter drift. Let the focal length be f, and its variation with temperature T can be expressed as:

[0040] f(T) = f0 + k f (T-T0);

[0041] Where f0 is the focal length at the reference temperature T0, and k f This is the coefficient of focal length as a function of temperature, which can be determined through calibration experiments;

[0042] The motion blur modeling described above refers to the motion blur that occurs when there is relative motion between an object and a camera during the imaging process. A mathematical model of motion blur is established based on factors such as the camera's shutter speed, the object's speed, and the direction of relative motion. A one-dimensional motion blur model can be represented as:

[0043]

[0044] Where g(x) is the blurred image, f(u) is the original sharp image, and h is the length of the motion blur, which is related to the speed of the moving object and the exposure time.

[0045] According to the above technical solution, the error compensation based on Kalman filtering treats the drift of camera parameters and the error of motion blur as dynamic changes in the system state, and uses the Kalman filtering algorithm for real-time estimation and compensation. The prediction and update steps of Kalman filtering include the following:

[0046] Prediction steps:

[0047]

[0048] in, F is the predicted state value at the current moment. k Let P be the state transition matrix. k|k-1 For the prediction error covariance matrix, Q k The process noise covariance matrix;

[0049] Update steps:

[0050]

[0051] P k|k =(IK k H k )P k|k-1 ;

[0052] Among them, K k For Kalman gain, H k Let z be the observation matrix. k R represents the observation value at the current moment. k To measure the noise covariance matrix, This is the updated state estimate, used to compensate for camera parameter errors;

[0053] The neural network-based error compensation is achieved by constructing a neural network model, using historical data of measurement errors and relevant influencing factors as inputs, including temperature and motion speed, and training the network to learn error compensation rules.

[0054] The MLP structure is adopted. The input layer receives the error-related feature vector X = [x1, x2, ..., xn], the hidden layer performs nonlinear transformation through the activation function, and the output layer gives the compensated measurement value (Y). The activation function is ReLU.

[0055] The network is trained by minimizing the loss function to optimize the weight parameters. The loss function is the mean squared error, and the formula is as follows:

[0056]

[0057] Where m is the number of training samples. Y is the compensation value predicted by the network. i These are actual, accurate measured values.

[0058] According to the above technical solution, step three specifically includes an online calibration process and a feedback optimization mechanism;

[0059] The online calibration process includes reference object selection and placement, reference object detection and parameter calculation;

[0060] The feedback optimization mechanism includes error monitoring and evaluation, and calibration parameter update strategy;

[0061] The reference object is selected and placed in the measurement scene with a known size and shape. The reference object should have obvious features, including a standard sphere or cube, to facilitate accurate detection and identification in point clouds and images. Its position should cover different positions and angles of the measurement area to obtain comprehensive calibration data.

[0062] The reference object detection and parameter calculation include detection algorithms and parameter calculation;

[0063] The detection algorithm is a feature-based detection method that locates reference objects in 3D point clouds and 2D images. Detection methods include template matching and RANSAC. The reference objects are included in the point cloud. The point set of the reference objects is found through cluster analysis and geometric shape matching. In the image, the outline of the reference objects is determined by edge detection and shape analysis.

[0064] The parameters are calculated based on the detected features of the reference object to determine the intrinsic and extrinsic parameters of the current camera.

[0065] The intrinsic parameters, including focal length and principal axis coordinates, can be obtained by solving the projection equation and optimizing using the least squares method.

[0066] For extrinsic parameters, including rotation matrix and translation vector, calculations are performed based on the positional relationship of the reference object in the world coordinate system and camera coordinate system.

[0067] Planar reference object: Let Pw = (X,Y,Z) be a point in the world coordinate system, Pc = (x,y,z) be a point in the camera coordinate system, and p = (u,v) be the pixel coordinates projected onto the image plane. Then the projection relationship can be expressed as:

[0068]

[0069] Among them, f x f y c represents the pixel value along the focal length in the x and y directions. x c y Using the principal optical axis coordinates, and through the coordinate correspondence of multiple reference points, a system of equations is established to solve for the camera's intrinsic parameters.

[0070] According to the above technical solution, the error monitoring and evaluation involves comparing the measurement result with the actual size of the reference object during each measurement process, calculating the measurement error, analyzing the distribution and trend of the error, and determining whether the error exceeds the allowable range. For length measurement, the error can be expressed as:

[0071]

[0072] Among them, L measured For the measured length, L true The true length of the reference object is given, and e represents the relative error.

[0073] The calibration parameter update strategy triggers an update when error monitoring results indicate a decrease in measurement accuracy or an error exceeding a threshold. Based on error analysis results, the camera's intrinsic and extrinsic parameters are adjusted. If a significant deviation in focal length estimation is detected, the focal length is recalculated and the camera's intrinsic parameter matrix is ​​updated.

[0074]

[0075] At the same time, external parameters are optimized and adjusted to ensure accurate alignment between the camera coordinate system and the world coordinate system, thereby improving measurement accuracy.

[0076] According to the above technical solution, step four specifically includes GPU-accelerated point cloud processing, deep learning model fusion and optimization;

[0077] The GPU-accelerated point cloud processing includes GPU storage and transmission of point cloud data, and parallel computing algorithm design.

[0078] The parallel computing algorithm design includes voxel mesh downsampling and normal estimation;

[0079] The deep learning model fusion and optimization includes lightweight deep learning model selection, model fusion and acceleration;

[0080] The model fusion and acceleration includes model fusion strategies, model quantization and compression, and the application of GPU acceleration libraries;

[0081] The GPU storage and transmission of the point cloud data involves transferring the point cloud data from the CPU to the GPU's video memory, utilizing the GPU's parallel computing architecture for efficient processing, and employing a suitable GPU memory layout to optimize data access and computing performance. The GPU memory layout includes a structure array or an array structure.

[0082] The voxel grid downsampling is implemented on the GPU using a voxel grid downsampling algorithm. Each thread block is responsible for processing a certain range of voxel grids. Inside the thread, the points within the grid are statistically analyzed and averaged to achieve fast downsampling.

[0083] The normal estimation utilizes the parallel computing power of the GPU to perform covariance matrix calculation and eigenvalue decomposition on each point and its neighboring points, quickly estimating the normal direction for each point p. i Calculate its neighborhood point set N(p) i The covariance matrix C of ))

[0084]

[0085] in, Let C be the mean of the neighborhood points. By solving for the eigenvalues ​​and eigenvectors of C, the eigenvector corresponding to the smallest eigenvalue is the normal direction.

[0086] According to the above technical solution, the lightweight deep learning model selects the MobileNet lightweight deep learning model as the basic network for feature extraction and classification. The model uses depthwise separable convolution and channel rearrangement techniques to greatly reduce the number of parameters and computational complexity while ensuring model performance.

[0087] The model fusion strategy is to fuse 3D point cloud processing models and 2D image processing models, and through feature stitching or attention fusion, enable the fused model to make full use of the advantages of multimodal data.

[0088] The model quantization and compression are performed on the deep learning model, converting floating-point parameters into low-bit integer representations to reduce storage space and computation. At the same time, pruning techniques are used to remove redundant connections in the model, thereby improving the model's running efficiency.

[0089] The GPU acceleration library application utilizes the GPU acceleration library to optimize the training and inference process of deep learning models, fully leverages the parallel computing capabilities of GPUs, and achieves real-time processing. The GPU acceleration library includes CUDA and cuDNN.

[0090] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0091] This invention combines 3D point cloud and 2D image texture information, and uses an attention mechanism to assign weights to features of different modalities. This makes the fusion process focus more on features that make significant contributions to the measurement task, solving the problems of insufficient multimodal data fusion and insufficient robustness of feature extraction in existing technologies. This effectively improves measurement accuracy and stability. Furthermore, by using the Kalman filter algorithm to estimate and compensate for temperature drift and motion blur errors in real time, compared with existing static or semi-static calibration methods, it can dynamically adjust in real time, improving the adaptability and stability of the measurement system. It performs particularly well in complex and ever-changing industrial environments. It can automatically update calibration parameters using reference objects without stopping the system. Unlike traditional methods that require stopping the system for calibration, it can continuously optimize the measurement system without affecting the production process, ensuring measurement accuracy and improving the efficiency and continuity of industrial production. Moreover, based on GPU-accelerated point cloud processing and deep learning model fusion, it adopts lightweight deep learning models, model quantization and compression techniques, which greatly reduces computational complexity and processing time. Compared with existing processing algorithms, it can meet the real-time measurement needs of high-speed production lines, improving the practicality and efficiency of the measurement system.

[0092] In summary, this invention improves the accuracy and stability of online measurement, overcomes measurement errors caused by factors such as camera parameter drift and motion blur in existing technologies, enhances the real-time performance of the measurement system, meets the real-time measurement needs of high-speed production lines, solves the problems of excessive computational complexity and slow processing speed in existing technologies, improves the adaptability of the measurement system, enabling it to work stably in complex and ever-changing industrial environments, solves the problems of sensitivity to environmental changes and the need for frequent shutdowns for calibration in existing technologies, and can fully utilize the potential of 3D cameras to achieve efficient and accurate online measurement, thus promoting the development of industrial automation and intelligence. Attached Figure Description

[0093] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0094] In the attached diagram:

[0095] Figure 1 This is a flowchart of the online measurement steps of the present invention. Detailed Implementation

[0096] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0097] Example: Figure 1 As shown, the present invention provides a technical solution, a high-precision online measurement algorithm based on a 3D camera, comprising the following online measurement steps:

[0098] Step 1: Multimodal data fusion;

[0099] Step 2, adaptive error compensation;

[0100] Step 3: Online calibration and feedback optimization;

[0101] Step 4: Lightweight real-time processing;

[0102] Step 1: Use a 3D camera to collect 3D point cloud data and 2D image texture information of the object under test, perform data preprocessing, extract features from 3D point cloud and 2D image, and perform feature fusion through a multimodal data fusion architecture to enhance the robustness of feature extraction.

[0103] Step 2: Establish an error model and use an adaptive error compensation mechanism to dynamically calibrate the measurement error, thereby improving measurement accuracy and stability;

[0104] Step 3: The calibration parameters are automatically updated using a reference object, enabling continuous optimization of the measurement system without downtime.

[0105] Step four: Based on GPU-accelerated point cloud processing and lightweight real-time processing algorithms, efficient real-time processing of the measurement process is achieved.

[0106] Based on the above technical solution, step one specifically includes data acquisition, data preprocessing, feature extraction and fusion;

[0107] Data acquisition includes 3D point cloud data acquisition and 2D image texture information acquisition;

[0108] Data preprocessing includes 3D point cloud data preprocessing and 2D image preprocessing;

[0109] 3D point cloud data preprocessing includes filtering and denoising, downsampling, and normal estimation; 2D image preprocessing includes grayscale correction and edge detection.

[0110] Feature extraction and fusion includes feature extraction and feature fusion;

[0111] Feature extraction includes 3D point cloud feature extraction and 2D image feature extraction; feature fusion includes feature-level fusion and attention-based fusion.

[0112] 3D point cloud data acquisition involves scanning the object under test using a 3D camera. A structured light camera is selected as the 3D camera to acquire point cloud data containing depth information of the object's surface. The camera's resolution and field of view parameters are configured according to the specific measurement scenario and accuracy requirements.

[0113] The acquisition frequency of point cloud data needs to meet real-time requirements. It is generally determined based on the moving speed of objects on the production line or the timeliness of the measurement task. In high-speed production lines, the acquisition frequency is set to tens of frames per second.

[0114] 2D image texture information acquisition: Simultaneously acquire 2D images corresponding to 3D point clouds to provide texture details of objects. The resolution of the 2D images should match that of the 3D camera to ensure that the information of each pixel can be accurately matched during the fusion process.

[0115] Use appropriate lighting conditions to ensure the clarity of 2D images and the recognizability of textures, and avoid affecting the quality of texture information due to uneven lighting or overexposure.

[0116] Based on the above technical solutions, the filtering and denoising uses voxel grid filtering and statistical filtering to remove noise points in the point cloud. Voxel grid filtering divides the space into voxel grids and takes the average or median value of the points in each voxel to reduce noise interference.

[0117] Downsampling is used to reduce the data size and improve processing efficiency when the point cloud data volume is too large. Random sampling and uniform sampling strategies can be used, and key features of the object need to be preserved.

[0118] Normal estimation involves calculating the normal direction at each point to provide geometric information for subsequent feature extraction and fusion. Commonly used methods include covariance analysis based on neighborhood points.

[0119] Grayscale correction is the process of correcting the grayscale of an image. Specifically, it involves histogram equalization to enhance the contrast of the image and make texture features more prominent.

[0120] Edge detection uses Canny edge detection and Sobel operators to extract image edges, highlighting the contour information of objects and preparing for feature matching.

[0121] Based on the above technical solution, 3D point cloud feature extraction is based on the geometric shape of the point cloud, extracting curvature and principal component analysis (PCA) features. Curvature can be determined by calculating the distribution of neighboring points. Areas with large curvature indicate drastic changes on the object's surface, which may be key feature points of edges or sharp corners.

[0122] 2D image feature extraction utilizes algorithms such as SIFT and SURF to extract key points and descriptors from images. These features are robust to changes in illumination and viewpoint. SIFT stands for Scale Invariant Feature Transform, and SURF stands for Accelerated Robust Feature Transform.

[0123] Feature-level fusion involves splicing and fusing 3D point cloud features and 2D image features to form a joint feature vector. Let the dimension of the 3D point cloud features be d1 and the dimension of the 2D image features be d2, then the dimension of the fused features is d = d1 + d2.

[0124] Attention-based fusion involves introducing an attention model to assign weights to features from different modalities. This makes the fusion process focus more on features that significantly contribute to the measurement task. The attention weights can be automatically adjusted based on training data using a learning algorithm, as shown in the following formula:

[0125] Let the 3D point cloud feature be F 3D 2D image features are F 2D Then the fused feature F fused for:

[0126] F fused =αF 3D +(1-α)F 2D ;

[0127] Here, α is the attention weight, which is learned through training the neural network and reflects the importance of different modal features in the current measurement task.

[0128] Based on the above technical solution, step two specifically includes error source analysis and modeling, and error compensation algorithm.

[0129] Error source analysis and modeling include temperature drift modeling and motion fuzz modeling;

[0130] Error compensation algorithms include Kalman filter-based error compensation and neural network-based error compensation;

[0131] Temperature drift modeling refers to the shift in camera's internal parameters with temperature changes. These parameters include focal length and optical center coordinates. By experimentally measuring the camera's imaging error at different temperatures, a model is established to represent the relationship between temperature and camera parameter drift. Let the focal length be f, and its variation with temperature T can be expressed as:

[0132] f(T) = f0 + k f (T-T0);

[0133] Where f0 is the focal length at the reference temperature T0, and k f This is the coefficient of focal length as a function of temperature, which can be determined through calibration experiments;

[0134] Motion blur modeling addresses the issue of motion blur that occurs when there is relative motion between an object and a camera during the imaging process. A mathematical model of motion blur is established based on factors such as the camera's shutter speed, the object's speed, and the direction of relative motion. A one-dimensional motion blur model can be represented as:

[0135]

[0136] Where g(x) is the blurred image, f(u) is the original sharp image, and h is the length of the motion blur, which is related to the speed of the moving object and the exposure time.

[0137] Based on the above technical solution, the error compensation based on Kalman filtering treats the drift of camera parameters and the error of motion blur as dynamic changes in the system state, and uses the Kalman filtering algorithm for real-time estimation and compensation. The prediction and update steps of Kalman filtering include the following:

[0138] Prediction steps:

[0139]

[0140]

[0141] in, F is the predicted state value at the current moment. k Let P be the state transition matrix. k|k-1 For the prediction error covariance matrix, Q k The process noise covariance matrix;

[0142] Update steps:

[0143]

[0144] P k|k =(IK k H k )P k|k-1 ;

[0145] Among them, K k For Kalman gain, H k Let z be the observation matrix. k R represents the observation value at the current moment. k To measure the noise covariance matrix, This is the updated state estimate, used to compensate for camera parameter errors;

[0146] Neural network-based error compensation involves constructing a neural network model, using historical data on measurement errors and relevant influencing factors as inputs, including temperature and motion speed, and training the network to learn error compensation patterns.

[0147] The MLP structure is adopted. The input layer receives the error-related feature vector X = [x1, x2, ..., xn], the hidden layer performs nonlinear transformation through the activation function, and the output layer gives the compensated measurement value (Y). The activation function is ReLU.

[0148] The network is trained by minimizing the loss function to optimize the weight parameters. The loss function is the mean squared error, and the formula is as follows:

[0149]

[0150] Where m is the number of training samples. Y is the compensation value predicted by the network. i These are actual, accurate measured values.

[0151] Based on the above technical solution, step three specifically includes the online calibration process and the feedback optimization mechanism.

[0152] The online calibration process includes reference object selection and placement, reference object detection, and parameter calculation;

[0153] Feedback optimization mechanisms include error monitoring and evaluation, and calibration parameter update strategies;

[0154] Reference object selection and placement: Set reference objects of known size and shape in the measurement scene. The reference objects should have obvious features, including standard spheres and cubes, to facilitate accurate detection and identification in point clouds and images. Their positions should cover different positions and angles of the measurement area to obtain comprehensive calibration data.

[0155] Reference object detection and parameter calculation include detection algorithms and parameter calculation;

[0156] The detection algorithm is a feature-based detection method that locates reference objects in 3D point clouds and 2D images. Detection methods include template matching and RANSAC. The reference objects are included in the point cloud. The point set of the reference objects is found through cluster analysis and geometric shape matching. In the image, the outline of the reference objects is determined by edge detection and shape analysis.

[0157] The parameters are calculated based on the detected features of the reference object to determine the intrinsic and extrinsic parameters of the current camera.

[0158] The intrinsic parameters, including focal length and principal axis coordinates, can be obtained by solving the projection equation and optimizing using the least squares method.

[0159] For extrinsic parameters, including rotation matrix and translation vector, calculations are performed based on the positional relationship of the reference object in the world coordinate system and camera coordinate system.

[0160] Planar reference object: Let Pw = (X,Y,Z) be a point in the world coordinate system, Pc = (x,y,z) be a point in the camera coordinate system, and p = (u,v) be the pixel coordinates projected onto the image plane. Then the projection relationship can be expressed as:

[0161]

[0162] Among them, f x f y c represents the pixel value along the focal length in the x and y directions. x c y Using the principal optical axis coordinates, and through the coordinate correspondence of multiple reference points, a system of equations is established to solve for the camera's intrinsic parameters.

[0163] Based on the above technical solution, error monitoring and evaluation involves comparing the measurement results with the actual dimensions of a reference object during each measurement process, calculating the measurement error, analyzing the distribution and trend of the error, and determining whether the error exceeds the allowable range. For length measurement, the error can be expressed as:

[0164]

[0165] Among them, L measured For the measured length, L true The true length of the reference object is given, and e represents the relative error.

[0166] The calibration parameter update strategy triggers an update when error monitoring results indicate a decrease in measurement accuracy or an error exceeding a threshold. Based on error analysis results, the camera's intrinsic and extrinsic parameters are adjusted. If a significant deviation in focal length estimation is detected, the focal length is recalculated and the camera's intrinsic parameter matrix is ​​updated.

[0167]

[0168] At the same time, external parameters are optimized and adjusted to ensure accurate alignment between the camera coordinate system and the world coordinate system, thereby improving measurement accuracy.

[0169] Based on the above technical solution, step four specifically includes GPU-accelerated point cloud processing, deep learning model fusion and optimization.

[0170] GPU-accelerated point cloud processing includes GPU storage and transmission of point cloud data, and design of parallel computing algorithms;

[0171] Parallel computing algorithm design includes voxel mesh downsampling and normal estimation;

[0172] Deep learning model fusion and optimization includes lightweight deep learning model selection, model fusion, and acceleration;

[0173] Model fusion and acceleration includes model fusion strategies, model quantization and compression, and the application of GPU acceleration libraries;

[0174] GPU storage and transmission of point cloud data involves transferring point cloud data from the CPU to the GPU's video memory, utilizing the GPU's parallel computing architecture for efficient processing, and employing a suitable GPU memory layout to optimize data access and computing performance. The GPU memory layout includes structure arrays or array structures.

[0175] Voxel grid downsampling is a voxel grid downsampling algorithm implemented on the GPU. Each thread block is responsible for processing a certain range of voxel grids. Inside the thread, the points within the grid are statistically analyzed and averaged to achieve fast downsampling.

[0176] Normal estimation leverages the parallel computing power of GPUs to perform covariance matrix calculation and eigenvalue decomposition for each point and its neighboring points, quickly estimating the normal direction for each point p. i Calculate its neighborhood point set N(p) i The covariance matrix C of ))

[0177]

[0178] in, Let C be the mean of the neighborhood points. By solving for the eigenvalues ​​and eigenvectors of C, the eigenvector corresponding to the smallest eigenvalue is the normal direction.

[0179] Based on the above technical solutions, the lightweight deep learning model adopts the MobileNet lightweight deep learning model as the basic network for feature extraction and classification. The model uses depthwise separable convolution and channel rearrangement techniques to greatly reduce the number of parameters and computational complexity while ensuring model performance.

[0180] The model fusion strategy is to fuse 3D point cloud processing models and 2D image processing models, and through feature stitching or attention fusion, enable the fused model to make full use of the advantages of multimodal data.

[0181] Model quantization and compression involves quantizing deep learning models, converting floating-point parameters into low-bit integer representations to reduce storage space and computation. At the same time, pruning techniques are used to remove redundant connections in the model, improving the model's running efficiency.

[0182] GPU acceleration libraries are used to optimize the training and inference processes of deep learning models, fully leveraging the parallel computing capabilities of GPUs to achieve real-time processing. GPU acceleration libraries include CUDA and cuDNN.

[0183] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A high-precision online measurement algorithm based on a 3D camera, characterized in that: The online measurement steps include the following: Step 1: Multimodal data fusion; Step 2, adaptive error compensation; Step 3: Online calibration and feedback optimization; Step 4: Lightweight real-time processing; In step one, a 3D camera is used to collect 3D point cloud data and 2D image texture information of the object under test, data preprocessing is performed, features of 3D point cloud and 2D image are extracted, and feature fusion is performed through a multimodal data fusion architecture to enhance the robustness of feature extraction. In step two, an error model is established, and an adaptive error compensation mechanism is used to dynamically calibrate the measurement error, thereby improving measurement accuracy and stability. In step three, the calibration parameters are automatically updated using a reference object, enabling continuous optimization of the measurement system without downtime. Step four involves using GPU-accelerated point cloud processing and lightweight real-time processing algorithms to achieve efficient real-time processing of the measurement process.

2. The high-precision online measurement algorithm based on a 3D camera according to claim 1, characterized in that: Step one specifically includes data acquisition, data preprocessing, feature extraction and fusion; The data acquisition includes 3D point cloud data acquisition and 2D image texture information acquisition; The data preprocessing includes 3D point cloud data preprocessing and 2D image preprocessing. The 3D point cloud data preprocessing includes filtering and denoising, downsampling, and normal estimation; the 2D image preprocessing includes grayscale correction and edge detection. The feature extraction and fusion includes feature extraction and feature fusion; The feature extraction includes 3D point cloud feature extraction and 2D image feature extraction, and the feature fusion includes feature-level fusion and attention mechanism-based fusion. The 3D point cloud data acquisition method involves using a 3D camera to scan the object under test and obtain point cloud data containing the depth information of the object's surface. The camera's resolution and field of view parameters are configured according to the specific measurement scenario and accuracy requirements. The frequency of point cloud data acquisition needs to meet real-time requirements, and is generally determined based on the moving speed of objects on the production line or the timeliness of the measurement task. The 2D image texture information acquisition synchronously acquires 2D images corresponding to the 3D point cloud to provide texture details of the object. The resolution of the 2D image should match that of the 3D camera to ensure that the information of each pixel can be accurately matched during the fusion process. Use appropriate lighting conditions to ensure the clarity of 2D images and the recognizability of textures, and avoid affecting the quality of texture information due to uneven lighting or overexposure.

3. The high-precision online measurement algorithm based on a 3D camera according to claim 2, characterized in that: The filtering and denoising method uses voxel grid filtering and statistical filtering to remove noise points in the point cloud. The downsampling mentioned above is used to reduce the data size and improve processing efficiency when the point cloud data volume is too large. Random sampling and uniform sampling strategies can be used, and key features of the object need to be preserved. The normal estimation is to calculate the normal direction of each point, providing geometric information for subsequent feature extraction and fusion. Commonly used methods include covariance analysis based on neighborhood points. The grayscale correction refers to the grayscale correction of the image, specifically through histogram equalization, which enhances the contrast of the image and makes the texture features more obvious. The edge detection uses Canny edge detection and Sobel operator to extract image edges, highlighting the contour information of objects and preparing for feature matching.

4. The high-precision online measurement algorithm based on a 3D camera according to claim 2, characterized in that: The 3D point cloud feature extraction is based on the geometry of the point cloud, extracting curvature and principal component analysis (PCA) features; The 2D image feature extraction uses SIFT and SURF algorithms to extract key points and descriptors in the image. These features are robust to changes in illumination and viewpoint. SIFT: Scale Invariant Feature Transform, SURF: Accelerated Robust Feature Transform. The feature-level fusion method involves splicing and fusing 3D point cloud features and 2D image features to form a joint feature vector. Let the dimension of the 3D point cloud features be d1 and the dimension of the 2D image features be d2, then the dimension of the fused feature vector is d = d1 + d2. The attention-based fusion mechanism involves introducing an attention model to assign weights to features from different modalities, making the fusion process more focused on features that significantly contribute to the measurement task. The attention weights can be automatically adjusted based on training data using a learning algorithm, as shown in the following formula: Let the 3D point cloud feature be F 3D 2D image features are F 2D Then the fused feature F fused for: F fused =αF 3D +(1-α)F 2D ; Here, α is the attention weight, which is learned through training the neural network and reflects the importance of different modal features in the current measurement task.

5. The high-precision online measurement algorithm based on a 3D camera according to claim 1, characterized in that: Step two specifically includes error source analysis and modeling, and error compensation algorithm; The error source analysis and modeling include temperature drift modeling and motion fuzz modeling; The error compensation algorithm includes Kalman filter-based error compensation and neural network-based error compensation; The temperature drift modeling refers to the shift in camera's internal parameters with temperature changes. These internal parameters include focal length and optical center coordinates. By experimentally measuring the camera's imaging error at different temperatures, a model is established to represent the relationship between temperature and camera parameter drift. Let the focal length be f, and its variation with temperature T can be expressed as: f(T)=f0+k f (T-T0); Where f0 is the focal length at the reference temperature T0, and k f This is the coefficient of focal length as a function of temperature, which can be determined through calibration experiments; The motion blur modeling described above refers to the motion blur that occurs when there is relative motion between an object and a camera during the imaging process. A mathematical model of motion blur is established based on factors such as the camera's shutter speed, the object's speed, and the direction of relative motion. A one-dimensional motion blur model can be represented as: Where g(x) is the blurred image, f(u) is the original sharp image, and h is the length of the motion blur, which is related to the speed of the moving object and the exposure time.

6. The high-precision online measurement algorithm based on a 3D camera according to claim 5, characterized in that: The Kalman filter-based error compensation treats camera parameter drift and motion blur errors as dynamic changes in the system state, and uses the Kalman filter algorithm for real-time estimation and compensation. The prediction and update steps of the Kalman filter include the following: Prediction steps: in, F is the predicted state value at the current moment. k Let P be the state transition matrix. k|k-1 For the prediction error covariance matrix, Q k The process noise covariance matrix; Update steps: P k|k =(I-K k H k )P k|k-1 ; Among them, K k For Kalman gain, H k Let z be the observation matrix. k R represents the observation value at the current moment. k To measure the noise covariance matrix, This is the updated state estimate, used to compensate for camera parameter errors; The neural network-based error compensation is achieved by constructing a neural network model, using historical data of measurement errors and relevant influencing factors as inputs, including temperature and motion speed, and training the network to learn error compensation rules. The MLP structure is adopted. The input layer receives the error-related feature vector X = [x1, x2, ..., xn], the hidden layer performs nonlinear transformation through the activation function, and the output layer gives the compensated measurement value (Y). The activation function is ReLU. The network is trained by minimizing the loss function to optimize the weight parameters. The loss function is the mean squared error, and the formula is as follows: Where m is the number of training samples. Y is the compensation value predicted by the network. i These are actual, accurate measured values.

7. The high-precision online measurement algorithm based on a 3D camera according to claim 1, characterized in that: Step three specifically includes an online calibration process and a feedback optimization mechanism; The online calibration process includes reference object selection and placement, reference object detection and parameter calculation; The feedback optimization mechanism includes error monitoring and evaluation, and calibration parameter update strategy; The reference object is selected and placed in the measurement scene with a known size and shape. The reference object should have obvious features, including a standard sphere or cube, to facilitate accurate detection and identification in point clouds and images. Its position should cover different positions and angles of the measurement area to obtain comprehensive calibration data. The reference object detection and parameter calculation include detection algorithms and parameter calculation; The detection algorithm is a feature-based detection method that locates reference objects in 3D point clouds and 2D images. Detection methods include template matching and RANSAC. The reference objects are included in the point cloud. The point set of the reference objects is found through cluster analysis and geometric shape matching. In the image, the outline of the reference objects is determined by edge detection and shape analysis. The parameters are calculated based on the detected features of the reference object to determine the intrinsic and extrinsic parameters of the current camera. The intrinsic parameters, including focal length and principal axis coordinates, can be obtained by solving the projection equation and optimizing using the least squares method. For extrinsic parameters, including rotation matrix and translation vector, calculations are performed based on the positional relationship of the reference object in the world coordinate system and camera coordinate system. Planar reference object: Let Pw = (X,Y,Z) be a point in the world coordinate system, Pc = (x,y,z) be a point in the camera coordinate system, and p = (u,v) be the pixel coordinates projected onto the image plane. Then the projection relationship can be expressed as: Among them, f x f y c represents the pixel value along the focal length in the x and y directions. x c y Using the principal optical axis coordinates, and through the coordinate correspondence of multiple reference points, a system of equations is established to solve for the camera's intrinsic parameters.

8. The high-precision online measurement algorithm based on a 3D camera according to claim 7, characterized in that: The error monitoring and evaluation involves comparing the measurement results with the actual dimensions of a reference object during each measurement process, calculating the measurement error, analyzing the distribution and trend of the error, and determining whether the error exceeds the allowable range. For length measurement, the error can be expressed as: Among them, L measured For the measured length, L true The true length of the reference object is given, and e represents the relative error. The calibration parameter update strategy triggers an update when error monitoring results indicate a decrease in measurement accuracy or an error exceeding a threshold. Based on error analysis results, the camera's intrinsic and extrinsic parameters are adjusted. If a significant deviation in focal length estimation is detected, the focal length is recalculated and the camera's intrinsic parameter matrix is ​​updated. At the same time, external parameters are optimized and adjusted to ensure accurate alignment between the camera coordinate system and the world coordinate system, thereby improving measurement accuracy.

9. The high-precision online measurement algorithm based on a 3D camera according to claim 1, characterized in that: Step four specifically includes GPU-accelerated point cloud processing, deep learning model fusion and optimization; The GPU-accelerated point cloud processing includes GPU storage and transmission of point cloud data, and parallel computing algorithm design. The parallel computing algorithm design includes voxel mesh downsampling and normal estimation; The deep learning model fusion and optimization includes lightweight deep learning model selection, model fusion and acceleration; The model fusion and acceleration includes model fusion strategies, model quantization and compression, and the application of GPU acceleration libraries; The GPU storage and transmission of the point cloud data involves transferring the point cloud data from the CPU to the GPU's video memory, utilizing the GPU's parallel computing architecture for efficient processing, and employing a suitable GPU memory layout to optimize data access and computing performance. The GPU memory layout includes a structure array or an array structure. The voxel grid downsampling is implemented on the GPU using a voxel grid downsampling algorithm. Each thread block is responsible for processing a certain range of voxel grids. Inside the thread, the points within the grid are statistically analyzed and averaged to achieve fast downsampling. The normal estimation utilizes the parallel computing power of the GPU to perform covariance matrix calculation and eigenvalue decomposition on each point and its neighboring points, quickly estimating the normal direction for each point p. i Calculate its neighborhood point set N(p) i The covariance matrix C of )) in, Let C be the mean of the neighborhood points. By solving for the eigenvalues ​​and eigenvectors of C, the eigenvector corresponding to the smallest eigenvalue is the normal direction.

10. The high-precision online measurement algorithm based on a 3D camera according to claim 9, characterized in that: The lightweight deep learning model adopts the MobileNet lightweight deep learning model as the base network for feature extraction and classification. The model uses depthwise separable convolution and channel rearrangement techniques to greatly reduce the number of parameters and computational complexity while ensuring model performance. The model fusion strategy is to fuse 3D point cloud processing models and 2D image processing models, and through feature stitching or attention fusion, enable the fused model to make full use of the advantages of multimodal data. The model quantization and compression are performed on the deep learning model, converting floating-point parameters into low-bit integer representations to reduce storage space and computation. At the same time, pruning techniques are used to remove redundant connections in the model, thereby improving the model's running efficiency. The GPU acceleration library application utilizes the GPU acceleration library to optimize the training and inference process of deep learning models, fully leverages the parallel computing capabilities of GPUs, and achieves real-time processing. The GPU acceleration library includes CUDA and cuDNN.

Images