A neural network-based imu camera calibration system and method

Abstract

The application discloses an IMU camera calibration system and method based on a neural network, and the method comprises the following steps: obtaining an original IMU data set and an original calibration board picture data set by using a light transmission type AR head-mounted display; a data processing terminal processes the original IMU data set and the original calibration board picture data set to obtain a joint calibration optimization model; a test data set is obtained by using the light transmission type AR head-mounted display; the data processing terminal pre-processes the test data set to obtain a pre-processed test data set; and the data processing terminal processes the pre-processed test data set based on the joint calibration optimization model to obtain a joint calibration result. The application simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and the calibration precision, and improves the automation degree and the precision of the camera calibration.

Description

Technical Field

[0001] This invention relates to the field of AR device positioning technology, and in particular to an IMU camera calibration system and method based on neural networks. Background Technology

[0002] The fusion of camera and IMU is considered to have great potential for achieving low-cost and high-precision positioning and mapping. This is because these two sensors are complementary. An Inertial Measurement Unit (IMU) is a device composed of a three-axis accelerometer and a three-axis gyroscope that can measure the three-axis attitude angles and acceleration of a rigid object, thus determining the object's motion and changing state. Joint calibration refers to obtaining the extrinsic parameters [R,t] from the IMU coordinate system to the camera coordinate system through calibration. After joint calibration of the camera and IMU, the position information measured by IMU integration can be accurately projected into the camera image, thereby overcoming the drawback of easily losing position information during rapid movement. Conversely, environmental information in the camera image can be used for loop closure detection and correction through visual matching, effectively correcting the cumulative drift error of the IMU. Joint calibration of IMU and camera is a crucial step in using equipment or technologies such as submarines, missiles, drones, and SLAM (simultaneous localization and mapping). By measuring the output of a well-matched IMU and camera device in motion and at rest, an error model is built. The accuracy of the calibration results and the simplicity of the calibration method directly affect the accuracy and cost of subsequent work.

[0003] In AR / VR headsets, relatively inexpensive but low-precision IMUs are often used. Therefore, improving calibration accuracy is a key focus of IMU positioning in AR devices. The optimized IMU calibration method utilizes the high precision, programmability, and high degrees of freedom of a six-axis robotic arm to directly and accurately move the IMU to the corresponding position, outputting information such as IMU time, three-axis acceleration, and three-axis angular velocity. This information is then combined with an error model to calculate the accelerometer and gyroscope scale factors, axis deviation, and zero bias.

[0004] Currently, commonly used IMU calibration methods are mainly based on Kalibr, which provides position and velocity references through a precision turntable, making it highly dependent on the turntable. Due to its simple principle, mature algorithm, and diverse error modeling capabilities, hierarchical calibration has been widely applied in various situations. Its experimental principle is as follows: by fixing the rotation and translation of the device under test (DUT), and by processing the sampled images from the camera, navigation parameters such as camera position and attitude are calculated. A B-spline is then drawn using the pose of discrete points to obtain the pose at any given time. The B-spline contains the camera's displacement and rotation vectors. By processing these vectors, the camera's velocity, angular velocity, and acceleration at any given time can be obtained. The IMU can measure the angular velocity at any given time. Ignoring noise and bias effects, we can align the time delay between the camera and IMU measurements using the angular velocity. After time alignment, the correspondence between the IMU and camera B-splines can be used to obtain the rotation and translation relationships between them. Finally, Kalibr's built-in optimizer can yield relatively accurate calibration results.

[0005] While the Kalibr-based method achieves high accuracy, it also has some limitations: 1) Kalibr has a high learning curve, and users without experience may need to spend a long time learning and using it; 2) It requires a large amount of data for calibration, and the quality and accuracy of the data are highly important; 3) Data processing takes a long time and requires high computing power from the equipment. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide an IMU camera calibration system and method based on neural networks, which simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0007] To address the aforementioned technical problems, the first aspect of this invention discloses a neural network-based IMU camera calibration system, comprising: a first robotic arm, a second robotic arm, a data processing terminal, an electromechanical gripper, a light-transmitting AR head-mounted display, a checkerboard marker, a stereo vision depth camera, an IMU sensor, and a digital display level.

[0008] The first robotic arm is used to move the light-transmitting AR headset according to a set trajectory; the first end of the first robotic arm is fixedly connected to the ground, and the second end of the first robotic arm is fixedly connected to the first end of the electromechanical gripper.

[0009] The stereo vision depth camera is used to acquire image data of the checkerboard markers to obtain the original calibration board image dataset and the test calibration board image dataset; the stereo vision depth camera is connected to the data processing terminal via USB.

[0010] The electromechanical gripper is used to fix the light-transmitting AR headset; the second end of the electromechanical gripper is fixedly connected to the light-transmitting AR headset.

[0011] The light-transparent AR headset is used to receive the original IMU dataset and the test IMU dataset, and output the original IMU dataset and the test IMU dataset to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal; the light-transparent AR headset has the stereo vision depth camera and the IMU sensor built in, and the light-transparent AR headset is connected to the data processing terminal via USB.

[0012] The IMU sensor is used to collect the original IMU dataset and the test IMU dataset, and output them to the light-transmitting AR headset;

[0013] The data processing terminal is used to receive and process the original IMU dataset, the test IMU dataset, the original calibration board image dataset, and the test calibration board image dataset to obtain the joint calibration result information;

[0014] The checkerboard marker is used to provide a calibration object for the stereo vision depth camera; the front of the checkerboard marker is printed with a checkerboard pattern, and the back is fixedly connected to the second robotic arm.

[0015] The second robotic arm is used to connect and fix the checkerboard marker; the first end of the second robotic arm is fixedly connected to the ground, and the second end of the second robotic arm is fixedly connected to the back of the checkerboard marker.

[0016] The digital display level is used to provide a calibration reference for the IMU camera calibration system.

[0017] A second aspect of this invention discloses an IMU camera calibration method based on a neural network, the method comprising:

[0018] S1, Obtain the original IMU dataset and the original calibration board image dataset;

[0019] S2, process the original IMU dataset and the original calibration board image dataset to construct a joint calibration training dataset;

[0020] S3, Based on the joint calibration model, the joint calibration training dataset is processed to obtain the joint calibration optimization model;

[0021] S4, Obtain the test dataset;

[0022] The test dataset includes a test IMU dataset and a test calibration board image dataset;

[0023] S5, preprocess the test dataset to obtain a preprocessed test dataset;

[0024] S6. Based on the joint calibration optimization model, the preprocessed test dataset is processed to obtain the joint calibration result.

[0025] As an optional implementation, in the second aspect of the present invention, obtaining the original IMU dataset and the original calibration board image dataset includes:

[0026] S11, using the light-transmitting AR headset, the original IMU dataset is collected;

[0027] The original IMU dataset includes a first original IMU dataset and a second original IMU dataset;

[0028] S12, using the stereo vision depth camera, the original calibration board image dataset is acquired.

[0029] As an optional implementation, in the first aspect of the present invention, the processing of the original IMU dataset and the original calibration board image dataset to construct a joint calibration training dataset includes:

[0030] S21, preprocess the original IMU dataset and the original calibration board image dataset to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset;

[0031] S22, the preprocessed IMU dataset and the preprocessed calibration board image dataset are fused to obtain a joint calibration training dataset.

[0032] As an optional implementation, in a second aspect of the present invention, the preprocessing of the original IMU dataset and the original calibration board image dataset to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset includes:

[0033] S211, the first original IMU dataset and the second original IMU dataset are aligned and combined according to the timestamps to obtain the first IMU dataset;

[0034] S212, perform outlier removal on the first IMU dataset to obtain the second IMU dataset;

[0035] S213, using the calibration label set, the second IMU dataset is labeled to obtain a preprocessed IMU dataset;

[0036] S214, Perform distortion correction processing on the original calibration board image dataset to obtain a preprocessed calibration board image dataset.

[0037] As an optional implementation, in a second aspect of the present invention, the outlier removal process on the first IMU dataset to obtain the second IMU dataset includes:

[0038] S2121, Based on the first preprocessing model, the first IMU dataset is processed to obtain the IMU outlier dataset and the IMU normal dataset;

[0039] The first preprocessing model expression is:

[0040]

[0041] ε k =H k +v k -H k x k+1 ;

[0042] Where τ is the first IMU dataset; x is the system prediction value; H k S is the measurement matrix; S is the measurement noise matrix; k is the index of the IMU data to be processed; v k It is a Gaussian white sound sequence; The correction matrix is ​​H; Q is the system noise covariance matrix; ε is the difference between the observed and predicted values; H is the measurement matrix; [ε] k ] i α is the i-th component of the k-th difference; β β is the forgetting factor; D is the forgetting coefficient; D indicates that the extraction diagonal matrix processing is performed.

[0043] S2122, Based on the second preprocessing model, the IMU outlier dataset is processed to obtain the IMU outlier corrected dataset;

[0044] The second preprocessing model expression is:

[0045]

[0046] ε k =H k +v k -H k x k+1 ;

[0047] in, To correct the difference; μ k τ is the weighted sequence; τ is the first preprocessed IMU dataset; x is the system prediction value; H kS is the measurement matrix; S is the measurement noise matrix; k is the time when outliers occur; v k It is a Gaussian white sound sequence; The correction matrix is ​​H; Q is the system noise covariance matrix; ε is the difference between the observed and predicted values; H is the measurement matrix; [ε] k ] i α is the i-th component of the k-th difference; β β is the forgetting factor; β is the forgetting coefficient;

[0048] S2123, The IMU normal dataset and the IMU outlier correction dataset are fused to obtain the second IMU dataset.

[0049] As an optional implementation, in the first aspect of the present invention, the distortion correction processing of the original calibration board image dataset to obtain a preprocessed calibration board image dataset includes:

[0050] S2141, Based on the original calibration board image dataset, determine all the original calibration board images;

[0051] The original calibration board image dataset includes several original calibration board images;

[0052] S2142, The original calibration board image is processed into a grid to obtain a calibration grid image dataset;

[0053] The calibration grid image dataset includes several calibration grid image data;

[0054] S2143, Based on the image distortion correction processing model, perform distortion removal processing on any of the calibration grid image data to obtain a preprocessed calibration grid image;

[0055] The expression for the image distortion correction processing model is:

[0056]

[0057] Wherein, (x, y) represents the coordinate value of any pixel in the calibration grid image; (x', y') represents the coordinate value of the pixel after distortion correction; α represents the first mirror distortion correction coefficient; β represents the second mirror distortion correction coefficient; γ represents the third mirror distortion correction coefficient; and L represents the distance from the pixel to the center of the calibration board image.

[0058] S2144, perform fusion processing on all the preprocessed calibration grid images to obtain the preprocessed original calibration board image;

[0059] The expression for the fusion process is:

[0060]

[0061] g(x”,y”)=(1-α)f1(x'1,x'1)+af2(x'2,x'2)

[0062] Where S represents the grayscale contrast of the fused image; m represents the height of the fused image; n represents the width of the fused image; g(x”,y”) represents the pixel value in the x”th row and y”th column of the fused image; ω represents the mean of the fused image; α represents the image weight coefficient, that is, the degree of spectral information involved in image fusion; f1(x'1,y'1) and f2(x'2,y'2) represent the pixel values ​​of the target image to be fused.

[0063] S2145, all the preprocessed calibration board images are combined in sequence to obtain a preprocessed calibration board image dataset.

[0064] As an optional implementation, in a second aspect of the present invention, the step of processing the female data in the joint calibration training dataset based on the joint calibration model to obtain a joint calibration optimization model includes:

[0065] S31, preset precision threshold and maximum number of iterations; preset number of calculations;

[0066] Randomly initialize the first and second parameters;

[0067] S32, Based on the fitness calculation algorithm, the joint calibration training dataset is processed to obtain fitness data; the number of calculations is increased by 1;

[0068] The expression for the fitness calculation algorithm is:

[0069]

[0070] Where C represents the fitness value; M is the number of the joint calibration training datasets; K is the output dimension; Q ij The expected output for jointly calibrated training data; x ij The predicted output corresponding to the joint calibration training data;

[0071] S33, determine whether the fitness value is less than the precision threshold, and obtain the fitness determination result;

[0072] S34, when the fitness determination result is yes, execute S35;

[0073] If the fitness determination result is negative, execute S37;

[0074] S35, determine whether the number of calculations is greater than the maximum number of iterations, and obtain the iteration number determination result;

[0075] If the result of the iteration count determination is negative, execute S36;

[0076] If the iteration count determination result is yes, execute S37;

[0077] S36, Update the joint calibration model using the parameter update function to obtain the updated joint calibration model, and then execute S33;

[0078] The expression for the parameter update function is:

[0079]

[0080] in, b represents the first parameter passed from layer l to layer (l-1) of the joint calibration model; l The parameter represents the second parameter passed from layer l to layer (l-1) of the joint calibration model; l represents the layer index of the currently used joint calibration model; i represents the neuron index corresponding to layer (l-1) of the joint calibration model; j represents the neuron index corresponding to layer l of the joint calibration model; μ represents the learning rate. This represents the actual output of the k-th neuron in layer l of the joint calibration model; f() represents the layer activation function currently used with the joint calibration model; This indicates that the k-th neuron in layer l has used the layer activation function, and

[0081] S37, Obtain the updated joint calibration model to obtain the joint calibration optimization model.

[0082] As an optional implementation, in a second aspect of the present invention,

[0083] The joint calibration model includes a data input layer, a first hidden layer, a second hidden layer, a third hidden layer, and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the data output layer are sequentially connected.

[0084] The data input layer is used to receive the preprocessed joint calibration dataset and output it to the first hidden layer;

[0085] The first hidden layer is used to receive and process the preprocessed joint calibration dataset to obtain a first activation result dataset, and send the first activation result dataset to the second hidden layer;

[0086] The second hidden layer is used to receive and process the first activation result dataset to obtain the second activation result dataset, and send the second activation result dataset to the third hidden layer;

[0087] The third hidden layer is used to receive and process the second activation result dataset, obtain the activation result dataset, and send the activation result dataset to the data output layer;

[0088] The data output layer is used to receive and output the activation result dataset.

[0089] As an optional implementation, in a second aspect of the present invention, the joint calibration model expression is:

[0090]

[0091] Where q represents the number of hidden layer neurons in the joint calibration model; i represents the neuron index of the currently processed hidden layer; K represents the neuron index of the currently processed output layer; w ij Indicates the first weight; w ki θ represents the second weight. i θ represents the first activation threshold; K The second activation threshold is represented by Δ; the error is represented by μ; the learning rate is represented by μ > 0; the iteration number is represented by t, and t is less than or equal to the maximum iteration number; w ij Indicates the first weight; w ki This indicates the second weight.

[0092] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0093] 1) Compared to traditional camera calibration, which requires manual data collection and parameter adjustment, this invention does not require manual intervention and performs camera calibration completely automatically;

[0094] 2) Compared with traditional methods, the present invention achieves higher performance and provides higher efficiency in various tasks;

[0095] 3) Traditional camera calibration techniques require high-precision datasets for training. This invention can automatically learn model parameters from the data through training, thereby reducing the dependence on high-precision data.

[0096] 4) Compared to joint calibration which requires known camera intrinsic parameters, this invention can extract feature information from a small number of images from a fixed viewpoint to obtain camera intrinsic parameters, greatly simplifying the calibration process. Attached Figure Description

[0097] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0098] Figure 1 This is a schematic diagram of the composition of an IMU camera calibration system based on a neural network disclosed in an embodiment of the present invention;

[0099] Figure 2 This is a flowchart illustrating a neural network-based IMU camera calibration method disclosed in an embodiment of the present invention. Detailed Implementation

[0100] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0101] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0102] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0103] This invention discloses an IMU camera calibration system and method based on neural networks, which simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration. These are described in detail below.

[0104] Example 1

[0105] Please see Figure 1 , Figure 1 This is an IMU camera calibration system based on a neural network disclosed in an embodiment of the present invention. Figure 1 As shown, the system includes: a first robotic arm 1, a second robotic arm 2, a data processing terminal (not shown in the figure), an electric mechanical gripper 3, a light-transmitting AR head-mounted display 4, a checkerboard marker 5, a stereo vision depth camera 6, an IMU sensor (not shown in the figure), and a digital display level (not shown in the figure).

[0106] The first robotic arm 1 is used to make the light-transmitting AR head-mounted display 4 move according to a set trajectory; the first end of the first robotic arm 1 is fixedly connected to the ground, and the second end of the first robotic arm 1 is fixedly connected to the first end of the electromechanical gripper 3.

[0107] It should be noted that in this embodiment, the first robotic arm 1 is a six-axis high-precision robotic arm;

[0108] The stereo vision depth camera 6 is used to acquire image data of the checkerboard marker 5 to obtain the original calibration board image dataset and the test calibration board image dataset; the stereo vision depth camera 6 is connected to the data processing terminal (not shown in the figure) via USB.

[0109] It should be noted that the stereo vision depth camera 6 used in this embodiment is a Realsense D435i camera;

[0110] The electromechanical gripper 3 is used to fix the light-transmitting AR headset 4; the second end of the electromechanical gripper 3 is fixedly connected to the light-transmitting AR headset 4.

[0111] The light-transmitting AR headset 4 is used to receive the original IMU dataset and the test IMU dataset, and output the original IMU dataset and the test IMU dataset to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal (not shown in the figure); the light-transmitting AR headset 4 has the stereo vision depth camera 6 and the IMU sensor (not shown in the figure) built in, and the light-transmitting AR headset 4 is connected to the data processing terminal (not shown in the figure) via USB;

[0112] The IMU sensor (not shown in the figure) is used to collect the original IMU dataset and the test IMU dataset, and output them to the light-transmitting AR headset 4;

[0113] The data processing terminal (not shown in the figure) is used to receive and process the original IMU dataset, the test IMU dataset, the original calibration board image dataset, and the test calibration board image dataset to obtain the joint calibration result information;

[0114] The checkerboard marker 5 is used to provide a calibration object for the stereo vision depth camera 4; the front of the checkerboard marker 5 is printed with a checkerboard pattern, and the back is fixedly connected to the second robotic arm 2.

[0115] The second robotic arm 2 is used to connect and fix the checkerboard marker 5; the first end of the second robotic arm 2 is fixedly connected to the ground, and the second end of the second robotic arm 2 is fixedly connected to the back of the checkerboard marker.

[0116] It should be noted that in this embodiment, the second robotic arm 2 is a six-axis high-precision robotic arm;

[0117] The digital level (not shown in the figure) is used to provide a calibration reference for the IMU camera calibration system.

[0118] Example 2

[0119] Please see Figure 2 , Figure 2 This invention discloses a neural network-based IMU camera calibration method, wherein... Figure 2 The described neural network-based IMU camera calibration method is applied to the neural network-based IMU camera calibration system described in Example 1. For example... Figure 2 As shown, this neural network-based IMU camera calibration method can include the following operations:

[0120] S1, Obtain the original IMU dataset and the original calibration board image dataset;

[0121] S2, process the original IMU dataset and the original calibration board image dataset to construct a joint calibration training dataset;

[0122] S3, Based on the joint calibration model, the joint calibration training dataset is processed to obtain the joint calibration optimization model;

[0123] S4, Obtain the test dataset;

[0124] It should be noted that the test dataset includes the test IMU dataset and the test calibration board image dataset;

[0125] It should be noted that the test IMU dataset includes a first test IMU dataset and a second test IMU dataset;

[0126] It should be noted that the first test IMU dataset refers to the gyroscope data in the IMU device under test collected at various moments during the test, when the light-transmitting AR headset 4 moves according to the set trajectory.

[0127] It should be noted that the second test IMU dataset refers to the accelerometer data collected at various moments during the test when the light-transmitting AR headset 4 moves according to the set trajectory.

[0128] S5, preprocess the test dataset to obtain a preprocessed test dataset;

[0129] S6. Based on the joint calibration optimization model, the preprocessed test dataset is processed to obtain the joint calibration result.

[0130] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0131] In an optional embodiment, the step S1 above, which involves obtaining the original IMU dataset and the original calibration board image dataset, includes:

[0132] S11, using the light-transmitting AR headset 4, the original IMU dataset is collected;

[0133] It should be noted that the original IMU dataset includes the first original IMU dataset and the second original IMU dataset;

[0134] It should be noted that the first original IMU dataset refers to the IMU gyroscope data collected at various moments during the motion of the light-transmitting AR headset 4 according to the set trajectory.

[0135] It should be noted that the first original IMU dataset refers to the IMU accelerometer data collected at various moments during the process of the light-transmitting AR headset moving according to the set trajectory.

[0136] S12, using the stereo vision depth camera 6, the original calibration board image dataset is acquired;

[0137] It should be noted that the original calibration board image dataset refers to the calibration board images captured by the stereo vision depth camera 6 at different times during the motion of the light-transmitting AR headset 4 according to the set trajectory.

[0138] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0139] In another optional embodiment, the process of processing the original IMU dataset and the original calibration board image dataset in step S2 above to construct a joint calibration training dataset includes:

[0140] S21, preprocess the original IMU dataset and the original calibration board image dataset to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset;

[0141] S22, the preprocessed IMU dataset and the preprocessed calibration board image dataset are fused to obtain a joint calibration training dataset.

[0142] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0143] In another optional embodiment, the preprocessing of the original IMU dataset and the original calibration board image dataset in step S21 above to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset includes:

[0144] S211, the first original IMU dataset and the second original IMU dataset are aligned and combined according to the timestamps to obtain the first IMU dataset;

[0145] S212, perform outlier removal on the first IMU dataset to obtain the second IMU dataset;

[0146] S213, using the calibration label set, the second IMU dataset is labeled to obtain a preprocessed IMU dataset;

[0147] S214, Perform distortion correction processing on the original calibration board image dataset to obtain a preprocessed calibration board image dataset.

[0148] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0149] In another optional embodiment, the outlier removal process performed on the first IMU dataset in step S212 above to obtain the second IMU dataset includes:

[0150] S2121, Based on the first preprocessing model, the first IMU dataset is processed to obtain the IMU outlier dataset and the IMU normal dataset;

[0151] The first preprocessing model expression is:

[0152]

[0153] ε k =H k +v k -H k x k+1 ;

[0154] Where τ is the first IMU dataset; x is the system prediction value; H k S is the measurement matrix; S is the measurement noise matrix; k is the index of the IMU data to be processed; v k It is a Gaussian white sound sequence; The correction matrix is ​​H; Q is the system noise covariance matrix; ε is the difference between the observed and predicted values; H is the measurement matrix; [ε] k ] i α is the i-th component of the k-th difference; β β is the forgetting factor; D is the forgetting coefficient; D indicates that the extraction diagonal matrix processing is performed.

[0155] S2122, Based on the second preprocessing model, the IMU outlier dataset is processed to obtain the IMU outlier corrected dataset;

[0156] The second preprocessing model expression is:

[0157]

[0158] ε k =H k +v k -H k x k+1 ;

[0159] in, To correct the difference; μ k τ is the weighted sequence; τ is the first preprocessed IMU dataset; x is the system prediction value; H k S is the measurement matrix; S is the measurement noise matrix; k is the time when outliers occur; v k It is a Gaussian white sound sequence; The correction matrix is ​​H; Q is the system noise covariance matrix; ε is the difference between the observed and predicted values; H is the measurement matrix; [ε] k ] i α is the i-th component of the k-th difference;β β is the forgetting factor; β is the forgetting coefficient;

[0160] S2123, The IMU normal dataset and the IMU outlier correction dataset are fused to obtain the second IMU dataset;

[0161] It should be noted that the fusion process refers to sorting and combining the elements in chronological order.

[0162] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0163] In an optional embodiment, the distortion correction processing of the original calibration board image dataset in step S214 above to obtain a preprocessed calibration board image dataset includes:

[0164] S2141, Based on the original calibration board image dataset, determine all the original calibration board images;

[0165] The original calibration board image dataset includes several original calibration board images;

[0166] S2142, The original calibration board image is processed into a grid to obtain a calibration grid image dataset;

[0167] The calibration grid image dataset includes several calibration grid image data;

[0168] S2143, Based on the image distortion correction processing model, perform distortion removal processing on any of the calibration grid image data to obtain a preprocessed calibration grid image;

[0169] The expression for the image distortion correction processing model is:

[0170]

[0171] Wherein, (x, y) represents the coordinate value of any pixel in the calibration grid image; (x', y') represents the coordinate value of the pixel after distortion correction; α represents the first mirror distortion correction coefficient; β represents the second mirror distortion correction coefficient; γ represents the third mirror distortion correction coefficient; and L represents the distance from the pixel to the center of the calibration board image.

[0172] S2144, perform fusion processing on all the preprocessed calibration grid images to obtain the preprocessed original calibration board image;

[0173] The expression for the fusion process is:

[0174]

[0175] g(x”,y”)=(1-α)f1(x'1,x'1)+af2(x'2,x'2)

[0176] Where S represents the grayscale contrast of the fused image; m represents the height of the fused image; n represents the width of the fused image; g(x”,y”) represents the pixel value in the x”th row and y”th column of the fused image; ω represents the mean of the fused image; α represents the image weight coefficient, that is, the degree of spectral information involved in image fusion; f1(x'1,y'1) and f2(x'2,y'2) represent the pixel values ​​of the target image to be fused.

[0177] S2145, all the preprocessed calibration board images are combined in sequence to obtain a preprocessed calibration board image dataset.

[0178] As can be seen, implementing the neural network-based IMU camera calibration system and method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0179] In another optional embodiment, the step S3 above, which involves processing the joint calibration training dataset based on the joint calibration model to obtain a joint calibration optimization model, includes:

[0180] S31, preset precision threshold and maximum number of iterations; preset number of calculations;

[0181] Randomly initialize the first and second parameters;

[0182] S32, Based on the fitness calculation algorithm, process the joint calibration training dataset to obtain fitness data; increase the number of calculations by 1;

[0183] The expression for the fitness calculation algorithm is:

[0184]

[0185] Where C represents the fitness value; M is the number of the joint calibration training datasets; K is the output dimension; Q ij The expected output for jointly calibrated training data; x ij The predicted output corresponding to the joint calibration training data;

[0186] S33, determine whether the fitness value is less than the precision threshold, and obtain the fitness determination result;

[0187] S34, when the fitness determination result is yes, execute S35;

[0188] If the fitness determination result is negative, execute S37;

[0189] S35, determine whether the number of calculations is greater than the maximum number of iterations, and obtain the iteration count determination result;

[0190] If the result of the iteration count determination is negative, execute S36;

[0191] If the iteration count determination result is yes, execute S37;

[0192] S36, Update the joint calibration model using the parameter update function to obtain the updated joint calibration model, and then execute S33;

[0193] The parameter update function expression is:

[0194]

[0195] in, b represents the first parameter passed from layer l to layer (l-1) of the joint calibration model; l The parameter represents the second parameter passed from layer l to layer (l-1) of the joint calibration model; l represents the layer index of the currently used joint calibration model; i represents the neuron index corresponding to layer (l-1) of the joint calibration model; j represents the neuron index corresponding to layer l of the joint calibration model; μ represents the learning rate. This represents the actual output of the k-th neuron in layer l of the joint calibration model; f() represents the layer activation function currently used with the joint calibration model; This indicates that the k-th neuron in layer l has used the layer activation function, and

[0196] S37, Obtain the updated joint calibration model to obtain the joint calibration optimization model.

[0197] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0198] In another optional embodiment, the joint calibration model includes a data input layer, a first hidden layer, a second hidden layer, a third hidden layer, and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the data output layer are sequentially connected in data.

[0199] The data input layer is used to receive the preprocessed joint calibration dataset and output it to the first hidden layer;

[0200] The first hidden layer is used to receive and process the preprocessed joint calibration dataset to obtain a first activation result dataset, and send the first activation result dataset to the second hidden layer;

[0201] The second hidden layer is used to receive and process the first activation result dataset to obtain the second activation result dataset, and send the second activation result dataset to the third hidden layer;

[0202] The third hidden layer is used to receive and process the second activation result dataset, obtain the activation result dataset, and send the activation result dataset to the data output layer;

[0203] The data output layer is used to receive and output the activation result dataset.

[0204] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0205] In another optional embodiment, the joint calibration model expression is:

[0206]

[0207] Where q represents the number of hidden layer neurons in the joint calibration model; i represents the neuron index of the currently processed hidden layer; K represents the neuron index of the currently processed output layer; w ij Indicates the first weight; w ki θ represents the second weight. i θ represents the first activation threshold; K The second activation threshold is represented by Δ; the error is represented by μ; the learning rate is represented by μ > 0; the iteration number is represented by t, and t is less than or equal to the maximum iteration number; w ij Indicates the first weight; w ki This indicates the second weight.

[0208] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0209] In another optional embodiment, step S5 above, the preprocessing of the test dataset to obtain a preprocessed test dataset, includes:

[0210] S51, the first test IMU dataset and the second test IMU dataset are aligned and combined according to the timestamp to obtain the first test IMU dataset;

[0211] S52, perform outlier removal on the first test IMU dataset to obtain the second test IMU dataset;

[0212] It should be noted that the process of removing outliers from the first test IMU dataset to obtain the second test IMU dataset is consistent with S212 above, and will not be described again here;

[0213] S53, using the calibration label set, the second test IMU dataset is labeled to obtain a preprocessed test IMU dataset;

[0214] S54, Perform distortion correction processing on the test calibration board image dataset to obtain a preprocessed test calibration board image dataset;

[0215] It should be noted that the method for performing distortion correction processing on the test calibration board image dataset to obtain the preprocessed test calibration board image dataset is the same as that in S214 above, and will not be described here again;

[0216] S55, the preprocessed test IMU dataset and the preprocessed test calibration board image dataset are combined in chronological order to obtain the preprocessed test dataset.

[0217] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0218] In another optional embodiment, step S6 above, which involves processing the preprocessed test dataset based on the joint calibration optimization model to obtain the joint calibration result, includes:

[0219] S61, using the joint calibration optimization model, the preprocessed test dataset is processed to obtain the error value of the IMU device under test;

[0220] S62, using the error value, the tested IMU device is corrected to obtain a joint calibration result.

[0221] As can be seen, implementing the neural network-based IMU camera calibration method described in the embodiments of the present invention simplifies the calibration process, reduces calibration costs, improves calibration efficiency and accuracy, and enhances the automation and accuracy of camera calibration.

[0222] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0223] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0224] Finally, it should be noted that the IMU camera calibration system and method based on neural networks disclosed in the embodiments of the present invention are limited; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A neural network-based IMU camera calibration system, characterized in that, The system includes: a first robotic arm, a second robotic arm, a data processing terminal, an electromechanical gripper, a light-transmitting AR head-mounted display, checkerboard markers, a stereo vision depth camera, an IMU sensor, and a digital display level. The first robotic arm is used to move the light-transmitting AR headset according to a set trajectory; the first end of the first robotic arm is fixedly connected to the ground, and the second end of the first robotic arm is fixedly connected to the first end of the electromechanical gripper. The stereo vision depth camera is used to acquire image data of the checkerboard markers to obtain the original calibration board image dataset and the test calibration board image dataset; the stereo vision depth camera is connected to the data processing terminal via USB. The electromechanical gripper is used to fix the light-transmitting AR headset; the second end of the electromechanical gripper is fixedly connected to the light-transmitting AR headset. The light-transparent AR headset is used to receive the original IMU dataset and the test IMU dataset, and output the original IMU dataset and the test IMU dataset to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal; the light-transparent AR headset has the stereo vision depth camera and the IMU sensor built in, and the light-transparent AR headset is connected to the data processing terminal via USB. The IMU sensor is used to collect the original IMU dataset and the test IMU dataset, and output them to the light-transmitting AR headset; The data processing terminal is used to receive and process the original IMU dataset, the test IMU dataset, the original calibration board image dataset, and the test calibration board image dataset to obtain the joint calibration result information; The checkerboard marker is used to provide a calibration object for the stereo vision depth camera; the front of the checkerboard marker is printed with a checkerboard pattern, and the back is fixedly connected to the second robotic arm. The second robotic arm is used to connect and fix the checkerboard marker; the first end of the second robotic arm is fixedly connected to the ground, and the second end of the second robotic arm is fixedly connected to the back of the checkerboard marker. The digital display level is used to provide a calibration reference for the IMU camera calibration system; The data processing terminal processes the original IMU dataset, the test IMU dataset, the original calibration board image dataset, and the test calibration board image dataset in the following manner to obtain the joint calibration result information: The original IMU dataset and the original calibration board image dataset are processed to construct a joint calibration training dataset; Based on the joint calibration model, the joint calibration training dataset is processed to obtain the joint calibration optimization model; The test IMU dataset and the test calibration board image dataset are preprocessed to obtain a preprocessed test dataset; Based on the joint calibration optimization model, the preprocessed test dataset is processed to obtain the joint calibration results; The step of processing the joint calibration training dataset based on the joint calibration model to obtain the joint calibration optimization model includes: S31, preset precision threshold and maximum number of iterations; preset number of calculations; Randomly initialize the first and second parameters; S32, Based on the fitness calculation algorithm, process the joint calibration training dataset to obtain fitness data; increase the number of calculations by 1; The expression for the fitness calculation algorithm is: ； in, This represents the fitness value; The number of the joint calibration training datasets; For output dimensions; The expected output for jointly calibrated training data; The predicted output corresponding to the joint calibration training data; S33, determine whether the fitness value is less than the precision threshold, and obtain the fitness determination result; S34, when the fitness determination result is yes, execute S35; If the fitness determination result is negative, execute S37; S35, determine whether the number of calculations is greater than the maximum number of iterations, and obtain the iteration count determination result; If the result of the iteration count determination is negative, execute S36; If the iteration count determination result is yes, execute S37; S36, Update the joint calibration model using the parameter update function to obtain the updated joint calibration model, and then execute S33; The expression for the parameter update function is: ； ； in, The joint calibration model is represented by the first... Layer to the first The first parameter of the layer; The joint calibration model is represented by the first... Layer to the first The second parameter of the layer; This indicates the layer index of the currently used joint calibration model; The joint calibration model represents The neuron index corresponding to the layer; The joint calibration model represents The neuron index corresponding to the layer; Indicates the learning rate; The joint calibration model represents Layer The actual output of each neuron; This indicates the layer activation function currently used in the joint calibration model; Indicates the first The layer has used the layer activation function for the th time. one neuron, and ; S37, Obtain the updated joint calibration model to obtain the joint calibration optimization model.

2. A neural network-based IMU camera calibration method, characterized in that, The system applied to the neural network-based IMU camera calibration system of claim 1 includes: S1, Obtain the original IMU dataset and the original calibration board image dataset; S2, process the original IMU dataset and the original calibration board image dataset to construct a joint calibration training dataset; S3, Based on the joint calibration model, the joint calibration training dataset is processed to obtain the joint calibration optimization model; S4, Obtain the test dataset; The test dataset includes a test IMU dataset and a test calibration board image dataset; S5, preprocess the test dataset to obtain a preprocessed test dataset; S6. Based on the joint calibration optimization model, the preprocessed test dataset is processed to obtain the joint calibration result; The step of processing the joint calibration training dataset based on the joint calibration model to obtain the joint calibration optimization model includes: S31, preset precision threshold and maximum number of iterations; preset number of calculations; Randomly initialize the first and second parameters; S32, Based on the fitness calculation algorithm, process the joint calibration training dataset to obtain fitness data; increase the number of calculations by 1; The expression for the fitness calculation algorithm is: ； in, This represents the fitness value; The number of the joint calibration training datasets; For output dimensions; The expected output for jointly calibrated training data; The predicted output corresponding to the joint calibration training data; S33, determine whether the fitness value is less than the precision threshold, and obtain the fitness determination result; S34, when the fitness determination result is yes, execute S35; If the fitness determination result is negative, execute S37; S35, determine whether the number of calculations is greater than the maximum number of iterations, and obtain the iteration count determination result; If the result of the iteration count determination is negative, execute S36; If the iteration count determination result is yes, execute S37; S36, Update the joint calibration model using the parameter update function to obtain the updated joint calibration model, and then execute S33; The expression for the parameter update function is: ； ； in, The joint calibration model is represented by the first... Layer to the first The first parameter of the layer; The joint calibration model is represented by the first... Layer to the first The second parameter of the layer; This indicates the layer index of the currently used joint calibration model; The joint calibration model represents The neuron index corresponding to the layer; The joint calibration model represents The neuron index corresponding to the layer; Indicates the learning rate; The joint calibration model represents Layer The actual output of each neuron; This indicates the layer activation function currently used in the joint calibration model; Indicates the first The layer has used the layer activation function for the th time. one neuron, and ; S37, Obtain the updated joint calibration model to obtain the joint calibration optimization model.

3. The IMU camera calibration method based on neural networks according to claim 2, characterized in that, The acquisition of the original IMU dataset and the original calibration board image dataset includes: S11, using the light-transmitting AR headset, the original IMU dataset is collected; The original IMU dataset includes a first original IMU dataset and a second original IMU dataset; S12, using the stereo vision depth camera, the original calibration board image dataset is acquired.

4. The IMU camera calibration method based on neural networks according to claim 3, characterized in that, The process of processing the original IMU dataset and the original calibration board image dataset to construct a joint calibration training dataset includes: S21, preprocess the original IMU dataset and the original calibration board image dataset to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset; S22, the preprocessed IMU dataset and the preprocessed calibration board image dataset are fused to obtain a joint calibration training dataset.

5. The IMU camera calibration method based on neural networks according to claim 4, characterized in that, The preprocessing of the original IMU dataset and the original calibration board image dataset to obtain a preprocessed IMU dataset and a preprocessed calibration board image dataset includes: S211, the first original IMU dataset and the second original IMU dataset are aligned and combined according to the timestamps to obtain the first IMU dataset; S212, perform outlier removal on the first IMU dataset to obtain the second IMU dataset; S213, using the calibration label set, the second IMU dataset is labeled to obtain a preprocessed IMU dataset; S214, Perform distortion correction processing on the original calibration board image dataset to obtain a preprocessed calibration board image dataset.

6. The IMU camera calibration method based on neural networks according to claim 5, characterized in that, The process of removing outliers from the first IMU dataset to obtain the second IMU dataset includes: S2121, Based on the first preprocessing model, the first IMU dataset is processed to obtain the IMU outlier dataset and the IMU normal dataset; The first preprocessing model expression is: ； ； ； in, This is the first IMU dataset; This is the system's predicted value; To measure the noise matrix; An index for the IMU data to be processed; It is a Gaussian white sound sequence; This is the correction matrix; The system noise covariance matrix; The difference between the observed value and the predicted value; For measurement matrix; For the first The difference of the first One component; Forgetting factor; Forgetting factor; This indicates that diagonal matrix extraction is being performed; S2122, Based on the second preprocessing model, the IMU outlier dataset is processed to obtain the IMU outlier corrected dataset; The second preprocessing model expression is: ； ； ； in, To correct the difference; It is a weighted sequence; This is the IMU outlier dataset; This is the system's predicted value; To measure the noise matrix; This refers to the moment when a wild value appears; It is a Gaussian white sound sequence; This is the correction matrix; The system noise covariance matrix; The difference between the observed value and the predicted value; For measurement matrix; For the first The difference of the first One component; S2123, The IMU normal dataset and the IMU outlier correction dataset are fused to obtain the second IMU dataset.

7. The IMU camera calibration method based on neural networks according to claim 5, characterized in that, The distortion correction process performed on the original calibration board image dataset to obtain a preprocessed calibration board image dataset includes: S2141, Based on the original calibration board image dataset, determine all the original calibration board images; The original calibration board image dataset includes several original calibration board images; S2142, The original calibration board image is processed into a grid to obtain a calibration grid image dataset; The calibration grid image dataset includes several calibration grid image data; S2143, Based on the image distortion correction processing model, perform distortion removal processing on any of the calibration grid image data to obtain a preprocessed calibration grid image; The expression for the image distortion correction processing model is: ； in, This represents the coordinate value of any pixel in the calibration grid image; This represents the coordinate value of the pixel after distortion correction; Indicates the first image distortion correction coefficient; Indicates the second image distortion correction coefficient; Indicates the third image distortion correction coefficient; This represents the distance from the pixel to the center of the calibration board image; S2144, perform fusion processing on all the preprocessed calibration grid images to obtain the preprocessed original calibration board image; The expression for the fusion process is: ； ； Where S represents the gray-level contrast of the fused image; Indicates the height of the fused image; This represents the width of the fused image; Indicates the first in the fused image Line number The pixel values ​​of the column; This represents the mean of the fused image; The image weight coefficient represents the degree of spectral information involved in image fusion. , This represents the pixel values ​​of the target image to be fused; S2145, all the preprocessed original calibration board images are combined in sequence to obtain a preprocessed calibration board image dataset.

8. The IMU camera calibration method based on neural networks according to claim 2, characterized in that, The joint calibration model includes a data input layer, a first hidden layer, a second hidden layer, a third hidden layer, and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer, and the data output layer are sequentially connected. The data input layer is used to receive the preprocessed joint calibration dataset and output it to the first hidden layer; The first hidden layer is used to receive and process the preprocessed joint calibration dataset to obtain a first activation result dataset, and send the first activation result dataset to the second hidden layer; The second hidden layer is used to receive and process the first activation result dataset to obtain the second activation result dataset, and send the second activation result dataset to the third hidden layer; The third hidden layer is used to receive and process the second activation result dataset, obtain the activation result dataset, and send the activation result dataset to the data output layer; The data output layer is used to receive and output the activation result dataset.

9. The IMU camera calibration method based on neural networks according to claim 2, characterized in that, The joint calibration model expression is: ； ； ； in, This indicates the number of hidden layer neurons in the joint calibration model; This represents the neuron index of the currently processed hidden layer; This indicates the neuron index of the output layer currently being processed; Indicates the first weight; Indicates the second weight; Indicates the first activation threshold; Indicates the second activation threshold; Indicates error; Indicates the learning rate, and ; Indicates the number of iterations, and Less than or equal to the maximum number of iterations.

Images