IMU camera calibration system and method based on neural network

Abstract

The invention discloses an IMU camera calibration system and method based on a neural network. The method comprises the following steps: acquiring an original IMU data set and an original calibration board picture data set by using a light transmission type AR head-mounted display; the data processing terminal is used for processing the original IMU data set and the original calibration board picture data set to obtain a joint calibration optimization model; acquiring a test data set by using the light transmission type AR head-mounted display; the data processing terminal preprocesses the test data set to obtain a preprocessed test data set; and the data processing terminal processes the preprocessing test data set based on the joint calibration optimization model to obtain a joint calibration result. According to the invention, the calibration process is simplified, the calibration cost is reduced, the calibration efficiency and the calibration precision are improved, and the automation degree and the precision of camera calibration are improved.

Description

Technical Field

[0001] The present invention relates to the field of AR device positioning technology, and in particular to an IMU camera calibration system and method based on a neural network. Background Art

[0002] The fusion of camera and IMU is considered to have great potential in achieving low-cost and high-precision positioning and mapping. This is because the two sensors are complementary. The inertial measurement unit (IMU) is a device composed of a three-axis accelerometer and a three-axis gyroscope that can measure the three-axis attitude angle and acceleration of a rigid object, and can determine the motion state and change state of the object. Joint calibration refers to the external parameters [R, t] obtained by calibrating the imu coordinate system to the camera coordinate system. After the camera and imu are jointly calibrated, the position information measurement value obtained by the imu integration can be accurately projected into the camera image, thereby achieving the disadvantage of easy loss of position information in fast motion. On the contrary, the environmental information in the camera image can complete loop detection and loop correction through visual matching, thereby effectively correcting the accumulated drift error of the IMU. IMU and camera joint calibration is a crucial step in the use of equipment or technologies such as submarines, missiles, drones, SLAM (simultaneous localization and mapping). By measuring the output of a set of well-coordinated IMU and camera devices when they are moving and stationary, the error model is modeled. The accuracy of the calibration results and the simplicity of the calibration method directly affect the accuracy and cost of subsequent work results.

[0003] In A / VR head-mounted displays, relatively cheap and low-precision IMUs are often used, so improving calibration accuracy is the focus of IMU positioning work for AR devices. The method of IMU calibration with optimization is to use the high-precision, programmable, and high-degree-of-freedom characteristics of the six-axis robotic arm to directly and accurately move the IMU device to the corresponding position, output IMU time, three-axis acceleration, three-axis angular velocity and other information, and then combine the error model to solve the accelerometer and gyroscope scale factor, axis deviation, and zero bias.

[0004] The commonly used IMU calibration method is mainly based on Kalibr, which provides position and velocity references through a precision turntable and is highly dependent on the turntable. Due to its simple principle, mature algorithm and diverse error modeling, hierarchical calibration has been used in many occasions. The experimental principle is: through the fixed rotation and translation of the device to be tested, the camera position, attitude and other navigation parameters are solved by solving the camera sampling pictures, and the B-spline is drawn through the posture of discrete points to obtain the posture at any time. The B-spline contains the displacement and rotation vector of the camera. The speed, angular velocity and acceleration of the camera at any time can be obtained through the solution. The IMU can measure the angular velocity at any time, ignoring the influence of noise and bias. We can align the time delay between the camera and the imu measurement through the angular velocity. After aligning the time, through the corresponding relationship between the imu and the camera's b-spline, we can get the rotation and translation relationship between the two. Finally, a more accurate calibration result can be obtained through the built-in optimizer of Kalibr.

[0005] Although the kalibr-based method achieves high accuracy, it also has some limitations: 1) Kalibr has a high threshold for use, and users with no usage experience may need to spend a long time learning and using it; 2) A large amount of data is required for calibration, and the quality and accuracy of the data are high; 3) It takes a long time to solve the data, and the computing power of the equipment is high. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide an IMU camera calibration system and method based on a neural network, which simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and enhances the automation and accuracy of camera calibration.

[0007] In order to solve the above technical problems, the first aspect of the embodiment of the present invention discloses an IMU camera calibration system based on a neural network, the system comprising: a first mechanical arm, a second mechanical arm, a data processing terminal, an electric mechanical gripper, a light-transmitting AR head display, a checkerboard marker, a stereoscopic vision depth camera, an IMU sensor and a digital level ruler;

[0008] The first mechanical arm is used to make the light-transmitting AR head display move according to a set trajectory; the first end of the first mechanical arm is fixedly connected to the ground, and the second end of the first mechanical arm is fixedly connected to the first end of the electric mechanical gripper;

[0009] The stereoscopic vision depth camera is used to collect the chessboard marker image data to obtain the original calibration plate image data set and the test calibration plate image data set; the stereoscopic vision depth camera is connected to the data processing terminal via USB;

[0010] The electric mechanical clamp is used to fix the light-transmitting AR head display; the second end of the electric mechanical clamp is fixedly connected to the light-transmitting AR head display;

[0011] The light-transmitting AR head display is used to receive the original IMU data set and the test IMU data set, output the original IMU data set and the test IMU data set to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal; the light-transmitting AR head display has the stereoscopic vision depth camera and the IMU sensor built in, and the light-transmitting AR head display is connected to the data processing terminal via USB;

[0012] The IMU sensor is used to collect the original IMU data set and the test IMU data set, and output them to the light-transmitting AR head display;

[0013] The data processing terminal is used to receive and process the original IMU data set, the test IMU data set, the original calibration plate image data set and the test calibration plate image data set to obtain the joint calibration result information;

[0014] The checkerboard marker is used to provide a calibration object for the stereoscopic vision depth camera; the front side of the checkerboard marker is printed with a checkerboard, and the back side is fixedly connected to the second mechanical arm;

[0015] The second mechanical arm is used to connect and fix the checkerboard marker; the first end of the second mechanical arm is fixedly connected to the ground, and the second end of the second mechanical arm is fixedly connected to the back of the checkerboard marker;

[0016] The digital level ruler is used to provide a calibration reference for the IMU camera calibration system.

[0017] A second aspect of an embodiment of the present invention discloses an IMU camera calibration method based on a neural network, the method comprising:

[0018] S1, obtain the original IMU data set and the original calibration plate image data set;

[0019] S2, processing the original IMU data set and the original calibration plate image data set to construct a joint calibration training data set;

[0020] S3, based on the joint calibration model, processing the joint calibration training data set to obtain a joint calibration optimization model;

[0021] S4, obtain the test data set;

[0022] The test data set includes a test IMU data set and a test calibration plate image data set;

[0023] S5, preprocessing the test data set to obtain a preprocessed test data set;

[0024] S6, based on the joint calibration optimization model, processing the preprocessed test data set to obtain a joint calibration result.

[0025] As an optional implementation, in the second aspect of the embodiment of the present invention, the obtaining of the original IMU data set and the original calibration plate image data set includes:

[0026] S11, using the light-transmitting AR head display, collecting an original IMU data set;

[0027] The original IMU data set includes a first original IMU data set and a second original IMU data set;

[0028] S12, using the stereoscopic vision depth camera to collect and obtain an original calibration plate image data set.

[0029] As an optional implementation, in the first aspect of the embodiment of the present invention, the processing of the original IMU data set and the original calibration plate image data set to construct a joint calibration training data set includes:

[0030] S21, preprocessing the original IMU data set and the original calibration plate image data set to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set;

[0031] S22, fusing the preprocessed IMU data set and the preprocessed calibration plate image data set to obtain a joint calibration training data set.

[0032] As an optional implementation, in the second aspect of the embodiment of the present invention, the preprocessing of the original IMU data set and the original calibration plate image data set to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set includes:

[0033] S211, aligning and combining the first original IMU dataset and the second original IMU dataset according to timestamps to obtain a first IMU dataset;

[0034] S212, performing outlier removal processing on the first IMU data set to obtain a second IMU data set;

[0035] S213, using the calibration label set, labeling the second IMU data set to obtain a preprocessed IMU data set;

[0036] S214, performing distortion correction processing on the original calibration plate image data set to obtain a preprocessed calibration plate image data set.

[0037] As an optional implementation, in the second aspect of the embodiment of the present invention, performing outlier processing on the first IMU data set to obtain the second IMU data set includes:

[0038] S2121, based on a first preprocessing model, processing the first IMU data set to obtain an IMU outlier data set and an IMU normal data set;

[0039] The first preprocessing model expression is:

[0040]

[0041] ε k =H k +v k -H k x k+1 ;

[0042] Wherein, τ is the first IMU data set; x is the system prediction value; H k is the measurement matrix; S is the measurement noise matrix; k is the index of the IMU data to be processed; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; α β is the forgetting factor; β is the forgetting coefficient; D means extracting the diagonal matrix;

[0043] S2122, based on the second preprocessing model, processing the IMU wild value data set to obtain an IMU wild value corrected data set;

[0044] The second preprocessing model expression is:

[0045]

[0046] ε k =H k +v k -H k x k+1 ;

[0047] in, is the correction difference; μ k is a weighted sequence; τ is the first preprocessed IMU data set; x is the system prediction value; H kis the measurement matrix; S is the measurement noise matrix; k is the time when the outlier appears; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; α β is the forgetting factor; β is the forgetting coefficient;

[0048] S2123, fusing the IMU normal data set and the IMU wild value correction data set to obtain a second IMU data set.

[0049] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the performing distortion correction processing on the original calibration plate image dataset to obtain the preprocessed calibration plate image dataset includes:

[0050] S2141, determining all original calibration plate images based on the original calibration plate image data set;

[0051] The original calibration plate image data set includes a plurality of original calibration plate images;

[0052] S2142, gridding the original calibration plate image to obtain a calibration grid image data set;

[0053] The calibration square picture data set includes a plurality of calibration square picture data;

[0054] S2143, based on the image distortion correction processing model, dedistorting any of the calibration grid image data to obtain a preprocessed calibration grid image;

[0055] The image distortion correction processing model expression is:

[0056]

[0057] Wherein, (x, y) represents the coordinate value of any pixel point of the calibration grid image; (x', y') represents the coordinate value of the pixel point after distortion correction; α represents the first mirror distortion correction coefficient; β represents the second mirror distortion correction coefficient; γ represents the third mirror distortion correction coefficient; L represents the distance from the pixel point to the center of the calibration plate image;

[0058] S2144, performing fusion processing on all the pre-processed calibration grid images to obtain a pre-processed original calibration plate image;

[0059] The expression of the fusion process is:

[0060]

[0061] g(x”,y”)=(1-α)f1(x'1,x'1)+af2(x'2,x'2)

[0062] Wherein, S represents the grayscale contrast of the fused image; m represents the height of the fused image; n represents the width of the fused image; g(x",y") represents the pixel value of the x"th row and y"th column in the fused image; ω represents the mean value of the fused image; α represents the image weight coefficient, which represents the degree of spectral information involved in image fusion; f1(x'1,y'1) and f2(x'2,y'2) represent the pixel values ​​of the target images to be fused;

[0063] S2145, all the pre-processed calibration plate images are combined in order to obtain a pre-processed calibration plate image data set.

[0064] As an optional implementation, in the second aspect of the embodiment of the present invention, the joint calibration model is based on, the female part of the joint calibration training data set is processed to obtain the joint calibration optimization model, including:

[0065] S31, preset the accuracy threshold and the maximum number of iterations; preset the number of calculations;

[0066] Randomly initialize the first parameter and the second parameter;

[0067] S32, based on the fitness calculation algorithm, processing the joint calibration training data set to obtain fitness data; increasing the number of calculations by 1;

[0068] The expression of the fitness calculation algorithm is:

[0069]

[0070] Wherein, C represents the fitness value; M is the number of the joint calibration training data sets; K is the output dimension; Q ij is the expected output of the joint calibration training data; x ij is the predicted output corresponding to the joint calibration training data;

[0071] S33, determining whether the fitness value is less than the accuracy threshold, and obtaining a fitness determination result;

[0072] S34, when the fitness determination result is yes, execute S35;

[0073] When the fitness determination result is no, executing S37;

[0074] S35, determining whether the number of calculations is greater than the maximum number of iterations, and obtaining a result of determining the number of iterations;

[0075] When the result of the iteration number determination is no, executing S36;

[0076] When the result of the iteration number determination is yes, executing S37;

[0077] S36, updating the joint calibration model using the parameter update function to obtain an updated joint calibration model, and executing S33;

[0078] The parameter update function expression is:

[0079]

[0080] in, represents the first parameter of the joint calibration model from the lth layer to the l-1th layer; b l represents the second parameter transmitted from the lth layer to the l-1th layer of the joint calibration model; l represents the layer index of the joint calibration model currently used; i represents the neuron index corresponding to the l-1th layer of the joint calibration model; j represents the neuron index corresponding to the lth layer of the joint calibration model; μ represents the learning rate; represents the actual output of the kth neuron in layer l of the joint calibration model; f() represents the layer activation function currently used in the joint calibration model; indicates that the kth neuron in the lth layer has used the layer activation function, and

[0081] S37, acquiring the updated joint calibration model to obtain a joint calibration optimization model.

[0082] As an optional implementation, in the second aspect of the embodiment of the present invention,

[0083] The joint calibration model comprises a data input layer, a first hidden layer, a second hidden layer, a third hidden layer and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer and the data output layer are sequentially data-connected;

[0084] The data input layer is used to receive the preprocessed joint calibration data set and output it to the first hidden layer;

[0085] The first hidden layer is used to receive and process the preprocessed joint calibration data set to obtain a first activation result data set, and send the first activation result data set to the second hidden layer;

[0086] The second hidden layer is used to receive and process the first activation result data set to obtain a second activation result data set, and send the second activation result data set to the third hidden layer;

[0087] The third hidden layer is used to receive and process the second activation result data set to obtain an activation result data set, and send the activation result data set to the data output layer;

[0088] The data output layer is used to receive and output the activation result data set.

[0089] As an optional implementation, in the second aspect of the embodiment of the present invention, the joint calibration model expression is:

[0090]

[0091] Wherein, q represents the number of neurons in the hidden layer of the joint calibration model; i represents the neuron index of the hidden layer currently being processed; K represents the neuron index of the output layer currently being processed; w ij represents the first weight; w ki represents the second weight; θ i represents the first activation threshold; θ K represents the second activation threshold; Δ represents the error; μ represents the learning rate, and μ>0; t represents the number of iterations, and t is less than or equal to the maximum number of iterations; w ij represents the first weight; w ki Represents the second weight.

[0092] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0093] 1) Compared with the traditional camera calibration which requires manual data collection and parameter adjustment, the present invention does not require manual intervention and performs camera calibration completely automatically;

[0094] 2) Compared with traditional methods, the present invention achieves higher performance on various tasks and provides higher efficiency;

[0095] 3) Traditional camera calibration technology requires high-precision data sets for training. The present invention can automatically learn model parameters from data through training, thereby reducing dependence on high-precision data;

[0096] 4) Compared with the joint calibration which requires the camera intrinsic parameters to be known, the present invention can extract feature information from a small number of images with a fixed viewing angle to obtain the camera intrinsic parameters, which greatly simplifies the calibration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0098] Figure 1 It is a schematic diagram of the composition of an IMU camera calibration system based on a neural network disclosed in an embodiment of the present invention;

[0099] Figure 2 It is a flowchart of a neural network-based IMU camera calibration method disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0100] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0101] The terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, device, product or equipment that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units that are inherent to these processes, methods, products or equipment.

[0102] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0103] The present invention discloses an IMU camera calibration system and method based on a neural network, which simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and enhances the automation and accuracy of camera calibration. The following are detailed descriptions.

[0104] Embodiment 1

[0105] See also Figure 1 , Figure 1 The invention discloses an IMU camera calibration system based on a neural network. Figure 1 As shown, the system includes: a first robotic arm 1, a second robotic arm 2, a data processing terminal (not shown in the figure), an electric mechanical gripper 3, a light-transmitting AR head display 4, a checkerboard marker 5, a stereoscopic vision depth camera 6, an IMU sensor (not shown in the figure) and a digital level ruler (not shown in the figure);

[0106] The first mechanical arm 1 is used to make the light-transmitting AR head display 4 move according to a set trajectory; the first end of the first mechanical arm 1 is fixedly connected to the ground, and the second end of the first mechanical arm 1 is fixedly connected to the first end of the electric mechanical gripper 3;

[0107] It should be noted that in this embodiment, the first mechanical arm 1 is a six-axis high-precision mechanical arm;

[0108] The stereoscopic vision depth camera 6 is used to collect the image data of the checkerboard marker 5 to obtain the original calibration plate image data set and the test calibration plate image data set; the stereoscopic vision depth camera 6 is connected to the data processing terminal (not shown in the figure) via USB;

[0109] It should be noted that the stereoscopic vision depth camera 6 used in this embodiment is a Realsense D435i camera;

[0110] The electric mechanical clamp 3 is used to fix the light-transmitting AR head display 4; the second end of the electric mechanical clamp 3 is fixedly connected to the light-transmitting AR head display 4;

[0111] The light-transmitting AR head display 4 is used to receive the original IMU data set and the test IMU data set, and output the original IMU data set and the test IMU data set to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal (not marked in the figure); the light-transmitting AR head display 4 has the stereoscopic vision depth camera 6 and the IMU sensor (not marked in the figure) built in, and the light-transmitting AR head display 4 is connected to the data processing terminal (not marked in the figure) via USB;

[0112] The IMU sensor (not shown in the figure) is used to collect the original IMU data set and the test IMU data set, and output them to the light-transmitting AR head display 4;

[0113] The data processing terminal (not shown in the figure) is used to receive and process the original IMU data set, the test IMU data set, the original calibration plate image data set and the test calibration plate image data set to obtain the joint calibration result information;

[0114] The checkerboard marker 5 is used to provide a calibration object for the stereoscopic vision depth camera 4; the front of the checkerboard marker 5 is printed with a checkerboard, and the back is fixedly connected to the second mechanical arm 2;

[0115] The second mechanical arm 2 is used to connect and fix the checkerboard marker 5; the first end of the second mechanical arm 2 is fixedly connected to the ground, and the second end of the second mechanical arm 2 is fixedly connected to the back of the checkerboard marker;

[0116] It should be noted that in this embodiment, the second mechanical arm 2 is a six-axis high-precision mechanical arm;

[0117] The digital level ruler (not shown in the figure) is used to provide a calibration reference for the IMU camera calibration system.

[0118] Embodiment 2

[0119] See also Figure 2 , Figure 2 An IMU camera calibration method based on a neural network is disclosed in an embodiment of the present invention, wherein: Figure 2 The described neural network-based IMU camera calibration method is applied to the neural network-based IMU camera calibration system described in Example 1. Figure 2 As shown, the neural network-based IMU camera calibration method may include the following operations:

[0120] S1, obtain the original IMU data set and the original calibration plate image data set;

[0121] S2, processing the original IMU data set and the original calibration plate image data set to construct a joint calibration training data set;

[0122] S3, based on the joint calibration model, processing the joint calibration training data set to obtain a joint calibration optimization model;

[0123] S4, obtain the test data set;

[0124] It should be noted that the test data set includes a test IMU data set and a test calibration plate image data set;

[0125] It should be noted that the test IMU data set includes a first test IMU data set and a second test IMU data set;

[0126] It should be noted that the first test IMU data set refers to the gyroscope data in the tested IMU device at each moment collected when the light-transmitting AR head display 4 moves according to the set trajectory during the test;

[0127] It should be noted that the second test IMU data set refers to the accelerometer data in the tested IMU device collected at each moment during the test when the light-transmitting AR head display 4 moves according to the set trajectory;

[0128] S5, preprocessing the test data set to obtain a preprocessed test data set;

[0129] S6, based on the joint calibration optimization model, processing the preprocessed test data set to obtain a joint calibration result.

[0130] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0131] In an optional embodiment, the step of obtaining the original IMU data set and the original calibration plate image data set in the above step S1 includes:

[0132] S11, using the light-transmitting AR head display 4, collecting and obtaining an original IMU data set;

[0133] It should be noted that the original IMU data set includes a first original IMU data set and a second original IMU data set;

[0134] It should be noted that the first original IMU data set refers to the IMU gyroscope data collected at each moment when the light-transmitting AR head display 4 moves according to the set trajectory;

[0135] It should be noted that the first original IMU data set refers to the IMU accelerometer data collected at each moment when the light-transmitting AR head display moves according to the set trajectory;

[0136] S12, using the stereoscopic vision depth camera 6 to collect and obtain an original calibration plate image data set;

[0137] It should be noted that the original calibration plate image data set refers to the calibration plate images taken by the stereoscopic vision depth camera 6 at different times when the light-transmitting AR head display 4 moves according to the set trajectory.

[0138] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0139] In another optional embodiment, the processing of the original IMU data set and the original calibration plate image data set in the above step S2 to construct a joint calibration training data set includes:

[0140] S21, preprocessing the original IMU data set and the original calibration plate image data set to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set;

[0141] S22, fusing the preprocessed IMU data set and the preprocessed calibration plate image data set to obtain a joint calibration training data set.

[0142] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0143] In yet another optional embodiment, the preprocessing of the original IMU data set and the original calibration plate image data set in the above step S21 to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set includes:

[0144] S211, aligning and combining the first original IMU dataset and the second original IMU dataset according to timestamps to obtain a first IMU dataset;

[0145] S212, performing outlier removal processing on the first IMU data set to obtain a second IMU data set;

[0146] S213, using the calibration label set, labeling the second IMU data set to obtain a preprocessed IMU data set;

[0147] S214, performing distortion correction processing on the original calibration plate image data set to obtain a preprocessed calibration plate image data set.

[0148] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0149] In another optional embodiment, the step S212 described above of performing outlier processing on the first IMU data set to obtain the second IMU data set includes:

[0150] S2121, based on a first preprocessing model, processing the first IMU data set to obtain an IMU outlier data set and an IMU normal data set;

[0151] The first preprocessing model expression is:

[0152]

[0153] ε k =H k +v k -H k x k+1 ;

[0154] Wherein, τ is the first IMU data set; x is the system prediction value; H k is the measurement matrix; S is the measurement noise matrix; k is the index of the IMU data to be processed; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; α β is the forgetting factor; β is the forgetting coefficient; D means extracting the diagonal matrix;

[0155] S2122, based on the second preprocessing model, processing the IMU wild value data set to obtain an IMU wild value corrected data set;

[0156] The second preprocessing model expression is:

[0157]

[0158] ε k =H k +v k -H k x k+1 ;

[0159] in, is the correction difference; μ k is a weighted sequence; τ is the first preprocessed IMU data set; x is the system prediction value; H k is the measurement matrix; S is the measurement noise matrix; k is the time when the outlier appears; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; αβ is the forgetting factor; β is the forgetting coefficient;

[0160] S2123, fusing the IMU normal data set and the IMU wild value correction data set to obtain a second IMU data set;

[0161] It should be noted that the fusion processing refers to sorting and combining in chronological order.

[0162] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0163] In an optional embodiment, the step S214 described above performs distortion correction processing on the original calibration plate image dataset to obtain a preprocessed calibration plate image dataset, including:

[0164] S2141, determining all original calibration plate images based on the original calibration plate image data set;

[0165] The original calibration plate image data set includes a plurality of original calibration plate images;

[0166] S2142, gridding the original calibration plate image to obtain a calibration grid image data set;

[0167] The calibration square picture data set includes a plurality of calibration square picture data;

[0168] S2143, based on the image distortion correction processing model, dedistorting any of the calibration grid image data to obtain a preprocessed calibration grid image;

[0169] The image distortion correction processing model expression is:

[0170]

[0171] Wherein, (x, y) represents the coordinate value of any pixel point of the calibration grid image; (x', y') represents the coordinate value of the pixel point after distortion correction; α represents the first mirror distortion correction coefficient; β represents the second mirror distortion correction coefficient; γ represents the third mirror distortion correction coefficient; L represents the distance from the pixel point to the center of the calibration plate image;

[0172] S2144, performing fusion processing on all the pre-processed calibration grid images to obtain a pre-processed original calibration plate image;

[0173] The expression of the fusion process is:

[0174]

[0175] g(x”,y”)=(1-α)f1(x'1,x'1)+af2(x'2,x'2)

[0176] Wherein, S represents the grayscale contrast of the fused image; m represents the height of the fused image; n represents the width of the fused image; g(x",y") represents the pixel value of the x"th row and y"th column in the fused image; ω represents the mean value of the fused image; α represents the image weight coefficient, which represents the degree of spectral information involved in image fusion; f1(x'1,y'1) and f2(x'2,y'2) represent the pixel values ​​of the target images to be fused;

[0177] S2145, all the pre-processed calibration plate images are combined in order to obtain a pre-processed calibration plate image data set.

[0178] It can be seen that the implementation of the neural network-based IMU camera calibration system and method described in the embodiments of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0179] In another optional embodiment, the step S3 described above processes the joint calibration training data set based on the joint calibration model to obtain the joint calibration optimization model, including:

[0180] S31, preset the accuracy threshold and the maximum number of iterations; preset the number of calculations;

[0181] Randomly initialize the first parameter and the second parameter;

[0182] S32, based on the fitness calculation algorithm, processing the joint calibration training data set to obtain fitness data; increasing the number of calculations by 1;

[0183] The expression of the fitness calculation algorithm is:

[0184]

[0185] Wherein, C represents the fitness value; M is the number of the joint calibration training data sets; K is the output dimension; Q ij is the expected output of the joint calibration training data; x ij is the predicted output corresponding to the joint calibration training data;

[0186] S33, determining whether the fitness value is less than the accuracy threshold, and obtaining a fitness determination result;

[0187] S34, when the fitness determination result is yes, execute S35;

[0188] When the fitness determination result is no, executing S37;

[0189] S35, determining whether the number of calculations is greater than the maximum number of iterations, and obtaining a result of determining the number of iterations;

[0190] When the result of the iteration number determination is no, executing S36;

[0191] When the result of the iteration number determination is yes, executing S37;

[0192] S36, updating the joint calibration model using the parameter update function to obtain an updated joint calibration model, and executing S33;

[0193] The parameter update function expression is:

[0194]

[0195] in, represents the first parameter of the joint calibration model from the lth layer to the l-1th layer; b l represents the second parameter transmitted from the lth layer to the l-1th layer of the joint calibration model; l represents the layer index of the joint calibration model currently used; i represents the neuron index corresponding to the l-1th layer of the joint calibration model; j represents the neuron index corresponding to the lth layer of the joint calibration model; μ represents the learning rate; represents the actual output of the kth neuron in layer l of the joint calibration model; f() represents the layer activation function currently used in the joint calibration model; indicates that the kth neuron in the lth layer has used the layer activation function, and

[0196] S37, acquiring the updated joint calibration model to obtain a joint calibration optimization model.

[0197] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0198] In another optional embodiment, the joint calibration model includes a data input layer, a first hidden layer, a second hidden layer, a third hidden layer and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer and the data output layer are sequentially data connected;

[0199] The data input layer is used to receive the preprocessed joint calibration data set and output it to the first hidden layer;

[0200] The first hidden layer is used to receive and process the preprocessed joint calibration data set to obtain a first activation result data set, and send the first activation result data set to the second hidden layer;

[0201] The second hidden layer is used to receive and process the first activation result data set to obtain a second activation result data set, and send the second activation result data set to the third hidden layer;

[0202] The third hidden layer is used to receive and process the second activation result data set to obtain an activation result data set, and send the activation result data set to the data output layer;

[0203] The data output layer is used to receive and output the activation result data set.

[0204] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0205] In another optional embodiment, the joint calibration model expression is:

[0206]

[0207] Wherein, q represents the number of neurons in the hidden layer of the joint calibration model; i represents the neuron index of the hidden layer currently being processed; K represents the neuron index of the output layer currently being processed; w ij represents the first weight; w ki represents the second weight; θ i represents the first activation threshold; θ K represents the second activation threshold; Δ represents the error; μ represents the learning rate, and μ>0; t represents the number of iterations, and t is less than or equal to the maximum number of iterations; w ij represents the first weight; w ki Represents the second weight.

[0208] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0209] In another optional embodiment, in the above step S5, preprocessing the test data set to obtain a preprocessed test data set includes:

[0210] S51, aligning and combining the first test IMU data set and the second test IMU data set according to timestamps to obtain a first test IMU data set;

[0211] S52, performing outlier removal processing on the first test IMU data set to obtain a second test IMU data set;

[0212] It should be noted that the step of performing outlier removal processing on the first test IMU data set to obtain the second test IMU data set is consistent with the above S212 and will not be described again here.

[0213] S53, using the calibration label set, labeling the second test IMU data set to obtain a preprocessed test IMU data set;

[0214] S54, performing distortion correction processing on the test calibration plate image data set to obtain a preprocessed test calibration plate image data set;

[0215] It should be noted that the method of performing distortion correction processing on the test calibration plate image data set to obtain the pre-processed test calibration plate image data set is consistent with the above S214 and will not be described here again;

[0216] S55, combining the preprocessed test IMU data set and the preprocessed test calibration plate image data set in chronological order to obtain a preprocessed test data set.

[0217] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0218] In another optional embodiment, in the above step S6, the pre-processed test data set is processed based on the joint calibration optimization model to obtain a joint calibration result, including:

[0219] S61, using the joint calibration optimization model, processing the preprocessed test data set to obtain an error value of the IMU device under test;

[0220] S62, using the error value, correcting the IMU device under test to obtain a joint calibration result.

[0221] It can be seen that the implementation of the neural network-based IMU camera calibration method described in the embodiment of the present invention simplifies the calibration process, reduces the calibration cost, improves the calibration efficiency and calibration accuracy, and improves the automation and accuracy of camera calibration.

[0222] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0223] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0224] Finally, it should be noted that: the neural network-based IMU camera calibration system and method disclosed in the embodiments of the present invention are limited; although the present invention has been described in detail with reference to the aforementioned embodiments, ordinary technicians in the field should understand that it is still possible to modify the technical solutions recorded in the aforementioned embodiments, or to replace some of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An IMU camera calibration system based on a neural network, characterized in that: The system includes: a first robotic arm, a second robotic arm, a data processing terminal, an electric mechanical gripper, a light-transmitting AR head display, a checkerboard marker, a stereoscopic depth camera, an IMU sensor, and a digital level; The first mechanical arm is used to make the light-transmitting AR head display move according to a set trajectory; the first end of the first mechanical arm is fixedly connected to the ground, and the second end of the first mechanical arm is fixedly connected to the first end of the electric mechanical gripper; The stereoscopic vision depth camera is used to collect the chessboard marker image data to obtain the original calibration plate image data set and the test calibration plate image data set; the stereoscopic vision depth camera is connected to the data processing terminal via USB; The electric mechanical clamp is used to fix the light-transmitting AR head display; the second end of the electric mechanical clamp is fixedly connected to the light-transmitting AR head display; The light-transmitting AR head display is used to receive the original IMU data set and the test IMU data set, output the original IMU data set and the test IMU data set to the data processing terminal, and receive the joint calibration result information fed back by the data processing terminal; the light-transmitting AR head display has the stereoscopic vision depth camera and the IMU sensor built in, and the light-transmitting AR head display is connected to the data processing terminal via USB; The IMU sensor is used to collect the original IMU data set and the test IMU data set, and output them to the light-transmitting AR head display; The data processing terminal is used to receive and process the original IMU data set, the test IMU data set, the original calibration plate image data set and the test calibration plate image data set to obtain the joint calibration result information; The checkerboard marker is used to provide a calibration object for the stereoscopic vision depth camera; the front side of the checkerboard marker is printed with a checkerboard, and the back side is fixedly connected to the second mechanical arm; The second mechanical arm is used to connect and fix the checkerboard marker; the first end of the second mechanical arm is fixedly connected to the ground, and the second end of the second mechanical arm is fixedly connected to the back of the checkerboard marker; The digital level ruler is used to provide a calibration reference for the IMU camera calibration system.

2. A neural network-based IMU camera calibration method, characterized in that: The neural network-based IMU camera calibration system applied to claim 1 comprises: S1, obtain the original IMU data set and the original calibration plate image data set; S2, processing the original IMU data set and the original calibration plate image data set to construct a joint calibration training data set; S3, based on the joint calibration model, processing the joint calibration training data set to obtain a joint calibration optimization model; S4, obtain the test data set; The test data set includes a test IMU data set and a test calibration plate image data set; S5, preprocessing the test data set to obtain a preprocessed test data set; S6, based on the joint calibration optimization model, processing the preprocessed test data set to obtain a joint calibration result.

3. The IMU camera calibration method based on neural network according to claim 2, characterized in that: The obtaining of the original IMU data set and the original calibration plate image data set includes: S11, using the light-transmitting AR head display, collecting an original IMU data set; The original IMU data set includes a first original IMU data set and a second original IMU data set; S12, using the stereoscopic vision depth camera to collect and obtain an original calibration plate image data set.

4. The IMU camera calibration method based on neural network according to claim 2, characterized in that: The processing of the original IMU data set and the original calibration plate image data set to construct a joint calibration training data set includes: S21, preprocessing the original IMU data set and the original calibration plate image data set to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set; S22, fusing the preprocessed IMU data set and the preprocessed calibration plate image data set to obtain a joint calibration training data set.

5. The IMU camera calibration method based on neural network according to claim 4, characterized in that: The preprocessing of the original IMU data set and the original calibration plate image data set to obtain a preprocessed IMU data set and a preprocessed calibration plate image data set includes: S211, aligning and combining the first original IMU dataset and the second original IMU dataset according to timestamps to obtain a first IMU dataset; S212, performing outlier removal processing on the first IMU data set to obtain a second IMU data set; S213, using the calibration label set, labeling the second IMU data set to obtain a preprocessed IMU data set: S214, performing distortion correction processing on the original calibration plate image data set to obtain a preprocessed calibration plate image data set.

6. The IMU camera calibration method based on neural network according to claim 5, characterized in that: The step of performing outlier removal processing on the first IMU data set to obtain a second IMU data set includes: S2121, based on a first preprocessing model, processing the first IMU data set to obtain an IMU outlier data set and an IMU normal data set; The first preprocessing model expression is: ε k =H k +v k -H k x k+1 ; Wherein, τ is the first IMU data set; x is the system prediction value; H k is the measurement matrix; S is the measurement noise matrix; k is the index of the IMU data to be processed; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; α β is the forgetting factor; β is the forgetting coefficient; D means extracting the diagonal matrix; S2122, based on the second preprocessing model, processing the IMU wild value data set to obtain an IMU wild value corrected data set; The second preprocessing model expression is: ε k =H k +v k -H k x k+1 ; in, is the correction difference; μ k is a weighted sequence; τ is the first preprocessed IMU data set; x is the system prediction value; H k is the measurement matrix; S is the measurement noise matrix; k is the time when the outlier appears; v k is a Gaussian white sound sequence; is the correction matrix; Q is the system noise covariance matrix; ε is the difference between the observed value and the predicted value; H is the measurement matrix; [ε k ] i is the i-th component of the k-th difference; α β is the forgetting factor; β is the forgetting coefficient; S2123, fusing the IMU normal data set and the IMU wild value correction data set to obtain a second IMU data set.

7. The IMU camera calibration method based on neural network according to claim 5, characterized in that: The step of performing distortion correction processing on the original calibration plate image data set to obtain a pre-processed calibration plate image data set includes: S2141, determining all original calibration plate images based on the original calibration plate image data set; The original calibration plate image data set includes a plurality of original calibration plate images; S2142, gridding the original calibration plate image to obtain a calibration grid image data set; The calibration square picture data set includes a plurality of calibration square picture data; S2143, based on the image distortion correction processing model, dedistorting any of the calibration grid image data to obtain a preprocessed calibration grid image; The image distortion correction processing model expression is: Wherein, (x, y) represents the coordinate value of any pixel point of the calibration grid image; (x', y') represents the coordinate value of the pixel point after distortion correction; α represents the first mirror distortion correction coefficient; β represents the second mirror distortion correction coefficient; γ represents the third mirror distortion correction coefficient; L represents the distance from the pixel point to the center of the calibration plate image; S2144, performing fusion processing on all the pre-processed calibration grid images to obtain a pre-processed original calibration plate image; The expression of the fusion process is: g(x”,y”)=(1-α)f1(x'1,y'1)+af2(x'2,y'2) Wherein, S represents the grayscale contrast of the fused image; m represents the height of the fused image; n represents the width of the fused image; g(x", y") represents the pixel value of the x"th row and y"th column in the fused image; ω represents the mean value of the fused image; α represents the image weight coefficient, which represents the degree of spectral information involved in image fusion; f1(x'1, y'1) and f2(x'2, y'2) represent the pixel values ​​of the target images to be fused; S2145, all the pre-processed calibration plate images are combined in order to obtain a pre-processed calibration plate image data set.

8. The IMU camera calibration method based on neural network according to claim 2, characterized in that: The method of processing the joint calibration training data set based on the joint calibration model to obtain a joint calibration optimization model includes: S31, preset the accuracy threshold and the maximum number of iterations; preset the number of calculations; Randomly initialize the first parameter and the second parameter; S32, based on the fitness calculation algorithm, processing the joint calibration training data set to obtain fitness data; increasing the number of calculations by 1; The expression of the fitness calculation algorithm is: Wherein, C represents the fitness value; M is the number of the joint calibration training data sets; K is the output dimension; Q ij is the expected output of the joint calibration training data; x ij is the predicted output corresponding to the joint calibration training data; S33, determining whether the fitness value is less than the accuracy threshold, and obtaining a fitness determination result; S34, when the fitness determination result is yes, execute S35; When the fitness determination result is no, executing S37; S35, determining whether the number of calculations is greater than the maximum number of iterations, and obtaining a result of determining the number of iterations; When the result of the iteration number determination is no, executing S36; When the result of the iteration number determination is yes, executing S37; S36, updating the joint calibration model using the parameter update function to obtain an updated joint calibration model, and executing S33; The parameter update function expression is: in, represents the first parameter of the joint calibration model from the lth layer to the l-1th layer; b l represents the second parameter transmitted from the lth layer to the l-1th layer of the joint calibration model; l represents the layer index of the joint calibration model currently used; i represents the neuron index corresponding to the l-1th layer of the joint calibration model; j represents the neuron index corresponding to the lth layer of the joint calibration model; μ represents the learning rate; represents the actual output of the kth neuron in layer l of the joint calibration model; f() represents the layer activation function currently used in the joint calibration model; indicates that the kth neuron in the lth layer has used the layer activation function, and S37, acquiring the updated joint calibration model to obtain a joint calibration optimization model.

9. The IMU camera calibration method based on neural network according to claim 8, characterized in that: The joint calibration model comprises a data input layer, a first hidden layer, a second hidden layer, a third hidden layer and a data output layer; the data input layer, the first hidden layer, the second hidden layer, the third hidden layer and the data output layer are sequentially data-connected; The data input layer is used to receive the preprocessed joint calibration data set and output it to the first hidden layer; The first hidden layer is used to receive and process the preprocessed joint calibration data set to obtain a first activation result data set, and send the first activation result data set to the second hidden layer; The second hidden layer is used to receive and process the first activation result data set to obtain a second activation result data set, and send the second activation result data set to the third hidden layer; The third hidden layer is used to receive and process the second activation result data set to obtain an activation result data set, and send the activation result data set to the data output layer; The data output layer is used to receive and output the activation result data set.

10. The IMU camera calibration method based on neural network according to claim 8, characterized in that: The joint calibration model expression is: Wherein, q represents the number of neurons in the hidden layer of the joint calibration model; i represents the neuron index of the hidden layer currently being processed; K represents the neuron index of the output layer currently being processed; w ij represents the first weight; w ki represents the second weight; θ i represents the first activation threshold; θ K represents the second activation threshold; Δ represents the error; μ represents the learning rate, and μ>0; t represents the number of iterations, and t is less than or equal to the maximum number of iterations; w ij represents the first weight; w ki Represents the second weight.

Images