A camera space triangulation data processing method and device

Abstract

The application discloses a camera space triangulation data processing method and device, acquires a plurality of feature point pairs of a calibration scene; according to the plurality of feature point pairs, a model optimization operation is performed on a multi-layer perception machine model, after error value convergence, model parameters of the multi-layer perception machine model are output as triangulation parameters of a camera to be calibrated; and the model parameters of the multi-layer perception machine model are optimized through a nonlinear least square optimization tool. Through the multi-layer perception machine model, distorted two-dimensional image data is nonlinearly converted into undistorted two-dimensional image data, triangulation three-dimensional data obtained based on the undistorted two-dimensional data realizes nonlinear transformation of camera triangulation transformation; through a loss function constructed between the triangulation three-dimensional data and three-dimensional point cloud data, model parameter adjustment is performed, the accuracy of triangulation parameter output is improved, and then the camera triangulation precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a camera space triangulation data processing method and device. Background Art

[0002] With the increasing maturity of drone and photographic imaging technologies, non-contact photogrammetry and 3D reconstruction based on sequential images are becoming widely used in industry. Currently, the key factor limiting the accuracy of photogrammetry and 3D reconstruction techniques is the choice of triangulation method. Different factors affect the triangulation results.

[0003] The most widely used method in the photogrammetry and 3D image reconstruction industries is linear triangulation. This method uses a pinhole camera model, obtains camera intrinsic parameters and distortion coefficients through pre-calibration, transforms the coordinates of image feature points into a normalized camera coordinate system, and then uses linear triangulation to solve for the coordinates of the 3D points. This method calibrates the camera projection matrix separately and, to simplify calculations, does not introduce nonlinear factors during the triangulation process. This method does not consider the nonlinear transformation of a distorted 2D image into a dedistorted 2D image, which can lead to reduced camera triangulation accuracy.

[0004] Therefore, a camera space triangulation data processing strategy is urgently needed to solve the problem of low camera triangulation accuracy. Summary of the Invention

[0005] Embodiments of the present invention provide a camera space triangulation data processing method and device to solve the problem of low camera triangulation accuracy.

[0006] To solve the above problem, an embodiment of the present invention provides a camera space triangulation data processing method, comprising:

[0007] Acquire several feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is captured by the camera to be calibrated in the calibration scene;

[0008] Based on a number of feature point pairs, the multi-layer perceptron model is optimized. After the error value converges, the model parameters of the multi-layer perceptron model are output as the triangulation parameters of the camera to be calibrated.

[0009] The model optimization operation is specifically as follows:

[0010] Selecting a target feature point pair from the feature point pairs that have not undergone the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, and inputting the distorted two-dimensional image data into a multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0011] inputting the distortion-free two-dimensional image data into a triangulation calculation matrix to obtain triangulated three-dimensional data;

[0012] substituting the triangulated three-dimensional data corresponding to the two-dimensional image data in the target feature point pair and the three-dimensional point cloud data into a loss function, calculating an error value, and taking error value convergence as the target, optimizing the model parameters of the multi-layer perception model through a nonlinear least squares optimization tool.

[0013] As an improvement of the above scheme, the multi-layer perception model satisfies the following conditions:

[0014]

[0015] In the formula, x is the value of the x-axis of the distortion-free two-dimensional image data, y is the value of the y-axis of the distortion-free two-dimensional image data, x distorted is the value of the x-axis of the distorted two-dimensional image data, y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer respectively, and w2 and b2 are the change parameter and bias of the first linear layer respectively.

[0016] As an improvement of the above scheme, the distortion of the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data comprises:

[0017] inputting the two-dimensional image data in the target feature point pair into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0018]

[0019] In the formula, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0020] As an improvement of the above scheme, the triangulation calculation matrix satisfies the following conditions:

[0021]

[0022] In the formula, P and Q are triangulated three-dimensional data, u l ,v l respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l is an element in the left view projection matrix, and mr Elements of the right view projection matrix.

[0023] As an improvement to the above solution, the loss function includes:

[0024]

[0025] In the formula, loss is the error value, X i 、Y i and Z i is the value of the x-axis, y-axis and z-axis of the i-th triangulated 3D data, _X i ,_Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0026] Accordingly, an embodiment of the present invention further provides a camera space triangulation data processing device, comprising: a data acquisition module and a model optimization module;

[0027] The data acquisition module is used to acquire a plurality of feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is captured by the camera to be calibrated in the calibration scene;

[0028] The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges;

[0029] The model optimization operation is specifically as follows:

[0030] Selecting a target feature point pair from the feature point pairs that have not undergone the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, and inputting the distorted two-dimensional image data into a multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0031] Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data;

[0032] The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and with the error value convergence as the goal, the model parameters of the multi-layer perceptron model are optimized through the nonlinear least squares optimization tool.

[0033] As an improvement to the above solution, the multi-layer perceptron model satisfies the following conditions:

[0034]

[0035] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer, respectively.

[0036] As an improvement to the above solution, distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0037] The two-dimensional image data of the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0038]

[0039] Where k1, k2, and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0040] As an improvement to the above solution, the triangulated calculation matrix satisfies the following conditions:

[0041]

[0042] Where P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements in the left view projection matrix, m r Elements of the right view projection matrix.

[0043] As an improvement to the above solution, the loss function includes:

[0044]

[0045] In the formula, loss is the error value, X i 、Y i and Z i is the value of the x-axis, y-axis and z-axis of the i-th triangulated 3D data, _X i ,_Y i and _Zi are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0046] As can be seen from the above, the present invention has the following beneficial effects:

[0047] The present invention provides a camera space triangulation data processing method, which obtains a plurality of feature point pairs of a calibration scene; performs a model optimization operation on a multi-layer perceptron model according to the plurality of feature point pairs, and outputs the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges; wherein the model optimization operation is specifically as follows: a target feature point pair is selected from the feature point pairs for which the model optimization operation has not been performed, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data; the dedistorted two-dimensional image data is input into a triangulation calculation matrix to obtain triangulated three-dimensional data; the triangulated three-dimensional data and the three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into a loss function to calculate the error value, and with the error value convergence as the goal, the model parameters of the multi-layer perceptron model are optimized by a nonlinear least squares optimization tool. The present invention uses a multi-layer perceptron model to nonlinearly convert distorted two-dimensional image data into dedistorted two-dimensional image data. The triangulated three-dimensional data obtained based on the distorted two-dimensional data realizes the nonlinear transformation of the camera triangulation transformation. By constructing a loss function between the triangulated three-dimensional data and the three-dimensional point cloud data to adjust the model parameters, the accuracy of the triangulation parameter output is improved, thereby improving the camera triangulation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 1 is a flow chart of a method for processing camera space triangulation data provided by one embodiment of the present invention;

[0049] Figure 2 1 is a schematic structural diagram of a camera space triangulation data processing device provided by one embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of a terminal device provided by one embodiment of the present invention;

[0051] Figure 4 The dual-viewing method provided by an embodiment of the present invention Figure 3 Schematic diagram of the keratinization model. DETAILED DESCRIPTION

[0052] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0053] Example 1

[0054] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a camera space triangulation data processing method provided by an embodiment of the present invention. Figure 1 As shown, this embodiment includes steps 101 to 102, and each step is specifically as follows:

[0055] Step 101: Acquire several feature point pairs of a calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is captured by the camera to be calibrated in the calibration scene.

[0056] In this embodiment, the acquisition of three-dimensional point cloud data: the coordinates of the corner points in the point cloud are obtained using the corner detection algorithm in the point cloud. As shown in Table 1, the coordinate data values ​​of the point cloud with the same name (relative to the image feature points) are large and the values ​​of the coordinates vary greatly, with the difference reaching 5 orders of magnitude. This data characteristic is not conducive to machine learning. The phenomenon of large differences in data magnitude will cause the error in the solution of the camera projection parameters to become larger or the solution to fail. Therefore, the data needs to be preprocessed before solving the camera parameters. The preprocessing method is shown in formula (1), where _x, _y, _z are the point cloud coordinate values ​​after preprocessing, and x min ,y min ,z min It is the minimum value in the point cloud coordinates, and x, y, and z are the original point cloud coordinate values. After preprocessing, the magnitude difference between the data is reduced, which is conducive to the algorithm to solve the stable camera projection parameters.

[0057] Table 1

[0058]

[0059] In a specific embodiment, in order to calculate the projection parameters of the camera, it is necessary to select corresponding point pairs in the dual-view images. In the present scheme, the two-dimensional coordinate information of the marked points in the scene of three-dimensional point cloud data acquisition is obtained through two visible light cameras installed on the tower (without dense matching between the two images to obtain a disparity map, and without strict requirements for the installation position and orientation of the cameras), a sift image feature point extraction algorithm is used, and a bidirectional matching strategy is used for corresponding point matching to obtain a feature point set S, and finally corresponding point pairs corresponding to the three-dimensional point cloud data are selected from the two-dimensional feature point set S, so that corresponding point pairs between the three-dimensional point cloud data and the corresponding two-dimensional image data are obtained as feature point pairs, as shown in Table 2.

[0060] Table 2

[0061]

[0062] Step 102: performing a model optimization operation on the multi-layer perception model according to a plurality of feature point pairs, and outputting model parameters of the multi-layer perception model as triangulation parameters of the camera to be calibrated after the error value converges; wherein the model optimization operation is specifically: selecting a target feature point pair from the feature point pairs on which the model optimization operation is not performed, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, inputting the distorted two-dimensional image data into the multi-layer perception model to obtain undistorted two-dimensional image data; inputting the undistorted two-dimensional image data into a triangulation calculation matrix to obtain triangulated three-dimensional data; substituting the triangulated three-dimensional data corresponding to the two-dimensional image data in the target feature point pair and the three-dimensional point cloud data into a loss function, calculating an error value, and taking error value convergence as the target, optimizing the model parameters of the multi-layer perception model through a nonlinear least squares optimization tool.

[0063] In the present embodiment, the multi-layer perception model satisfies the following conditions:

[0064]

[0065] wherein x is the value of the x-axis of the undistorted two-dimensional image data, y is the value of the y-axis of the undistorted two-dimensional image data, x distorted is the value of the x-axis of the distorted two-dimensional image data, y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and the bias of the first linear layer respectively, and w2 and b2 are the change parameter and the bias of the first linear layer respectively.

[0066] In a specific embodiment, the multi-layer perception model is a two-layer linear layer perception model with a sigmoid activation function, which models the nonlinear distortion process of the camera, and the parameters of the multi-layer perception model are as shown in Table 3.

[0067] Table 3

[0068]

[0069] In this embodiment, distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0070] The two-dimensional image data of the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0071]

[0072] Where k1, k2, and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0073] In a specific embodiment, a schematic diagram of a dual-view stereoscopic model is shown as follows: Figure 4 As shown, the optical axis of the camera in this model, Z cl ,Z cr Not parallel, let any point A in space be a l (u l ,v l ),a r (u r ,v r ) The corresponding homogeneous coordinates are a l (u l ,v l ,1),a r (u r ,v r ,1).

[0074] In this embodiment, the triangulated calculation matrix satisfies the following conditions:

[0075]

[0076] Where P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements in the left view projection matrix, m r Elements of the right view projection matrix.

[0077] To better illustrate, from the photogrammetry adjustment theory, the solving process of the dual-view camera parameters in triangulation is comprehensively considered, and a method for solving the dual-view camera parameters by adjustment is proposed. Figure 4 According to the dual-view triangulation principle diagram shown in the figure, the projection processes of the left and right cameras can be written into the following calculation formulas.

[0078] The left camera projection calculation formula is:

[0079]

[0080] The right camera projection calculation formula is:

[0081]

[0082] According to the matrix multiplication, the expansion calculation formula of the projection processes of the left and right cameras can be obtained.

[0083]

[0084] The expansion calculation formula is arranged in the form of a matrix, and the first matrix calculation formula is obtained.

[0085]

[0086] According to the least square method, the coordinates of the three-dimensional space point A (X, Y, Z) can be represented as a triangulation calculation matrix.

[0087] In this embodiment, the loss function comprises:

[0088]

[0089] In the formula, loss is an error value, X i , Y i and Z i are the values of the x-axis, y-axis and z-axis of the i-th triangulation three-dimensional data, _X i , _Y i and _Z i are the values of the x-axis, y-axis and z-axis of the i-th three-dimensional point cloud data, and N is the number of homonymous points observed in the left and right views.

[0090] In this embodiment, the multi-layer perception model proposed in this embodiment contains four parts of model parameters to be initialized in the triangulation process, including 17 parameters of the distortion model of the left camera and 12 parameters of the projection matrix of the left camera, 17 parameters of the distortion model of the right camera and 12 parameters of the projection matrix of the right camera, and the entire model needs to be parameterized with 58 parameters. The following gives the camera distortion model parameter initialization method and the camera projection matrix initialization method.

[0091] Camera distortion model parameter initialization method: using the image coordinate distortion correction function (undistortImagePoints) provided by Opencv (image processing library) to transform the observed feature point image coordinates in the last four columns of Table 2 into non-distorted coordinates as shown in Table 4. The data in the table has been normalized in width and height. After obtaining the distorted and non-distorted data, train the camera model using these data, minimize the distortion error, and obtain the camera distortion model distortion parameters as shown in Table 5.

[0092] Table 4

[0093]

[0094]

[0095] Table 5

[0096]

[0097] In a specific embodiment, the nonlinear least squares optimization tool can be the ceres optimization program. The three-dimensional point cloud coordinates and normalized two-dimensional image coordinate values in Table 2 and Table 4 are used as input data for the ceres optimization program. In order to make the optimization algorithm quickly converge to the global optimal solution, the camera model initialization parameters given in Table 5 are used in this scheme. The optimization algorithm converges after 2263 iterations, and the convergence value is shown in Table 6. Table 7 is the error analysis of the double-view nonlinear camera model parameter triangulation method, and the average error in x, y, z directions is less than 0.8m.

[0098] Table 6

[0099]

[0100] Table 7

[0101]

[0102] Referring to Figure 2 , Figure 2 is a structural schematic diagram of a camera space triangulation data processing device provided by an embodiment of the application, comprising a data acquisition module 201 and a model optimization module 202.

[0103] The data acquisition module is configured to acquire a plurality of feature point pairs of a calibration scene; wherein the feature point pairs comprise three-dimensional point cloud data and corresponding two-dimensional image data; and the two-dimensional image data is photographed by a camera to be calibrated in the calibration scene.

[0104] The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges;

[0105] The model optimization operation is specifically as follows:

[0106] Selecting a target feature point pair from the feature point pairs that have not undergone the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, and inputting the distorted two-dimensional image data into a multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0107] Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data;

[0108] The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and with the error value convergence as the goal, the model parameters of the multi-layer perceptron model are optimized through the nonlinear least squares optimization tool.

[0109] As an improvement to the above solution, the multi-layer perceptron model satisfies the following conditions:

[0110]

[0111] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer, respectively.

[0112] As an improvement to the above solution, distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0113] The two-dimensional image data of the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0114]

[0115] Where k1, k2, and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0116] As an improvement of the above scheme, the triangularization calculation matrix satisfies the following conditions:

[0117]

[0118] In the formula, P, Q are the triangularization three-dimensional data, u l ,v l respectively represent the x coordinate value and y coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r respectively represent the x coordinate value and y coordinate value of the pixel point in the image coordinate system of the right view, m l is an element in the left view projection matrix, m r is an element in the right view projection matrix.

[0119] As an improvement of the above scheme, the loss function comprises:

[0120]

[0121] In the formula, loss is the error value, X i , Y i and Z i are the values of the x axis, y axis and z axis of the i-th triangularization three-dimensional data, _X i , _Y i and _Z i are the values of the x axis, y axis and z axis of the i-th three-dimensional point cloud data, and N is the number of homonymous points observed in the left and right views.

[0122] This embodiment obtains several feature point pairs of the calibration scene; performs a model optimization operation on the multi-layer perceptron model based on the several feature point pairs, and after the error value converges, outputs the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated; wherein the model optimization operation is specifically: selecting a target feature point pair from the feature point pairs that have not performed the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, inputting the distorted two-dimensional image data into the multi-layer perceptron model to obtain dedistorted two-dimensional image data; inputting the dedistorted two-dimensional image data into the triangulation calculation matrix to obtain triangulated three-dimensional data; substituting the triangulated three-dimensional data and the three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair into the loss function, calculating the error value, and optimizing the model parameters of the multi-layer perceptron model through a nonlinear least squares optimization tool with the error value convergence as the goal. The present invention uses a multi-layer perceptron model to nonlinearly convert distorted two-dimensional image data into dedistorted two-dimensional image data. The triangulated three-dimensional data obtained based on the distorted two-dimensional data realizes the nonlinear transformation of the camera triangulation transformation. By constructing a loss function between the triangulated three-dimensional data and the three-dimensional point cloud data to adjust the model parameters, the accuracy of the triangulation parameter output is improved, thereby improving the camera triangulation accuracy.

[0123] Example 2

[0124] See also Figure 3 , Figure 3 It is a schematic diagram of the terminal device structure provided by one embodiment of the present invention.

[0125] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the camera space triangulation data processing method in the embodiment are implemented, such as Figure 1 Alternatively, when the processor executes the computer program, the functions of the modules in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of the camera space triangulation data processing device are shown.

[0126] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the camera space triangulation data processing method described in any of the above embodiments.

[0127] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0128] The processor 301 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor. The processor 301 is the control center of the terminal device, connecting various parts of the entire terminal device using various interfaces and lines.

[0129] The memory 302 can be used to store the computer programs and / or modules. The processor 301 implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area. The program storage area can store an operating system and at least one application required for a function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0130] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0131] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0132] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A camera space triangulation data processing method, characterized in that: include: Acquire several feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is captured by the camera to be calibrated in the calibration scene; Based on a number of feature point pairs, the multi-layer perceptron model is optimized. After the error value converges, the model parameters of the multi-layer perceptron model are output as the triangulation parameters of the camera to be calibrated. The model optimization operation is specifically as follows: Selecting a target feature point pair from the feature point pairs that have not undergone the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, and inputting the distorted two-dimensional image data into a multi-layer perceptron model to obtain dedistorted two-dimensional image data; Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data; The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and with the error value convergence as the goal, the model parameters of the multi-layer perceptron model are optimized through the nonlinear least squares optimization tool.

2. The camera space triangulation data processing method according to claim 1, characterized in that: The multi-layer perceptron model meets the following conditions: Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the second linear layer.

3. The camera space triangulation data processing method according to claim 2, characterized in that: The step of distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes: The two-dimensional image data of the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically: Where k1, k2, and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

4. The camera space triangulation data processing method according to claim 3, characterized in that: The triangulated calculation matrix satisfies the following conditions: Where P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements in the left view projection matrix, m r Elements of the right view projection matrix.

5. The camera space triangulation data processing method according to claim 4, characterized in that: The loss function includes: In the formula, loss is the error value, X i 、Y i and Z i is the value of the x-axis, y-axis and z-axis of the i-th triangulated 3D data, _X i ,_Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

6. A camera space triangulation data processing device, characterized in that: include: Data acquisition module and model optimization module; The data acquisition module is used to acquire a plurality of feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is captured by the camera to be calibrated in the calibration scene; The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges; The model optimization operation is specifically as follows: Selecting a target feature point pair from the feature point pairs that have not undergone the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, and inputting the distorted two-dimensional image data into a multi-layer perceptron model to obtain dedistorted two-dimensional image data; Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data; The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and with the error value convergence as the goal, the model parameters of the multi-layer perceptron model are optimized through the nonlinear least squares optimization tool.

7. The camera space triangulation data processing device according to claim 6, characterized in that: The multi-layer perceptron model meets the following conditions: Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the second linear layer.

8. The camera space triangulation data processing device according to claim 7, characterized in that: The step of distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes: The two-dimensional image data of the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically: Where k1, k2, and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

9. The camera space triangulation data processing device according to claim 8, characterized in that: The triangulated calculation matrix satisfies the following conditions: Where P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements in the left view projection matrix, m r Elements of the right view projection matrix.

10. The camera space triangulation data processing device according to claim 9, characterized in that: The loss function includes: In the formula, loss is the error value, X i 、Y i and Z i is the value of the x-axis, y-axis and z-axis of the i-th triangulated 3D data, _X i ,_Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

Images