Camera space triangulation data processing method and device

Abstract

The invention discloses a camera space triangulation data processing method and device. The method comprises the following steps: acquiring a plurality of feature point pairs of a calibration scene; according to the plurality of feature point pairs, performing model optimization operation on the multi-layer perceptron model, and outputting model parameters of the multi-layer perceptron model as triangularization parameters of the camera to be calibrated after error value convergence; and optimizing model parameters of the multi-layer perceptron model through a nonlinear least square optimization tool. According to the method, distorted two-dimensional image data are converted into distorted two-dimensional image data in a nonlinear mode through a multi-layer perceptron model, and triangulation three-dimensional data are obtained based on the distorted two-dimensional data, so that nonlinear transformation of camera triangulation transformation is achieved; model parameter adjustment is carried out by constructing the loss function between the triangulation three-dimensional data and the three-dimensional point cloud data, the accuracy of triangulation parameter output is improved, and then the camera triangulation precision is improved.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a camera space triangulation data processing method and device. Background Art

[0002] As drone technology and photographic imaging technology become more mature, non-contact photogrammetry and 3D reconstruction based on sequence images are widely used in the industry. At present, the key factor restricting the accuracy of photogrammetry and image 3D reconstruction is how to choose the triangulation method. Due to different influencing factors, triangulation will produce different effects.

[0003] The most widely used method in the photogrammetry industry and the image 3D reconstruction industry is the linear triangulation method. The linear triangulation method uses a pinhole camera model, obtains the camera's intrinsic parameters and distortion coefficients through a pre-calibration method, transforms the image feature point coordinates into a normalized camera coordinate system, and then uses a linear triangulation method to solve the coordinates of the 3D space point. This method calibrates the camera's projection matrix separately, and in order to simplify the calculation during the triangulation process, no nonlinear factors are introduced, and the nonlinear transformation of the distorted 2D image into a dedistorted 2D image is not considered, which will lead to a decrease in the accuracy of camera triangulation.

[0004] Therefore, there is an urgent need for a camera space triangulation data processing strategy to solve the problem of low camera triangulation accuracy. Summary of the invention

[0005] The embodiments of the present invention provide a camera space triangulation data processing method and device to solve the problem of low camera triangulation accuracy.

[0006] In order to solve the above problem, an embodiment of the present invention provides a camera space triangulation data processing method, comprising:

[0007] Acquire several feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene;

[0008] According to a number of feature point pairs, a model optimization operation is performed on the multi-layer perceptron model. After the error value converges, the model parameters of the multi-layer perceptron model are output as the triangulation parameters of the camera to be calibrated.

[0009] The model optimization operation is specifically as follows:

[0010] A target feature point pair is selected from the feature point pairs that have not been subjected to the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, and the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0011] Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data;

[0012] The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and the model parameters of the multilayer perceptron model are optimized through the nonlinear least squares optimization tool with the error value convergence as the goal.

[0013] As an improvement of the above solution, the multi-layer perceptron model satisfies the following conditions:

[0014]

[0015] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

[0016] As an improvement of the above solution, the distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0017] The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0018]

[0019] Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0020] As an improvement of the above solution, the triangulated calculation matrix satisfies the following conditions:

[0021]

[0022] In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, mr Elements of the right view projection matrix.

[0023] As an improvement of the above solution, the loss function includes:

[0024]

[0025] In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0026] Accordingly, an embodiment of the present invention further provides a camera space triangulation data processing device, comprising: a data acquisition module and a model optimization module;

[0027] The data acquisition module is used to acquire a plurality of feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene;

[0028] The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges;

[0029] The model optimization operation is specifically as follows:

[0030] A target feature point pair is selected from the feature point pairs that have not been subjected to the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, and the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0031] Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data;

[0032] The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and the model parameters of the multilayer perceptron model are optimized through the nonlinear least squares optimization tool with the error value convergence as the goal.

[0033] As an improvement of the above solution, the multi-layer perceptron model satisfies the following conditions:

[0034]

[0035] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

[0036] As an improvement of the above solution, the distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0037] The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0038]

[0039] Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0040] As an improvement of the above solution, the triangulated calculation matrix satisfies the following conditions:

[0041]

[0042] In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, m r Elements of the right view projection matrix.

[0043] As an improvement of the above solution, the loss function includes:

[0044]

[0045] In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Zi are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0046] As can be seen from the above, the present invention has the following beneficial effects:

[0047] The present invention provides a camera space triangulation data processing method, which obtains a plurality of feature point pairs of a calibration scene; performs a model optimization operation on a multilayer perceptron model according to the plurality of feature point pairs, and outputs the model parameters of the multilayer perceptron model as the triangulation parameters of a camera to be calibrated after the error value converges; wherein the model optimization operation is specifically as follows: a target feature point pair is selected from the feature point pairs that have not performed the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, the distorted two-dimensional image data is input into the multilayer perceptron model to obtain dedistorted two-dimensional image data; the dedistorted two-dimensional image data is input into a triangulation calculation matrix to obtain triangulated three-dimensional data; the triangulated three-dimensional data and the three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into a loss function to calculate the error value, and with the error value convergence as the goal, the model parameters of the multilayer perceptron model are optimized by a nonlinear least squares optimization tool. The present invention uses a multi-layer perceptron model to nonlinearly convert distorted two-dimensional image data into dedistorted two-dimensional image data, and realizes the nonlinear transformation of camera triangulation transformation based on the triangulated three-dimensional data obtained from the distorted two-dimensional data; and adjusts the model parameters by constructing a loss function between the triangulated three-dimensional data and the three-dimensional point cloud data, thereby improving the accuracy of the triangulation parameter output and thus improving the camera triangulation accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 is a flowchart of a camera space triangulation data processing method provided by an embodiment of the present invention;

[0049] Figure 2 is a structural schematic diagram of a camera space triangulation data processing device provided by an embodiment of the present invention;

[0050] Figure 3 This is a schematic diagram of the structure of a terminal device provided by an embodiment of the present invention;

[0051] Figure 4 The dual-viewing method provided by an embodiment of the present invention Figure 3 Schematic diagram of the keratinization model. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Embodiment 1

[0054] See also Figure 1 , Figure 1 FIG. 1 is a flow chart of a camera space triangulation data processing method provided by an embodiment of the present invention. Figure 1 As shown, this embodiment includes step 101 to step 102, and each step is specifically as follows:

[0055] Step 101: Acquire several feature point pairs of a calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene.

[0056] In this embodiment, the acquisition of three-dimensional point cloud data: the coordinates of the corner points in the point cloud are obtained using the corner point detection algorithm in the point cloud. For example, the coordinate data values ​​of the point cloud with the same name (relative to the image feature points) in Table 1 are large and the values ​​of the coordinate values ​​differ greatly, reaching 5 orders of magnitude. This data feature is not conducive to machine learning. The phenomenon of large differences in data magnitude will cause the error of the solution result of the camera projection parameters to become larger or the solution fails. Therefore, the data needs to be preprocessed before solving the camera parameters. The preprocessing method is shown in formula (1), where _x, _y, _z are the point cloud coordinate values ​​after preprocessing, x min ,y min ,z min is the minimum value in the point cloud coordinates, x, y, z are the original point cloud coordinate values. After preprocessing, the magnitude difference between the data is reduced, which is conducive to the algorithm to solve the stable camera projection parameters.

[0057] Table 1

[0058]

[0059] In a specific embodiment, in order to calculate the projection parameters of the camera, it is necessary to select the same-name point pairs in the dual-view images. In this scheme, the two-dimensional coordinate information of the marked points in the images of the two visible light cameras installed on the tower is first obtained in the scene of the three-dimensional point cloud data collection (no need to perform dense matching between the two images to obtain the disparity map, and no strict requirements are required on the camera installation position and orientation), and the sift image feature point extraction algorithm is used. The feature points are matched with the same-name points using a two-way matching strategy to obtain a feature point set S, and finally the same-name point pairs corresponding to the three-dimensional point cloud data are selected from the two-dimensional feature point set S. In this way, we obtain the same-name point pairs between the three-dimensional point cloud data and the corresponding two-dimensional image data as feature point pairs, as shown in Table 2.

[0060] Table 2

[0061]

[0062] Step 102: According to a number of feature point pairs, a model optimization operation is performed on the multilayer perceptron model. After the error value converges, the model parameters of the multilayer perceptron model are output as the triangulation parameters of the camera to be calibrated; wherein the model optimization operation is specifically as follows: a target feature point pair is selected from the feature point pairs that have not performed the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, the distorted two-dimensional image data is input into the multilayer perceptron model to obtain dedistorted two-dimensional image data; the dedistorted two-dimensional image data is input into the triangulation calculation matrix to obtain triangulated three-dimensional data; the triangulated three-dimensional data and the three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and with the error value convergence as the goal, the model parameters of the multilayer perceptron model are optimized by a nonlinear least squares optimization tool.

[0063] In this embodiment, the multi-layer perceptron model satisfies the following conditions:

[0064]

[0065] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

[0066] In a specific embodiment, the multi-layer perceptron model is a two-layer linear layer perceptron model with a sigmoid activation function, which models the nonlinear distortion process of the camera. The parameters of the multi-layer perceptron model are shown in Table 3.

[0067] Table 3

[0068]

[0069] In this embodiment, distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0070] The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0071]

[0072] Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0073] In a specific embodiment, a schematic diagram of a dual-view stereoscopic viewing model is shown as follows: Figure 4 As shown, the optical axis of the camera in this model, Z cl ,Z cr Not parallel, suppose any point A in space is a l (u l ,v l ),a r (u r ,v r ) and their corresponding homogeneous coordinates are a l (u l ,v l ,1),a r (u r ,v r ,1).

[0074] In this embodiment, the triangulated calculation matrix satisfies the following conditions:

[0075]

[0076] In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, m r Elements of the right view projection matrix.

[0077] To better illustrate, starting from the photogrammetry adjustment theory, the process of solving the dual-view camera parameters in triangulation is considered comprehensively, and a method of solving the dual-view camera parameters by adjustment is proposed. Figure 4 The dual-view triangulation principle diagram shown in the figure, the projection process of the left and right cameras can be written as the following calculation formula:

[0078] Left camera projection calculation formula:

[0079]

[0080] Right camera projection calculation formula:

[0081]

[0082] After expanding the projection process of the left and right cameras according to matrix multiplication, the expanded calculation formula can be obtained:

[0083]

[0084] Arrange the expanded calculation formula and write it in the form of a matrix. The first matrix calculation formula is:

[0085]

[0086] The coordinates of the three-dimensional space point A (X, Y, Z) calculated by the least squares method can be expressed as a triangulated calculation matrix.

[0087] In this embodiment, the loss function includes:

[0088]

[0089] In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0090] In this embodiment, the multi-layer perceptron model proposed in this embodiment contains four parts of model parameters that need to be initialized during the triangulation process. The distortion model parameters of the left camera have a total of 17 parameters and the projection matrix of the left camera has a total of 12 parameters. The distortion model parameters of the right camera have a total of 17 parameters and the projection matrix of the right camera has a total of 12 parameters. The entire model needs to parameterize 58 parameters in total. Below we give the initialization method of the camera distortion model parameters and the initialization method of the camera projection matrix respectively;

[0091] Camera distortion model parameter initialization method: Use the image coordinate distortion correction function (undistortImagePoints) provided by Opencv (image processing library) to transform the observed feature point image coordinates as shown in the last 4 columns of Table 2 into undistorted coordinates as shown in Table 4. The data in the table has been normalized in width and height. After obtaining the distorted and undistorted data, use these data to train the camera model to minimize the distortion error, and obtain the camera distortion model distortion parameters as shown in Table 5.

[0092] Table 4

[0093]

[0094]

[0095] Table 5

[0096]

[0097] In a specific embodiment, the nonlinear least squares optimization tool can be a CERES optimization program. The three-dimensional point cloud coordinates and the normalized two-dimensional image coordinate values ​​in Tables 2 and 4 are used as input data of the CERES optimization program. In order to make the optimization algorithm converge quickly to the global optimal solution, the camera model initialization parameters given in Table 5 are used in this scheme. The optimization algorithm converged after 2263 iterations, and the convergence value is shown in Table 6. Table 7 is the error analysis of triangulation of the dual-view nonlinear camera model parameters solved by the adjustment method. The average error in the x, y, and z directions is less than 0.8m.

[0098] Table 6

[0099]

[0100] Table 7

[0101]

[0102] See also Figure 2 , Figure 2 2 is a schematic structural diagram of a camera space triangulation data processing device provided by an embodiment of the present invention, comprising: a data acquisition module 201 and a model optimization module 202;

[0103] The data acquisition module is used to acquire a plurality of feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene;

[0104] The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges;

[0105] The model optimization operation is specifically as follows:

[0106] A target feature point pair is selected from the feature point pairs that have not been subjected to the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, and the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data;

[0107] Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data;

[0108] The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and the model parameters of the multilayer perceptron model are optimized through the nonlinear least squares optimization tool with the error value convergence as the goal.

[0109] As an improvement of the above solution, the multi-layer perceptron model satisfies the following conditions:

[0110]

[0111] Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

[0112] As an improvement of the above solution, the distorting the two-dimensional image data of the target feature point pair to obtain distorted two-dimensional image data includes:

[0113] The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically:

[0114]

[0115] Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

[0116] As an improvement of the above solution, the triangulated calculation matrix satisfies the following conditions:

[0117]

[0118] In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, m r Elements of the right view projection matrix.

[0119] As an improvement of the above solution, the loss function includes:

[0120]

[0121] In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

[0122] This embodiment obtains several feature point pairs of the calibration scene; performs model optimization operations on the multi-layer perceptron model according to the several feature point pairs, and after the error value converges, outputs the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated; wherein the model optimization operation is specifically as follows: selecting a target feature point pair from the feature point pairs that have not performed the model optimization operation, distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data, inputting the distorted two-dimensional image data into the multi-layer perceptron model to obtain dedistorted two-dimensional image data; inputting the dedistorted two-dimensional image data into the triangulation calculation matrix to obtain triangulated three-dimensional data; substituting the triangulated three-dimensional data and the three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair into the loss function, calculating the error value, and optimizing the model parameters of the multi-layer perceptron model through a nonlinear least squares optimization tool with the error value convergence as the goal. The present invention uses a multi-layer perceptron model to nonlinearly convert distorted two-dimensional image data into dedistorted two-dimensional image data, and realizes the nonlinear transformation of camera triangulation transformation based on the triangulated three-dimensional data obtained from the distorted two-dimensional data; and adjusts the model parameters by constructing a loss function between the triangulated three-dimensional data and the three-dimensional point cloud data, thereby improving the accuracy of the triangulation parameter output and thus improving the camera triangulation accuracy.

[0123] Embodiment 2

[0124] See also Figure 3 , Figure 3 It is a schematic diagram of the structure of a terminal device provided in one embodiment of the present invention.

[0125] A terminal device of this embodiment includes: a processor 301, a memory 302, and a computer program stored in the memory 302 and executable on the processor 301. When the processor 301 executes the computer program, the steps of the above-mentioned camera space triangulation data processing method in the embodiment are implemented, for example Figure 1 Alternatively, when the processor executes the computer program, the functions of each module in the above-mentioned device embodiments are realized, for example: Figure 2 All modules of the camera space triangulation data processing device are shown.

[0126] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the camera space triangulation data processing method described in any of the above embodiments.

[0127] Those skilled in the art will understand that the schematic diagram is merely an example of a terminal device and does not constitute a limitation on the terminal device. The terminal device may include more or fewer components than shown in the diagram, or a combination of certain components, or different components. For example, the terminal device may also include input and output devices, network access devices, buses, etc.

[0128] The processor 301 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor 301 is the control center of the terminal device, and uses various interfaces and lines to connect various parts of the entire terminal device.

[0129] The memory 302 can be used to store the computer program and / or module. The processor 301 implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory 302. The memory 302 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0130] Wherein, if the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0131] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art may understand and implement it without paying any creative effort.

[0132] The above is a preferred embodiment of the present invention. It should be pointed out that a person skilled in the art can make several improvements and modifications without departing from the principle of the present invention. These improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A camera space triangulation data processing method, characterized in that: include: Acquire several feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene; According to a number of feature point pairs, a model optimization operation is performed on the multi-layer perceptron model. After the error value converges, the model parameters of the multi-layer perceptron model are output as the triangulation parameters of the camera to be calibrated. The model optimization operation is specifically as follows: A target feature point pair is selected from the feature point pairs that have not been subjected to the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, and the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data; Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data; The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and the model parameters of the multilayer perceptron model are optimized through the nonlinear least squares optimization tool with the error value convergence as the goal.

2. The camera space triangulation data processing method according to claim 1, characterized in that: The multi-layer perceptron model meets the following conditions: Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

3. The camera space triangulation data processing method according to claim 2, characterized in that: The step of distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data includes: The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically: Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

4. The camera space triangulation data processing method according to claim 3, characterized in that: The triangulated calculation matrix satisfies the following conditions: In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, m r Elements of the right view projection matrix.

5. The camera space triangulation data processing method according to claim 4, characterized in that: The loss function includes: In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

6. A camera space triangulation data processing device, characterized in that: include: Data acquisition module and model optimization module; The data acquisition module is used to acquire a plurality of feature point pairs of the calibration scene; wherein the feature point pairs include: three-dimensional point cloud data and corresponding two-dimensional image data; the two-dimensional image data is taken by the camera to be calibrated in the calibration scene; The model optimization module is used to perform a model optimization operation on the multi-layer perceptron model according to a plurality of feature point pairs, and output the model parameters of the multi-layer perceptron model as the triangulation parameters of the camera to be calibrated after the error value converges; The model optimization operation is specifically as follows: A target feature point pair is selected from the feature point pairs that have not been subjected to the model optimization operation, the two-dimensional image data in the target feature point pair is distorted to obtain distorted two-dimensional image data, and the distorted two-dimensional image data is input into the multi-layer perceptron model to obtain dedistorted two-dimensional image data; Inputting the dedistorted two-dimensional image data into a triangulated calculation matrix to obtain triangulated three-dimensional data; The triangulated three-dimensional data and three-dimensional point cloud data corresponding to the two-dimensional image data in the target feature point pair are substituted into the loss function, the error value is calculated, and the model parameters of the multilayer perceptron model are optimized through the nonlinear least squares optimization tool with the error value convergence as the goal.

7. The camera space triangulation data processing device according to claim 6, characterized in that: The multi-layer perceptron model meets the following conditions: Where x is the value of the x-axis of the dedistorted two-dimensional image data, y is the value of the y-axis of the dedistorted two-dimensional image data, and x distorted is the value of the x-axis of the distorted two-dimensional image data, and y distorted is the value of the y-axis of the distorted two-dimensional image data, w1 and b1 are the change parameter and bias of the first linear layer, w2 and b2 are the change parameter and bias of the first linear layer.

8. The camera space triangulation data processing device according to claim 7, characterized in that: The step of distorting the two-dimensional image data in the target feature point pair to obtain distorted two-dimensional image data includes: The two-dimensional image data in the target feature point pair is input into a distortion calculation formula to calculate the distorted two-dimensional image data; wherein the distortion calculation formula is specifically: Wherein, k1, k2 and k3 are radial distortion coefficients, p1 and p2 are tangential distortion coefficients, x is the value of the x-axis of the two-dimensional image data, y is the value of the y-axis of the two-dimensional image data, and r is the distance from the point (x, y) to the origin.

9. The camera space triangulation data processing device according to claim 8, characterized in that: The triangulated calculation matrix satisfies the following conditions: In the formula, P and Q are triangulated 3D data, u l ,v l Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the left view, u r ,v r Respectively represent the x-coordinate value and y-coordinate value of the pixel point in the image coordinate system of the right view, m l Elements of the left view projection matrix, m r Elements of the right view projection matrix.

10. The camera space triangulation data processing device according to claim 9, characterized in that: The loss function includes: In the formula, loss is the error value, X i , Y i and Z i is the value of the x-axis, y-axis, and z-axis of the i-th triangulated 3D data, _X i , _Y i and _Z i are the values ​​of the x-axis, y-axis, and z-axis of the i-th 3D point cloud data, and N is the number of points of the same name observed in the left and right views.

Images