The invention provides a non-orthogonal nonlinear three-dimensional scanning probe calibration method under a cylindrical coordinate measuring machine, which includes: using a standard sphere as a calibration reference, constructing multiple scanning trajectories, establishing an optimization problem with spherical constraints as the objective function, and For the coefficient of the third-order item is too small, the coefficient estimation is adjusted to obtain a spherical constraint objective function suitable for optimization, and the Levenberg-Marquardt algorithm is used to realize the optimization. The third-order nonlinear optimization model of non-orthogonal 3D scanning probe is established by multivariate Taylor function expansion theory, which is the extension of orthogonal 3D scanning probe, and the model has wide applicability. The optimization is carried out twice, the first optimization of low-order coefficients, based on which the third-order coefficients are estimated, the order of magnitude of the coefficients is adjusted to the same order of magnitude in the second optimization, and the objective function is fine-tuned synchronously, directly optimizing all third-order coefficients, avoiding the When the third-order optimization is performed directly, the third-order coefficient cannot be optimized due to the large difference in magnitude.