A camera self-calibration method, device, equipment, medium and program product

Abstract

The application discloses a camera self-calibration method, device, equipment, medium and program product, and relates to the technical field of camera self-calibration, and comprises the following steps: constructing at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent; for each adjacent image pair, a target fundamental matrix of the adjacent image pair is determined; a target function based on an essential matrix manifold constraint is determined according to the target fundamental matrix of each adjacent image pair, and the target function is optimized to obtain optimal intrinsic parameters of the target camera; and the pitch angle of the target camera is determined according to the optimal intrinsic parameters and the continuous image sequence. The embodiment of the application realizes joint calibration of camera intrinsic parameters and the pitch angle, ensures the matching of the camera intrinsic parameters and the pitch angle, and thus improves the accuracy of visual navigation by using the camera intrinsic parameters and the pitch angle.

Description

Technical Field

[0001] This invention relates to the field of camera self-calibration technology, and in particular to a camera self-calibration method, apparatus, device, medium, and program product. Background Technology

[0002] Camera calibration is a fundamental technology in the field of computer vision. Its purpose is to obtain the intrinsic and extrinsic parameters of the camera to provide an accurate geometric model for subsequent agricultural machinery vision navigation applications.

[0003] In the existing technology, the camera self-calibration method cannot solve the problem of camera pitch angle calibration. The pitch angle needs to be determined separately, which leads to the separate calibration of camera intrinsic parameters and pitch angle, resulting in a mismatch between camera intrinsic parameters and pitch angle, affecting the navigation accuracy of subsequent agricultural machinery. Summary of the Invention

[0004] This invention provides a camera self-calibration method, apparatus, device, medium, and program product to achieve joint calibration of camera intrinsic parameters and pitch angle.

[0005] In a first aspect, embodiments of the present invention provide a camera self-calibration method, comprising:

[0006] Based on the continuous image sequence acquired by the target camera, at least one adjacent image pair is constructed; the adjacent image pair includes two images; the two images in the adjacent image pair are acquired at adjacent times;

[0007] For each adjacent image pair, determine the target basis matrix of the adjacent image pair;

[0008] Based on the target basis matrix of each adjacent image pair, a target function based on the essential matrix manifold constraint is determined, and the target function is optimized to obtain the optimal intrinsic parameters of the target camera;

[0009] The pitch angle of the target camera is determined based on the optimal intrinsic parameters and the continuous image sequence.

[0010] Secondly, embodiments of the present invention also provide a camera self-calibration device, comprising:

[0011] An image pair construction module is used to construct at least one adjacent image pair based on a continuous image sequence acquired by a target camera; the adjacent image pair includes two images; the two images in the adjacent image pair were acquired at adjacent times;

[0012] A matrix determination module is used to determine the target basis matrix of each adjacent image pair.

[0013] The intrinsic parameter determination module is used to determine the objective function based on the essential matrix manifold constraint according to the target basis matrix of each adjacent image pair, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera;

[0014] The pitch angle determination module is used to determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0015] Thirdly, embodiments of the present invention also provide an electronic device, comprising:

[0016] At least one processor; and

[0017] A memory that is communicatively connected to at least one processor; wherein

[0018] The memory stores instructions that can be executed by at least one processor, which enables the at least one processor to perform the camera self-calibration method provided in any embodiment of the present invention.

[0019] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the camera self-calibration method of any embodiment of the present invention.

[0020] Fifthly, embodiments of the present invention also provide a computer program product, characterized in that the computer program product includes a computer program, which, when executed by a processor, implements the camera self-calibration method of any embodiment of the present invention.

[0021] This invention constructs at least one adjacent image pair based on a continuous image sequence acquired by a target camera; each adjacent image pair comprises two images; the two images in the adjacent image pair are acquired at adjacent times; for each adjacent image pair, a target basis matrix is ​​determined; based on the target basis matrix of each adjacent image pair, an objective function based on the essential matrix manifold constraint is determined, and the objective function is optimized to obtain the optimal intrinsic parameters of the target camera; based on the optimal intrinsic parameters and the continuous image sequence, the pitch angle of the target camera is determined. This enables the determination of the camera pitch angle after determining the camera intrinsic parameters, achieving joint calibration of the camera intrinsic parameters and pitch angle, ensuring that the camera intrinsic parameters and pitch angle are matched, thereby improving the accuracy of visual navigation using camera intrinsic parameters and pitch angle.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a camera self-calibration method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a camera self-calibration method according to Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of a camera self-calibration device according to Embodiment 3 of the present invention;

[0027] Figure 4 This is a structural diagram of an electronic device that implements a camera self-calibration method according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first" and "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] The acquisition, storage, and application of continuous image sequences and other related technologies in the technical solutions of this invention comply with relevant laws and regulations and do not violate public order and good morals.

[0031] Example 1

[0032] Figure 1 This is a flowchart of a camera self-calibration method provided in Embodiment 1 of the present invention. This embodiment is applicable to the calibration of camera intrinsic parameters and pitch angle. The method can be executed by a camera self-calibration device, which can be implemented in hardware and / or software and specifically configured in an electronic device.

[0033] See Figure 1 The camera self-calibration method shown includes:

[0034] S101. Based on the continuous image sequence acquired by the target camera, construct at least one adjacent image pair; the adjacent image pair includes two images; the acquisition times of the two images in the adjacent image pair are adjacent.

[0035] S102. For each adjacent image pair, determine the target basis matrix of the adjacent image pair.

[0036] S103. Based on the target basis matrix of each adjacent image pair, determine the objective function based on the essential matrix manifold constraint, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera.

[0037] S104. Determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0038] In this embodiment, the target camera can be a camera whose intrinsic parameters and pitch angle are to be calibrated. The continuous image sequence can be at least two images continuously acquired by the target camera at a set frame rate to ensure that adjacent images in the continuous image sequence avoid excessive motion differences and have sufficient parallax. It should be noted that the set frame rate can be set independently by technicians based on actual needs or practical experience; this invention does not limit this. The target fundamental matrix can be used to characterize the epipolar geometric relationship between two images in an adjacent image pair. The optimal intrinsic parameters can be camera intrinsic parameters obtained after optimizing the objective function. Camera intrinsic parameters can include, but are not limited to, at least one of the following: horizontal focal length, vertical focal length, horizontal coordinates of the principal point, and vertical coordinates of the principal point.

[0039] In one optional embodiment, the target camera is a monocular camera, which is fixedly mounted on the agricultural machinery body and faces the direction of travel of the agricultural machinery. During the travel of the agricultural machinery, images are continuously acquired at a fixed frame rate to obtain a continuous image sequence.

[0040] Specifically, in a continuous image sequence, images adjacent to each other at any two acquisition times are defined as an adjacent image pair. For each adjacent image pair, feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented Fast and Rotated BRIE) are used to extract matching feature point pairs from the two images of the adjacent image pair. Feature point pairs with descriptor distances less than a preset threshold are defined as weak feature point pairs, and these weak feature point pairs are removed, while feature point pairs with descriptor distances greater than or equal to the preset threshold are retained. RANSAC (Random Sampled Algorithm) is then used to further refine the feature point pairs. The Consensus algorithm (random sampling consensus) calculates the target basis matrix between two images in an adjacent image pair based on the retained feature point pairs; for each adjacent image pair, it determines the essential matrix corresponding to the adjacent image pair; it determines the manifold constraint terms of the essential matrix corresponding to the adjacent image pair; it fuses the manifold constraint terms of the essential matrix corresponding to each adjacent image pair to obtain an objective function based on the manifold constraint of the essential matrix, and optimizes the objective function using the least squares method to solve for the optimal intrinsic parameters of the target camera; and it uses a certain algorithm to determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0041] Optionally, constructing at least one adjacent image pair based on a continuous image sequence acquired by the target camera includes: constructing at least one original image pair based on the continuous image sequence acquired by the target camera; the original image pair includes two images; the two images in the original image pair were acquired at adjacent times; for each original image pair, determining the structural similarity between the two images in the original image pair, and using the structural similarity as the structural similarity of the original image pair; and selecting adjacent image pairs from each original image pair based on the structural similarity of each original image pair.

[0042] Specifically, each pair of adjacent images acquired in a continuous image sequence is defined as an original image pair. For each original image pair, the mean and variance of pixel values ​​of the two images in the pair, as well as the covariance of pixel values ​​between the two images, are calculated. The structural similarity between the two images in the original image pair is then calculated using the mean and variance of pixel values, and the covariance of pixel values, and is used as the structural similarity of the original image pair. For example, the structural similarity can be determined using the following formula:

[0043] ;

[0044] in, Indicates structural similarity; This represents the average pixel value of one image in the original image pair. This represents the variance of the pixel values ​​in the image. The average pixel value of the other image in the original image pair. This represents the variance of the pixel values ​​in the image. Represents the brightness constant; Represents the contrast constant; This represents the pixel value covariance between the two images in the original image pair.

[0045] Original image pairs with structural similarity within a preset range are identified as adjacent image pairs. It should be noted that the preset range can be set independently by technicians based on actual needs or practical verification, and this invention does not limit it.

[0046] Understandably, by adopting the above scheme, at least one original image pair is constructed based on the continuous image sequence acquired by the target camera, and the structural similarity of each original image pair is calculated. Original image pairs with structural similarity within a preset range are identified as adjacent image pairs. This can remove original image pairs with high structural similarity (too small image differences) and original image pairs with low structural similarity (large motion or blur), thereby ensuring that the geometric constraints of adjacent images are effective and the matching is stable, and improving the stability and success rate of subsequent intrinsic parameter calibration.

[0047] Optionally, determining the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence includes: for each image in the continuous image sequence, identifying the scene of the image; if the scene is a road scene, extracting the first and second road boundary lines of the road mask in the image, and determining the intersection point between the first and second road boundary lines as the vanishing point; if the scene is a farmland scene, extracting the center lines of each crop row mask in the image; determining the vanishing point based on the intersection point between each of the center lines; determining the camera pitch angle corresponding to the image based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters; and determining the pitch angle of the target camera based on the camera pitch angle corresponding to each image.

[0048] The first road boundary line and the second road boundary line can be the road boundary lines at both ends of the road mask. Specifically, for each image in the continuous image sequence, a target detection model can be used to identify the road mask or crop row mask in the image; if a road mask is identified in the image, the scene of the image is determined to be a road scene; if no road mask is identified in the image, but a crop row mask is identified, the scene of the image is determined to be a farmland scene.

[0049] If the scene is a road scene, the boundary lines at both ends of the road mask are extracted and vertical deduplication is performed; the first road boundary line and the second road boundary line are fitted using a line fitting algorithm such as least squares or RANSAC; the intersection of the first road boundary line and the second road boundary line is calculated as the vanishing point; if the scene is a farmland scene, the center line of each crop row mask is fitted using a line fitting algorithm, and the intersection between every two center lines is calculated; the median of the horizontal coordinates and the median of the vertical coordinates are determined based on the horizontal and vertical coordinates of each intersection point in the image; the intersection of the horizontal coordinate and the vertical coordinate is determined as the vanishing point; the camera pitch angle corresponding to the image is determined based on the vertical coordinates of the vanishing point and the vertical focal length and principal point vertical coordinates in the optimal intrinsic parameters; for example, the camera pitch angle can be determined by the following formula:

[0050] ;

[0051] in, Indicates the first The camera pitch angle corresponding to each image; Indicates the first The vertical coordinates of the vanishing point in each image; Indicates the vertical coordinates of the principal point; Indicates the vertical focal length.

[0052] Calculate the average pitch angle of the camera corresponding to each image, and determine the average pitch angle as the pitch angle of the target camera.

[0053] Understandably, by adopting the above technical solution, after determining the optimal intrinsic parameters, the pitch angle can be determined in a unified process without the need for a calibration board and a dedicated sensor. This enables online, low-cost, and highly robust joint self-calibration of the pitch angle involving the camera. Furthermore, it achieves pitch angle self-calibration in road or farmland scenarios, thus broadening the applicable scenarios for self-calibration of the pitch angle involving the camera.

[0054] This invention constructs at least one adjacent image pair based on a continuous image sequence acquired by a target camera. Each adjacent image pair comprises two images acquired at adjacent times. For each adjacent image pair, a target basis matrix is ​​determined. Based on the target basis matrix of each adjacent image pair, an objective function based on the essential matrix manifold constraint is determined and optimized to obtain the optimal intrinsic parameters of the target camera. Based on the optimal intrinsic parameters and the continuous image sequence, the pitch angle of the target camera is determined. After determining the camera intrinsic parameters, the camera pitch angle is further determined based on the camera intrinsic parameters, achieving joint calibration of the camera intrinsic parameters and pitch angle. This ensures that the camera intrinsic parameters and pitch angle are matched, thereby improving the accuracy of visual navigation using the camera intrinsic parameters and pitch angle.

[0055] Example 2

[0056] Figure 2 This is a flowchart of a camera self-calibration method provided in Embodiment 2 of the present invention. Based on the technical solution of the above embodiments, the present invention optimizes and improves the determination operation of the target basis matrix.

[0057] Furthermore, the step of "determining the target basis matrix of each adjacent image pair" is refined to "for each adjacent image pair, extracting feature matching point pairs from the adjacent image pair; the feature matching point pair includes two matching feature points; determining the first basis matrix of the adjacent image pair based on each feature matching point pair; selecting first matching point pairs that satisfy geometric constraints from each feature matching point pair based on the first basis matrix of the adjacent image pair; discarding adjacent image pairs with an interior point ratio less than a set threshold; the interior point ratio is the ratio between the number of first matching point pairs and the number of feature matching point pairs; for each adjacent image pair that is not discarded, determining the target basis matrix of the adjacent image pair based on the first matching point pairs of the adjacent image pair", thus improving the operation of determining the target basis matrix.

[0058] It should be noted that for any parts not described in detail in the embodiments of the present invention, please refer to the description in the foregoing embodiments.

[0059] See Figure 2 The camera self-calibration method shown includes:

[0060] S201. Based on the continuous image sequence acquired by the target camera, construct at least one adjacent image pair; the adjacent image pair includes two images; the acquisition times of the two images in the adjacent image pair are adjacent.

[0061] S202. For each adjacent image pair, extract the feature matching point pair in the adjacent image pair; the feature matching point pair includes two matching feature points.

[0062] S203. Determine the first basis matrix of the adjacent image pair based on each of the feature matching point pairs.

[0063] S204. Based on the first fundamental matrix of the adjacent image pair, select the first matching point pair that satisfies the geometric constraints from each of the feature matching point pairs.

[0064] S205. Discard adjacent image pairs whose inlier ratio is less than a set threshold; the inlier ratio is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs.

[0065] S206. For each pair of adjacent images that has not been discarded, determine the target basis matrix of the adjacent image pair based on the first matching point pair of the adjacent image pair.

[0066] S207. Based on the target basis matrix of each adjacent image pair, determine the objective function based on the essential matrix manifold constraint, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera.

[0067] S208. Determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0068] In this embodiment, the first fundamental matrix is ​​the fundamental matrix determined based on the feature matching point pairs extracted from adjacent image pairs; the target fundamental matrix can be the fundamental matrix determined based on the first matching point pairs.

[0069] Specifically, for each adjacent image pair, feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented Fast and Rotated BRIE) are used to extract matching feature point pairs from the two images of the adjacent image pair. Feature point pairs with a descriptor distance less than a preset threshold are identified as weak feature point pairs, and these weak feature point pairs are removed, retaining those with a descriptor distance greater than or equal to the preset threshold. The RANSAC (Random Sample Consensus) algorithm is used to calculate the first fundamental matrix between the two images in the adjacent image pair based on the retained feature point pairs. Based on the first fundamental matrix of the adjacent image pair, the first matching point pair satisfying the geometric constraints is selected from each of the feature matching point pairs. For example, the geometric constraints of the adjacent image pair can be represented by the following formula:

[0070] ;

[0071] in, Indicates the first The homogeneous coordinates of a matching point in a feature matching point pair of adjacent image pairs. Indicates the first Transpose of the homogeneous coordinates of another matching point in a pair of adjacent image feature matching points; Indicates the first The first fundamental matrix corresponding to each image pair;

[0072] The feature matching point pairs that satisfy the geometric constraints are identified as the first matching point pairs; the ratio between the number of first matching point pairs and the number of feature matching point pairs in the adjacent image pairs is determined; adjacent image pairs with an inlier ratio less than a set threshold are discarded; for each adjacent image pair that is not discarded, the target basis matrix of the adjacent image pair is determined based on the first matching point pairs of the adjacent image pair.

[0073] It should be noted that the threshold can be set independently by technical personnel based on actual needs or practical experience, and this invention does not limit this; the process of determining the target basis matrix based on the first matching point pair is similar to the process of determining the first basis matrix based on the feature matching point pair, and will not be described in detail here.

[0074] Optionally, determining the objective function based on the essential matrix manifold constraint according to the target basis matrix of each of the adjacent image pairs includes: for each adjacent image pair, determining the essential matrix of the adjacent image pair according to the target basis matrix of the adjacent image pair and preset prior camera intrinsic parameters; the prior camera intrinsic parameters include prior horizontal focal length, prior vertical focal length, prior principal point horizontal coordinates, and prior principal point vertical coordinates; performing singular value decomposition on the essential matrix, and calculating the rank constraint residual and singular value equality residual corresponding to the adjacent image pair according to the singular value decomposition results; for each of the adjacent image pairs... The neighboring images are fused together with the corresponding rank-constrained residuals and singular value equality residuals to obtain the target residual. Based on the residual weights and residual weight coefficients, the first focal length residual between the target camera's focal length in the horizontal direction and its focal length in the vertical direction, the principal point prior residual between the target camera's principal point coordinates and the preset prior principal point coordinates, and the second focal length residual between the target camera's focal length and the preset prior focal length are determined. The target residual, the first focal length residual, the principal point prior residual, and the second focal length residual are fused together to obtain the objective function based on the essential matrix manifold constraint.

[0075] The preset prior camera intrinsic parameters can be pre-defined prior values ​​of the camera intrinsic parameters; the camera intrinsic parameters can be represented in matrix form; for example, the camera intrinsic parameter matrix can be represented by the following matrix:

[0076] ;

[0077] in, Indicates the horizontal focal length; This represents the horizontal coordinates of the principal point.

[0078] The residual weighting coefficients may include the first focal length residual weighting coefficient, the second focal length residual weighting coefficient, the first principal point residual weighting coefficient, and the second principal point residual weighting coefficient.

[0079] The singular value decomposition results can include the first singular value, the second singular value, and the third singular value; the rank constraint residual and the singular value equality residual are used to characterize the degree of deviation of the essential matrix from the manifold constraint; the rank constraint residual can be used to characterize the degree of satisfaction of the rank constraint; the singular value equality residual can be used to characterize the degree of satisfaction of the singular value equality.

[0080] Specifically, for each adjacent image pair, the essential matrix of the adjacent image pair is determined based on the target basis matrix of the adjacent image pair and the preset prior camera intrinsic parameters; for example, the essential matrix of the adjacent image pair can be determined by the following formula:

[0081] ;

[0082] in, Indicates the first The essential matrix of adjacent image pairs; Indicates the first The target basis matrix of adjacent image pairs; This represents the transpose of the camera intrinsic parameter matrix.

[0083] Singular value decomposition is performed on the essential matrix of the adjacent image pair to obtain the first, second, and third singular values ​​corresponding to the adjacent image pair; the rank-constrained residual is determined based on the first, second, and third singular values; for example, the rank-constrained residual can be determined by the following formula:

[0084] ;

[0085] in, Indicates the first Rank-constrained residuals of adjacent image pairs; Indicates the first The third singular value of a pair of adjacent images; Indicates the first The first singular value of a pair of adjacent images; Indicates the first The second singular value of a pair of adjacent images.

[0086] The singular value equality residual is determined based on the first and second singular values; for example, the singular value equality residual can be determined using the following formula:

[0087] ;

[0088] in, Indicates the first The singular values ​​of adjacent image pairs are equal residuals.

[0089] The rank-constrained residuals and singular value equality residuals corresponding to each pair of adjacent images are fused to obtain the target residual. Based on the residual weights and residual weight coefficients, the following are determined: a first focal length residual between the target camera's focal length in the horizontal direction and its focal length in the vertical direction; a principal point prior residual between the target camera's principal point coordinates and a preset prior principal point coordinate; and a second focal length residual between the target camera's focal length and the preset prior focal length. For example, the first focal length residual can be determined using the following formula:

[0090] ;

[0091] in, Indicates the residual at the first focal length; Indicates the residual weight; This represents the weighting coefficient of the residual at the first focal length.

[0092] The principal point prior residuals can include the principal point horizontal prior residuals and the principal point vertical prior residuals; for example, the principal point horizontal prior residuals can be determined by the following formula:

[0093] ;

[0094] in, This represents the principal point level prior residual; Represents the prior horizontal coordinates of the principal point; Indicates the width of the image; This represents the residual weighting coefficient of the first principal point;

[0095] For example, the vertical prior residual of the principal point can be determined by the following formula:

[0096] ;

[0097] in, Indicates the vertical prior residual of the principal point; Represents the prior vertical coordinates of the principal point; Indicates the height of the image; This represents the residual weighting coefficient of the second principal point;

[0098] For example, the second focal length residual can be determined by the following formula:

[0099] ;

[0100] in, Indicates the residual at the second focal length; Indicates the prior horizontal focal length; Indicates the prior vertical focal length; This represents the weighting coefficient for the second focal length residual;

[0101] The target residual, the first focal length residual, the principal point prior residual, and the second focal length residual are fused to obtain an objective function based on the essential matrix manifold constraint; for example, the objective function can be expressed by the following formula:

[0102] ;

[0103] in, Represent the objective function; Indicates the target residual; This indicates the number of adjacent image pairs.

[0104] Understandably, by adopting the above technical solution, the essential matrix is ​​determined, and its singularity is solved. Relying on the rank of the essential matrix being 2 and the algebraic property that the first and second singular values ​​are equal, a direct mapping of the camera's intrinsic parameters to the geometric relationship of the image is constructed. This avoids the error accumulation and propagation caused by indirect constraints in existing technologies, and improves the accuracy of the subsequently determined intrinsic parameters. By determining regularization constraint terms including the first focal length residual, the principal point horizontal prior residual, the principal point vertical prior residual, and the second focal length residual, the optimization of the objective function is guided towards a reasonable convergence direction by prior knowledge when the number and quality of adjacent image pairs are insufficient. When the number and quality of adjacent image pairs are good, the influence of the regularization constraint terms is minimized, avoiding over-constraint.

[0105] Optionally, optimizing the objective function to obtain the optimal intrinsic parameters of the target camera includes: calculating a target residual weight value based on the number of adjacent image pairs and the number of first matching point pairs for each adjacent image pair; assigning the residual weight in the objective function to the target residual weight value and assigning the residual weight coefficient in the objective function to a preset value, and iteratively optimizing the prior camera intrinsic parameters with the objective function minimizing its value to obtain the first camera intrinsic parameters of the target camera; calculating a first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters, and using the first residual weight coefficient value as an auxiliary coefficient value; assigning the residual weight in the objective function to the target residual weight value and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iteratively optimizing the prior camera intrinsic parameters with the objective function minimizing its value to obtain the first camera intrinsic parameters of the target camera; calculating a first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters, and using the first residual weight coefficient value as an auxiliary coefficient value; ... based on the objective function minimizing its value to obtain the first camera intrinsic parameters of the target camera; calculating a first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters, and using the first residual weight coefficient as an auxiliary coefficient value; and iteratively optimizing the prior camera intrinsic parameters based on the objective function minimizing its value to obtain the first camera intrinsic parameters of the target camera; calculating a first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters of the target camera, and iteratively optimizing the prior camera intrinsic parameters based on the objective function minimizing its value to obtain the first camera intrinsic parameters of the target camera; calculating a first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters of the target camera, and iteratively optimizing the prior camera intrinsic parameters based on the objective function mini With the objective function to be minimized, the prior camera intrinsic parameters are iterated to obtain the second camera intrinsic parameters of the target camera. Based on the second camera intrinsic parameters and the prior camera intrinsic parameters, a second residual weight coefficient value is calculated. If the second residual weight coefficient value is less than or equal to a preset threshold, the second camera intrinsic parameters are determined as the optimal intrinsic parameters. If the second residual weight coefficient value is greater than the preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the process returns to execute the steps of assigning the residual weight in the objective function to the target residual weight value, and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iterating the prior camera intrinsic parameters with the objective function to be minimized to obtain the second camera intrinsic parameters of the target camera, until the second residual weight coefficient value is less than or equal to the preset threshold.

[0106] The first residual weight coefficient value includes the first coefficient value corresponding to the first focal length residual weight coefficient, the second coefficient value corresponding to the second focal length residual weight coefficient, the third coefficient value corresponding to the first principal point residual weight coefficient, and the fourth coefficient value corresponding to the second principal point residual weight coefficient.

[0107] The auxiliary coefficient values ​​include the auxiliary coefficient values ​​corresponding to the first focal length residual weight coefficient, the auxiliary coefficient values ​​corresponding to the second focal length residual weight coefficient, the auxiliary coefficient values ​​corresponding to the first principal point residual weight coefficient, and the auxiliary coefficient values ​​corresponding to the second principal point residual weight coefficient.

[0108] Specifically, a first average is determined among the first matching point pairs of each of the adjacent image pairs; a target residual weight value is calculated based on the number of adjacent image pairs and the first average; for example, the target residual weight value is calculated based on the number of adjacent image pairs and the number of first matching point pairs of each of the adjacent image pairs:

[0109] ;

[0110] ;

[0111] ;

[0112] in, This represents the target residual weight value; This represents the first average. Indicates the number of adjacent image pairs; , and This represents the set coefficient constant.

[0113] The residual weights in the objective function are assigned the target residual weight value, and the residual weight coefficients in the objective function are assigned preset values. The prior camera intrinsic parameters are iteratively optimized with the objective function value being minimized to obtain the first camera intrinsic parameters of the target camera. In one specific embodiment, the preset value is 1.

[0114] Based on the first camera intrinsic parameters and the prior camera intrinsic parameters, calculate the first deviation between the horizontal and vertical focal lengths in the first camera intrinsic parameters; calculate the second deviation between the horizontal coordinates of the principal point in the first camera intrinsic parameters and the prior horizontal coordinates of the principal point in the prior camera intrinsic parameters, and the third deviation between the vertical coordinates of the principal point in the first camera intrinsic parameters and the prior vertical coordinates in the prior camera intrinsic parameters; calculate the first focal length sum between the horizontal and vertical focal lengths in the first camera intrinsic parameters, and the second focal length sum between the horizontal and vertical focal lengths in the prior camera intrinsic parameters; calculate the fourth deviation between the first focal length sum and the second focal length sum.

[0115] Based on the first deviation, determine the first coefficient value corresponding to the first focal length residual weighting coefficient; based on the second deviation, determine the second coefficient value corresponding to the second focal length residual weighting coefficient; based on the third deviation, determine the third coefficient value corresponding to the first principal point residual weighting coefficient; based on the fourth deviation, determine the fourth coefficient value corresponding to the second principal point residual weighting coefficient; and use the first residual weighting coefficient value as an auxiliary coefficient value.

[0116] For example, the value of the first coefficient can be determined by the following formula:

[0117] ;

[0118] in, Indicates the value of the first coefficient; This represents a set constant; Indicates the first deviation; This represents the lower limit threshold of the deviation of the set first focal length residual weighting coefficient. This represents the upper limit threshold of the deviation of the first focal length residual weighting coefficient.

[0119] It should be noted that the determination process for the second, third, and fourth coefficient values ​​is similar to that for the first coefficient value, and will not be repeated here. The lower limit threshold for deviation is different among the four coefficients: the first focal length residual weight coefficient, the second focal length residual weight coefficient, the first principal point residual weight coefficient, and the second principal point residual weight coefficient. The upper limit threshold for deviation is also different.

[0120] The residual weights in the objective function are assigned the target residual weight value, and the residual weight coefficients in the objective function are assigned the auxiliary coefficient value. The prior camera intrinsics are iterated with the objective function value minimized to obtain the second camera intrinsics of the target camera. A similar process to determining the first residual weight coefficient value is used to calculate the second residual weight coefficient value based on the second camera intrinsics and the prior camera intrinsics; this will not be elaborated further here. If any item in the second residual weight coefficient value is less than or equal to a preset threshold, the second camera intrinsics are determined as the optimal intrinsics. If any item in the second residual weight coefficient value is greater than the preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the process returns to the step of assigning the residual weights in the objective function the target residual weight value, assigning the residual weight coefficients in the objective function the auxiliary coefficient value, and iterating with the objective function value minimized to obtain the second camera intrinsics of the target camera, until all items in the second residual weight coefficient value are less than or equal to the preset threshold.

[0121] In an optional embodiment, the preset threshold may include a first threshold corresponding to the first focal length residual weight coefficient, a second threshold corresponding to the second focal length residual weight coefficient, a third threshold corresponding to the first principal point residual weight coefficient, and a fourth threshold corresponding to the second principal point residual weight coefficient. The first threshold, second threshold, third threshold, and fourth threshold may be different, or at least two of them may be the same. Accordingly, if all items in the second residual weight coefficient value are less than or equal to the corresponding preset threshold, the second camera intrinsic parameter is determined as the optimal intrinsic parameter. If any item in the second residual weight coefficient value is greater than the corresponding preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the process of assigning the residual weight in the objective function to the target residual weight value, and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iterating the prior camera intrinsic parameter with the objective function value as the minimum as the objective value, to obtain the second camera intrinsic parameter of the target camera, continues until all items in the second residual weight coefficient value are less than or equal to the corresponding preset threshold.

[0122] In an optional embodiment, while calculating the first residual weight coefficient value, the RANSAC algorithm can also be used for each adjacent image pair to redetermine the first matching point pair in the adjacent image pair based on the first camera intrinsic parameters, so as to obtain a more accurate first matching point pair.

[0123] Understandably, by adopting the above technical solution, a coarse solution of the camera intrinsic parameters, namely the first camera intrinsic parameters, is first determined. Then, the objective function is optimized again by refining the interior points and dynamically adjusting the residual weight coefficients, thereby improving convergence and the reliability of the results. Through the dynamic adjustment of the residual weight coefficients during the iteration process, the data constraints and prior constraints are automatically balanced to address the differences in data quality and the degree of parameter deviation. This can resolve the contradiction in traditional self-calibration where weak priors lead to divergence and strong priors lead to convergence to incorrect solutions, thereby improving convergence stability and the reliability of the determined optimal intrinsic parameters.

[0124] This invention, through feature extraction and feature point matching, and filtering of adjacent image pairs based on their inlier ratios, improves data quality and enhances the efficiency of optimizing the objective function. For each adjacent image pair, feature matching points are extracted, each pair comprising two matched feature points. A first basis matrix for the adjacent image pair is determined based on each feature matching point pair. Based on this first basis matrix, first matching point pairs satisfying geometric constraints are selected from the feature matching point pairs. Adjacent image pairs with an inlier ratio less than a set threshold are discarded. The inlier ratio is the ratio between the number of first matching point pairs and the number of feature matching point pairs. For each non-discarded adjacent image pair, a target basis matrix is ​​determined based on its first matching point pairs. This process, through feature extraction and feature point matching, and filtering adjacent image pairs based on inlier ratios, removes adjacent image pairs with poor data validity, thereby improving data quality and enhancing the efficiency of optimizing the objective function.

[0125] Example 3

[0126] Figure 3 This is a schematic diagram of a camera self-calibration device provided in Embodiment 3 of the present invention. This embodiment of the invention is applicable to the calibration of camera intrinsic parameters and pitch angles. The device can execute a camera self-calibration method and can be implemented in hardware and / or software. The device can be configured in an electronic device.

[0127] See Figure 3 The camera self-calibration device shown includes an image pair construction module 301, a matrix determination module 302, an intrinsic parameter determination module 303, and a pitch angle determination module 304, wherein...

[0128] The image pair construction module 301 is used to construct at least one adjacent image pair based on a continuous image sequence acquired by the target camera; the adjacent image pair includes two images; the two images in the adjacent image pair are acquired at adjacent times;

[0129] The matrix determination module 302 is used to determine the target basis matrix of each adjacent image pair;

[0130] The intrinsic parameter determination module 303 is used to determine the objective function based on the essential matrix manifold constraint according to the target basis matrix of each adjacent image pair, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera;

[0131] The pitch angle determination module 304 is used to determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0132] This invention employs an image pair construction module to construct at least one adjacent image pair based on a continuous image sequence acquired by a target camera. Each adjacent image pair comprises two images acquired at adjacent times. A matrix determination module determines the target fundamental matrix for each adjacent image pair. An intrinsic parameter determination module determines an objective function based on the essential matrix manifold constraint, based on the target fundamental matrix of each adjacent image pair, and optimizes the objective function to obtain the optimal intrinsic parameters of the target camera. A pitch angle determination module determines the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence. This allows for the determination of the camera pitch angle after the intrinsic parameters are determined, achieving joint calibration of the camera intrinsic parameters and pitch angle, ensuring matching between camera intrinsic parameters and pitch angle, thereby improving the accuracy of visual navigation using camera intrinsic parameters and pitch angle.

[0133] Optional, the matrix determination module includes:

[0134] An extraction unit is used to extract feature matching point pairs from each adjacent image pair; the feature matching point pair includes two matching feature points.

[0135] The first determining unit is used to determine the first basis matrix of the adjacent image pair based on each of the feature matching point pairs;

[0136] The filtering unit is used to filter out a first matching point pair that satisfies geometric constraints from each of the feature matching point pairs based on the first basis matrix of the adjacent image pairs.

[0137] A discarding unit is used to discard adjacent image pairs whose inlier ratio is less than a set threshold; the inlier ratio is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs.

[0138] The second determining unit is used to determine the target basis matrix of each non-discarded adjacent image pair based on the first matching point pair of the adjacent image pair.

[0139] Optionally, the intrinsic parameter determination module 303 includes:

[0140] The first matrix determination unit is used to determine the essential matrix of each adjacent image pair based on the target basis matrix of the adjacent image pair and the preset prior camera intrinsic parameters; the prior camera intrinsic parameters include prior horizontal focal length, prior vertical focal length, prior principal point horizontal coordinates and prior principal point vertical coordinates;

[0141] The first residual calculation unit is used to perform singular value decomposition on the essential matrix, and calculate the rank-constrained residual and singular value equal residual corresponding to the adjacent image pair based on the singular value decomposition result.

[0142] The first fusion unit is used to fuse the rank-constrained residuals and singular value equal residuals corresponding to each of the adjacent image pairs to obtain the target residual.

[0143] The second residual calculation unit is used to determine, based on the residual weight and residual weight coefficient, the first focal length residual between the focal length of the target camera in the horizontal direction and the focal length in the vertical direction, the principal point prior residual between the principal point coordinates of the target camera and the preset prior principal point coordinates, and the second focal length residual between the focal length of the target camera and the preset prior focal length.

[0144] The second fusion unit is used to fuse the target residual, the first focal length residual, the principal point prior residual, and the second focal length residual to obtain an objective function based on the essential matrix manifold constraint.

[0145] Optionally, the intrinsic parameter determination module 303 includes:

[0146] The weight calculation unit is used to calculate the target residual weight value based on the number of adjacent image pairs and the number of first matching point pairs of each adjacent image pair;

[0147] The first optimization unit is used to assign the residual weight in the objective function to the target residual weight value, and to assign the residual weight coefficient in the objective function to a preset value, and to iteratively optimize the prior camera intrinsic parameters with the objective function value being minimized, so as to obtain the first camera intrinsic parameters of the target camera.

[0148] The first coefficient calculation unit is used to calculate the first residual weight coefficient value based on the first camera intrinsic parameters and the prior camera intrinsic parameters, and to use the first residual weight coefficient value as an auxiliary coefficient value.

[0149] The second optimization unit is used to assign the residual weight in the objective function to the target residual weight value, and to assign the residual weight coefficient in the objective function to the auxiliary coefficient value, and to iterate the prior camera intrinsic parameters with the objective function value as the minimum as the objective function value, so as to obtain the second camera intrinsic parameters of the target camera.

[0150] The second coefficient calculation unit is used to calculate the second residual weight coefficient value based on the second camera intrinsic parameters and the prior camera intrinsic parameters.

[0151] The intrinsic parameter determination unit is used to determine the second camera intrinsic parameter as the optimal intrinsic parameter if the value of the second residual weight coefficient is less than or equal to a preset threshold.

[0152] The loop unit is configured to update the auxiliary coefficient value to the second residual weight coefficient value if the second residual weight coefficient value is greater than the preset threshold, and return to execute the steps of assigning the residual weight in the objective function to the target residual weight value, and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iterating the prior camera intrinsic parameters with the objective function value as the minimum as the objective function value to obtain the second camera intrinsic parameters of the target camera, until the second residual weight coefficient value is less than or equal to the preset threshold.

[0153] Optionally, the image pair building module 301 includes:

[0154] Based on the continuous image sequence acquired by the target camera, at least one original image pair is constructed; the original image pair includes two images; the acquisition times of the two images in the original image pair are adjacent.

[0155] For each original image pair, determine the structural similarity between the two images in the original image pair, and use the structural similarity as the structural similarity of the original image pair;

[0156] Based on the structural similarity of each original image pair, adjacent image pairs are selected from each original image pair.

[0157] Optionally, the pitch angle determination module 304 includes:

[0158] A scene recognition unit is used to recognize the scene of each image in the continuous image sequence;

[0159] The first point determination unit is used to extract the first road boundary line and the second road boundary line of the road mask in the image if the scene is a road scene, and determine the intersection point between the first road boundary line and the second road boundary line as the vanishing point;

[0160] The centerline determination unit is used to extract the centerline of each crop row mask in the image if the scene is a farmland scene;

[0161] The second point determination unit is used to determine the vanishing point based on the intersection points between the center lines.

[0162] The first angle determination unit is used to determine the camera pitch angle corresponding to the image based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters;

[0163] The second angle determination unit is used to determine the pitch angle of the target camera based on the camera pitch angle corresponding to each image.

[0164] The camera self-calibration device provided in the embodiments of the present invention can execute the camera self-calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the camera self-calibration method.

[0165] Example 4

[0166] Figure 4 A schematic diagram of a camera self-calibration device 410, which can be used to implement embodiments of the present invention, is shown. The camera self-calibration device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The camera self-calibration device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0167] like Figure 4 As shown, the camera self-calibration device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the camera self-calibration device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0168] Multiple components in the camera self-calibration device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a disk, optical disk, etc.; and a communication unit 419, such as a network card, modem, wireless transceiver, etc. The communication unit 419 allows the camera self-calibration device 410 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0169] Processor 411 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 411 performs the various methods and processes described above, such as camera self-calibration methods.

[0170] In some embodiments, the camera self-calibration method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 418. In some embodiments, part or all of the computer program may be loaded and / or installed on the camera self-calibration device 410 via ROM 412 and / or communication unit 419. When the computer program is loaded into RAM 413 and executed by processor 411, one or more steps of the camera self-calibration method described above may be performed. Alternatively, in other embodiments, processor 411 may be configured to perform the camera self-calibration method by any other suitable means (e.g., by means of firmware).

[0171] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0172] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable camera self-calibration device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0173] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0174] To provide user interaction, the systems and techniques described herein can be implemented on a camera self-calibration device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the camera self-calibration device. Other types of devices can also be used to provide user interaction; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0175] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0176] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system. It addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability.

[0177] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0178] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A camera self-calibration method, characterized in that, The method includes: Based on the continuous image sequence acquired by the target camera, at least one adjacent image pair is constructed; the adjacent image pair includes two images; the two images in the adjacent image pair are acquired at adjacent times; For each adjacent image pair, determine the target basis matrix of the adjacent image pair; Based on the target basis matrix of each adjacent image pair, a target function based on the essential matrix manifold constraint is determined, and the target function is optimized to obtain the optimal intrinsic parameters of the target camera; Based on the optimal intrinsic parameters and the continuous image sequence, the pitch angle of the target camera is determined, and the target camera is a monocular camera; The step of determining the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence includes: For each image in the continuous image sequence, identify the scene depicted in that image; If the scene is a road scene, then the first road boundary line and the second road boundary line of the road mask in the image are extracted, and the intersection point between the first road boundary line and the second road boundary line is determined as the vanishing point; If the scene is a farmland scene, then extract the center line of the mask for each crop row in the image; The vanishing point is determined based on the intersections between the centerlines. Based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters, determine the camera pitch angle corresponding to the image; The pitch angle of the target camera is determined based on the camera pitch angle corresponding to each image.

2. The method according to claim 1, characterized in that, Determining the target basis matrix for each adjacent image pair includes: For each adjacent image pair, extract the feature matching point pair in the adjacent image pair; the feature matching point pair includes two matching feature points. Based on each of the aforementioned feature matching point pairs, determine the first fundamental matrix of the adjacent image pair; Based on the first fundamental matrix of the adjacent image pair, select the first matching point pair that satisfies the geometric constraints from each of the feature matching point pairs; Discard adjacent image pairs whose inlier ratio is less than a set threshold; the inlier ratio is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs. For each pair of neighboring images that is not discarded, the target basis matrix of the neighboring image pair is determined based on the first matching point pair of the neighboring image pair.

3. The method according to claim 2, characterized in that, The step of determining the objective function based on the essential matrix manifold constraint according to the target basis matrix of each of the adjacent image pairs includes: For each adjacent image pair, the essential matrix of the adjacent image pair is determined based on the target basis matrix of the adjacent image pair and the preset prior camera intrinsic parameters; the prior camera intrinsic parameters include prior horizontal focal length, prior vertical focal length, prior principal point horizontal coordinates and prior principal point vertical coordinates; Perform singular value decomposition on the essential matrix, and calculate the rank-constrained residual and singular value equality residual corresponding to the adjacent image pairs based on the singular value decomposition results; The target residual is obtained by fusing the rank-constrained residual and the singular value equal residual corresponding to each of the adjacent image pairs. Based on the residual weights and residual weight coefficients, the first focal length residual between the focal length of the target camera in the horizontal direction and the focal length in the vertical direction, the principal point prior residual between the principal point coordinates of the target camera and the preset prior principal point coordinates, and the second focal length residual between the focal length of the target camera and the preset prior focal length are determined. The target residual, the first focal length residual, the principal point prior residual, and the second focal length residual are fused to obtain the objective function based on the essential matrix manifold constraint.

4. The method according to claim 3, characterized in that, The optimization of the objective function to obtain the optimal intrinsic parameters of the target camera includes: The target residual weight value is calculated based on the number of adjacent image pairs and the number of first matching point pairs of each adjacent image pair; The residual weights in the objective function are assigned the target residual weight values, and the residual weight coefficients in the objective function are assigned preset values. The prior camera intrinsic parameters are iteratively optimized with the goal of minimizing the value of the objective function to obtain the first camera intrinsic parameters of the target camera. Based on the first camera intrinsic parameters and the prior camera intrinsic parameters, calculate the first residual weight coefficient value, and use the first residual weight coefficient value as an auxiliary coefficient value; The residual weights in the objective function are assigned the target residual weight values, and the residual weight coefficients in the objective function are assigned the auxiliary coefficient values. The prior camera intrinsic parameters are iterated with the objective function value minimized to obtain the second camera intrinsic parameters of the target camera. Calculate the second residual weight coefficient value based on the second camera intrinsic parameters and the prior camera intrinsic parameters; If the value of the second residual weight coefficient is less than or equal to the preset threshold, then the second camera intrinsic parameter is determined as the optimal intrinsic parameter; If the second residual weight coefficient value is greater than the preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the process returns to the step of assigning the residual weight in the objective function to the target residual weight value, and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iterating the prior camera intrinsic parameters with the objective function value as the minimum as the objective function value to obtain the second camera intrinsic parameters of the target camera, until the second residual weight coefficient value is less than or equal to the preset threshold.

5. The method according to claim 1, characterized in that, Based on a continuous sequence of images captured by the target camera, construct at least one adjacent image pair, including: Based on the continuous image sequence acquired by the target camera, at least one original image pair is constructed; the original image pair includes two images; the acquisition times of the two images in the original image pair are adjacent. For each original image pair, determine the structural similarity between the two images in the original image pair, and use the structural similarity as the structural similarity of the original image pair; Based on the structural similarity of each original image pair, adjacent image pairs are selected from each original image pair.

6. A camera self-calibration device, characterized in that, The device includes: An image pair construction module is used to construct at least one adjacent image pair based on a continuous image sequence acquired by a target camera; the adjacent image pair includes two images; the two images in the adjacent image pair were acquired at adjacent times; A matrix determination module is used to determine the target basis matrix of each adjacent image pair. The intrinsic parameter determination module is used to determine the objective function based on the essential matrix manifold constraint according to the target basis matrix of each adjacent image pair, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera; The pitch angle determination module is used to determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence, wherein the target camera is a monocular camera; The pitch angle determination module includes: A scene recognition unit is used to recognize the scene of each image in the continuous image sequence; The first point determination unit is used to extract the first road boundary line and the second road boundary line of the road mask in the image if the scene is a road scene, and determine the intersection point between the first road boundary line and the second road boundary line as the vanishing point; The centerline determination unit is used to extract the centerline of each crop row mask in the image if the scene is a farmland scene; The second point determination unit is used to determine the vanishing point based on the intersection points between the center lines. The first angle determination unit is used to determine the camera pitch angle corresponding to the image based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters; The second angle determination unit is used to calculate the average pitch angle of the camera corresponding to each image, and determine the average pitch angle as the pitch angle of the target camera.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a camera self-calibration method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the camera self-calibration method according to any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the camera self-calibration method according to any one of claims 1-5.

Images