Camera self-calibration method, device, equipment, medium and program product

Abstract

The invention discloses a camera self-calibration method, device and equipment, a medium and a program product, and relates to the technical field of camera self-calibration, and the method comprises the steps: constructing at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair comprises two images; the collection moments of the two images in the adjacent image pair are adjacent; for each adjacent image pair, determining a target basis matrix of the adjacent image pair; according to the target basic matrix of each adjacent image pair, determining a target function based on essential matrix manifold constraint, and optimizing the target function to obtain an optimal internal reference of the target camera; and determining the pitch angle of the target camera according to the optimal internal reference and the continuous image sequence. According to the embodiment of the invention, joint calibration of the internal reference and the pitch angle of the camera is realized, and pitch angle matching in the camera is ensured, so that the accuracy of visual navigation by using the internal reference and the pitch angle of the camera is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera self-calibration, and particularly to a camera self-calibration method, device, equipment, medium and program product. BACKGROUND

[0002] Camera calibration is a basic technology in the field of computer vision, and its purpose is to obtain the intrinsic and extrinsic parameters of a camera, and to provide an accurate geometric model for subsequent agricultural machine vision navigation applications.

[0003] In the prior art, the camera self-calibration method cannot solve the problem of camera pitch angle calibration, and the pitch angle needs to be determined separately, resulting in separate calibration of the camera intrinsic parameters and the pitch angle, and further causing the camera intrinsic parameters and the pitch angle to be mismatched, which affects the navigation accuracy of the subsequent agricultural machine. SUMMARY

[0004] The present application provides a camera self-calibration method, device, equipment, medium and program product to realize joint calibration of camera intrinsic parameters and pitch angle.

[0005] In a first aspect, an embodiment of the present application provides a camera self-calibration method, comprising:

[0006] According to a continuous image sequence collected by a target camera, at least one adjacent image pair is constructed; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent;

[0007] For each adjacent image pair, a target fundamental matrix of the adjacent image pair is determined;

[0008] According to the target fundamental matrix of each adjacent image pair, a target function based on an essential matrix manifold constraint is determined, and the target function is optimized to obtain optimal intrinsic parameters of the target camera;

[0009] According to the optimal intrinsic parameters and the continuous image sequence, the pitch angle of the target camera is determined.

[0010] In a second aspect, an embodiment of the present application further provides a camera self-calibration device, comprising:

[0011] An image pair construction module is configured to construct at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent;

[0012] A matrix determination module is configured to determine, for each adjacent image pair, a target fundamental matrix of the adjacent image pair;

[0013] The inner parameter determination module is configured to determine a target function based on the essential matrix manifold constraint according to the target fundamental matrix of each of the adjacent image pairs, and optimize the target function to obtain the optimal inner parameter of the target camera.

[0014] The pitch angle determination module is configured to determine the pitch angle of the target camera according to the optimal inner parameter and the continuous image sequence.

[0015] In a third aspect, an electronic device is provided, including:

[0016] at least one processor; and

[0017] a memory in communication with the at least one processor; wherein

[0018] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the camera self-calibration method provided by any of the embodiments of the present application.

[0019] In a fourth aspect, a computer readable storage medium is provided, and the computer readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the camera self-calibration method of any of the embodiments of the present application when executed by the processor.

[0020] In a fifth aspect, a computer program product is provided, and the computer program product includes a computer program, and the computer program is used to implement the camera self-calibration method of any of the embodiments of the present application when executed by a processor.

[0021] According to the continuous image sequence collected by the target camera, at least one adjacent image pair is constructed; the adjacent image pair includes two images; the two images in the adjacent image pair are adjacent in time; for each adjacent image pair, a target fundamental matrix of the adjacent image pair is determined; a target function based on the essential matrix manifold constraint is determined according to the target fundamental matrix of each of the adjacent image pairs, and the target function is optimized to obtain the optimal inner parameter of the target camera; the pitch angle of the target camera is determined according to the optimal inner parameter and the continuous image sequence, which can determine the camera inner parameter, and then determine the camera pitch angle according to the camera inner parameter, realize the joint calibration of the camera inner parameter and the pitch angle, ensure the matching of the camera inner parameter and the pitch angle, and thus improve the accuracy of visual navigation using the camera inner parameter and the pitch angle.

[0022] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiments description. Obviously, the drawings in the following description only show some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0024] Figure 1 is a flow chart of a camera self-calibration method according to an embodiment of the present application;

[0025] Figure 2 is a flow chart of a camera self-calibration method according to an embodiment of the present application;

[0026] Figure 3 is a structural schematic diagram of a camera self-calibration device according to an embodiment of the present application;

[0027] Figure 4 is a structural diagram of an electronic device for implementing a camera self-calibration method according to an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the person skilled in the art better understand the present application, the following will combine the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0029] It should be noted that the terms "first" and "second" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] In the technical solutions of the embodiments of the present application, the acquisition, storage and application of the continuous image sequence and the like are in line with the relevant legal regulations and do not violate public order and good customs.

[0031] Embodiment one

[0032] Figure 1 A flowchart of a camera self-calibration method provided for Embodiment One of the present application, which can be applicable to the calibration of camera intrinsic parameters and pitch angle, can be executed by a camera self-calibration device, which can be realized in the form of hardware and / or software and specifically configured in an electronic device.

[0033] Referring to the camera self-calibration method shown in Figure 1 includes the following steps:

[0034] S101. Construct at least one adjacent image pair according to a continuous image sequence captured by a target camera; the adjacent image pair includes two images; the two images in the adjacent image pair are adjacent in time.

[0035] S102. Determine a target fundamental matrix of each adjacent image pair.

[0036] S103. Determine a target function based on an essential matrix manifold constraint according to the target fundamental matrix of each adjacent image pair, and optimize the target function to obtain optimal intrinsic parameters of the target camera.

[0037] S104. Determine the pitch angle of the target camera according to the optimal intrinsic parameters and the continuous image sequence.

[0038] In this embodiment, the target camera can be a camera whose intrinsic parameters and pitch angle are to be calibrated. The continuous image sequence can be at least two images continuously captured by the target camera at a set frame rate, so as to ensure that adjacent images in the continuous image sequence avoid excessive motion difference and have sufficient parallax. It should be noted that the set frame rate can be set by the technician according to actual needs or practical experience, and the present application does not limit this. The target fundamental matrix can be used to represent the epipolar geometric relationship between the two images in the adjacent image pair. The optimal intrinsic parameters can be the camera intrinsic parameters obtained after optimizing the target function. The camera intrinsic parameters can include, but are not limited to, at least one of the horizontal focal length, the vertical focal length, the horizontal coordinate of the principal point, and the vertical coordinate of the principal point.

[0039] In an optional embodiment, the target camera is a monocular camera fixedly installed on a farm vehicle body and faces the driving direction of the farm vehicle, continuously captures images at a fixed frame rate during the driving of the farm vehicle, and obtains a continuous image sequence.

[0040] Specifically, each two adjacent images in the continuous image sequence are determined as an adjacent image pair; for each adjacent image pair, a feature point pair matched in the two images of the adjacent image pair is extracted by a feature extraction algorithm such as a SIFT (Scale-Invariant Feature Transform) algorithm or an ORB (Oriented FAST and Rotated BRIE) algorithm; a feature point pair with a descriptor distance less than a preset threshold is determined as a weak feature point pair, the weak feature point pairs are removed, and a feature point pair with a descriptor distance greater than or equal to the preset threshold is reserved; a target fundamental matrix between the two images in the adjacent image pair is calculated according to the reserved feature point pair by using a RANSAC (Random Sample Consensus) algorithm; for the target fundamental matrix of each adjacent image pair, an essential matrix corresponding to the adjacent image pair is determined; an essential matrix manifold constraint term corresponding to the adjacent image pair is determined; the essential matrix manifold constraint terms corresponding to the adjacent image pairs are fused to obtain a target function based on the essential matrix manifold constraint, and the least square method is used to optimize the target function to solve the optimal intrinsic parameter of the target camera; and a certain algorithm is used to determine the pitch angle of the target camera according to the optimal intrinsic parameter and the continuous image sequence.

[0041] Optionally, at least one adjacent image pair is constructed according to the continuous image sequence collected by the target camera, including: at least one original image pair is constructed according to the continuous image sequence collected by the target camera; the original image pair includes two images; the two images in the original image pair are adjacent in collection time; for each original image pair, a structural similarity between the two images in the original image pair is determined, and the structural similarity is taken as a structural similarity of the original image pair; and the adjacent image pairs are selected from the original image pairs according to the structural similarities of the original image pairs.

[0042] Specifically, each two adjacent images in the continuous image sequence are determined as an original image pair; for each original image pair, a pixel value mean and a pixel value variance of the two images in the original image pair are calculated, and a pixel value covariance between the two images in the original image pair is calculated; a structural similarity between the two images in the original image pair is calculated by the pixel value mean and the pixel value variance of the two images in the original image pair and the pixel value covariance between the two images in the original image pair, and the structural similarity is taken as a structural similarity of the original image pair; and the structural similarity can be determined by the following formula, for example:

[0043] ;

[0044] wherein, denotes a structural similarity; denotes a mean value of pixel values of one image in the original image pair, denotes a variance of pixel values of the image; denotes a mean value of pixel values of the other image in the original image pair, denotes a variance of pixel values of the image; denotes a luminance constant; denotes a contrast constant; denotes a covariance of pixel values between the two images in the original image pair.

[0045] The original image pair with the structural similarity in the preset interval is determined as the adjacent image pair; it should be noted that the preset interval can be set by the technical personnel according to actual needs or practical inspection, and the present application does not limit this.

[0046] It can be understood that, according to the above scheme, at least one original image pair is constructed according to the continuous image sequence collected by the target camera, and the structural similarity of each original image pair is calculated; the original image pair with the structural similarity in the preset interval is determined as the adjacent image pair, which can remove the original image pair with high structural similarity (small image difference) and the original image pair with low structural similarity (large motion or blur), thereby ensuring that the geometric constraint of the adjacent image is effective and the matching is stable, and improving the stability and success rate of subsequent intrinsic parameter calibration.

[0047] Optionally, the pitch angle of the target camera is determined according to the optimal intrinsic parameter and the continuous image sequence, comprising: for each image in the continuous image sequence, identifying the scene of the image; if the scene is a road scene, extracting a first road boundary line and a second road boundary line of a road mask in the image, and determining the intersection between the first road boundary line and the second road boundary line as a vanishing point; if the scene is a farmland scene, extracting a center line of each crop row mask in the image; determining the vanishing point according to the intersection between each center line; determining the camera pitch angle corresponding to the image according to the vertical coordinate of the vanishing point and the optimal intrinsic parameter; and determining the pitch angle of the target camera according to the camera pitch angles corresponding to each image.

[0048] wherein, the first road boundary line and the second road boundary line can be the road boundary lines at both ends of the road mask. Specifically, for each image in the continuous image sequence, the road mask or the crop row mask in the image can be identified through a target detection model; if the road mask is identified in the image, it is determined that the scene of the image is a road scene; if the road mask is not identified in the image, and the crop row mask is identified, it is determined that the scene of the image is a farmland scene.

[0049] If the scene is a road scene, boundary lines at both ends of the road mask are extracted, and vertical de-duplication is performed; a least squares method or a RANSAC algorithm or other line fitting algorithm is used to fit the first road boundary line and the second road boundary line respectively; the intersection of the first road boundary line and the second road boundary line is calculated as the vanishing point; if the scene is a farmland scene, a line fitting algorithm is used to fit the center line of each crop row mask respectively, and the intersection between each two center lines is calculated; according to the horizontal coordinates and vertical coordinates of each intersection point in the image, the median of the horizontal coordinates and the median of the vertical coordinates are determined; the intersection point with the horizontal coordinate being the median of the horizontal coordinates and the vertical coordinate being the median of the vertical coordinates is determined as the vanishing point; according to the vertical coordinate of the vanishing point and the vertical focal length and the principal point vertical coordinate in the optimal intrinsic parameter, the camera pitch angle corresponding to the image is determined; for example, the camera pitch angle can be determined by the following formula:

[0050] ;

[0051] wherein, represents the camera pitch angle corresponding to the i-th image; represents the vertical coordinate of the vanishing point in the i-th image; represents the principal point vertical coordinate; represents the vertical focal length. The mean value of the camera pitch angles corresponding to each of the images is calculated, and the mean value of the camera pitch angles is determined as the pitch angle of the target camera.

[0052] It can be understood that by using the above technical solution, after the optimal intrinsic parameter is determined, the pitch angle can be determined in a unified process without the need for a calibration board and a special sensor, realizing online, low-cost and high-robustness joint self-calibration of camera intrinsic parameters and pitch angle, and realizing pitch angle self-calibration in road scenes or farmland scenes, which widens the applicable scenarios of camera intrinsic parameter and pitch angle self-calibration.

[0053] It can be understood that by using the above technical solution, after the optimal intrinsic parameter is determined, the pitch angle can be determined in a unified process without the need for a calibration board and a special sensor, realizing online, low-cost and high-robustness joint self-calibration of camera intrinsic parameters and pitch angle, and realizing pitch angle self-calibration in road scenes or farmland scenes, which widens the applicable scenarios of camera intrinsic parameter and pitch angle self-calibration.

[0054] ​The embodiment of the application constructs at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent; for each adjacent image pair, a target fundamental matrix of the adjacent image pair is determined; a target function based on an essential matrix manifold constraint is determined according to the target fundamental matrix of each adjacent image pair, and the target function is optimized to obtain an optimal intrinsic parameter of the target camera; and the pitch angle of the target camera is determined according to the optimal intrinsic parameter and the continuous image sequence. After the camera intrinsic parameter is determined, the camera pitch angle is continuously determined according to the camera intrinsic parameter, the joint calibration of the camera intrinsic parameter and the pitch angle is realized, the camera intrinsic parameter and the pitch angle are matched, and therefore the accuracy of visual navigation using the camera intrinsic parameter and the pitch angle is improved.

[0055] Embodiment two

[0056] Figure 2 A flowchart of a camera self-calibration method provided by the embodiment two of the application is shown in the figure, and the embodiment of the application optimizes and improves the determination operation of the target fundamental matrix on the basis of the technical solutions of the above-mentioned embodiments.

[0057] Further, the determination of the target fundamental matrix is refined into "for each adjacent image pair, extracting a feature matching point pair in the adjacent image pair; the feature matching point pair comprises two matched feature points; determining a first fundamental matrix of the adjacent image pair according to each feature matching point pair; from each feature matching point pair, screening a first matching point pair satisfying a geometric constraint according to the first fundamental matrix of the adjacent image pair; discarding an adjacent image pair with an inlier ratio less than a set threshold; the inlier ratio is a ratio between the number of the first matching point pair and the number of the feature matching point pair; for each un-discarded adjacent image pair, determining a target fundamental matrix of the adjacent image pair according to the first matching point pair of the adjacent image pair", so as to perfect the determination operation of the target fundamental matrix.

[0058] It should be noted that the parts not described in detail in the embodiment of the application can refer to the descriptions of the foregoing embodiments.

[0059] Referring to Figure 2 The camera self-calibration method comprises:

[0060] S201, constructing at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent.

[0061] S202, for each adjacent image pair, extracting a feature matching point pair in the adjacent image pair; the feature matching point pair comprises two matched feature points.

[0062] S203. Determine the first basis matrix of the adjacent image pair based on each of the feature matching point pairs.

[0063] S204. Based on the first fundamental matrix of the adjacent image pair, select the first matching point pair that satisfies the geometric constraints from each of the feature matching point pairs.

[0064] S205. Discard adjacent image pairs whose inlier ratio is less than a set threshold; the inlier ratio is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs.

[0065] S206. For each pair of adjacent images that has not been discarded, determine the target basis matrix of the adjacent image pair based on the first matching point pair of the adjacent image pair.

[0066] S207. Based on the target basis matrix of each adjacent image pair, determine the objective function based on the essential matrix manifold constraint, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera.

[0067] S208. Determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

[0068] In this embodiment, the first fundamental matrix is ​​the fundamental matrix determined based on the feature matching point pairs extracted from adjacent image pairs; the target fundamental matrix can be the fundamental matrix determined based on the first matching point pairs.

[0069] Specifically, for each adjacent image pair, feature extraction algorithms such as SIFT (Scale-Invariant Feature Transform) or ORB (Oriented Fast and Rotated BRIE) are used to extract matching feature point pairs from the two images of the adjacent image pair. Feature point pairs with a descriptor distance less than a preset threshold are identified as weak feature point pairs, and these weak feature point pairs are removed, retaining those with a descriptor distance greater than or equal to the preset threshold. The RANSAC (Random Sample Consensus) algorithm is used to calculate the first fundamental matrix between the two images in the adjacent image pair based on the retained feature point pairs. Based on the first fundamental matrix of the adjacent image pair, the first matching point pair satisfying the geometric constraints is selected from each of the feature matching point pairs. For example, the geometric constraints of the adjacent image pair can be represented by the following formula:

[0070] ;

[0071] in, Indicates the first The homogeneous coordinates of a matching point in a feature matching point pair of adjacent image pairs. Indicates the first Transpose of the homogeneous coordinates of another matching point in a pair of adjacent image feature matching points; Indicates the first The first fundamental matrix corresponding to each image pair;

[0072] The feature matching point pairs that satisfy the geometric constraints are identified as the first matching point pairs; the ratio between the number of first matching point pairs and the number of feature matching point pairs in the adjacent image pairs is determined; adjacent image pairs with an inlier ratio less than a set threshold are discarded; for each adjacent image pair that is not discarded, the target basis matrix of the adjacent image pair is determined based on the first matching point pairs of the adjacent image pair.

[0073] It should be noted that the threshold can be set independently by technical personnel based on actual needs or practical experience, and this invention does not limit this; the process of determining the target basis matrix based on the first matching point pair is similar to the process of determining the first basis matrix based on the feature matching point pair, and will not be described in detail here.

[0074] Optionally, determining the objective function based on the essential matrix manifold constraint according to the target basis matrix of each of the adjacent image pairs includes: for each adjacent image pair, determining the essential matrix of the adjacent image pair according to the target basis matrix of the adjacent image pair and preset prior camera intrinsic parameters; the prior camera intrinsic parameters include prior horizontal focal length, prior vertical focal length, prior principal point horizontal coordinates, and prior principal point vertical coordinates; performing singular value decomposition on the essential matrix, and calculating the rank constraint residual and singular value equality residual corresponding to the adjacent image pair according to the singular value decomposition results; for each of the adjacent image pairs... The neighboring images are fused together with the corresponding rank-constrained residuals and singular value equality residuals to obtain the target residual. Based on the residual weights and residual weight coefficients, the first focal length residual between the target camera's focal length in the horizontal direction and its focal length in the vertical direction, the principal point prior residual between the target camera's principal point coordinates and the preset prior principal point coordinates, and the second focal length residual between the target camera's focal length and the preset prior focal length are determined. The target residual, the first focal length residual, the principal point prior residual, and the second focal length residual are fused together to obtain the objective function based on the essential matrix manifold constraint.

[0075] The preset prior camera intrinsic parameters can be pre-defined prior values ​​of the camera intrinsic parameters; the camera intrinsic parameters can be represented in matrix form; for example, the camera intrinsic parameter matrix can be represented by the following matrix:

[0076] ;

[0077] in, Indicates the horizontal focal length; This represents the horizontal coordinates of the principal point.

[0078] The residual weighting coefficients may include the first focal length residual weighting coefficient, the second focal length residual weighting coefficient, the first principal point residual weighting coefficient, and the second principal point residual weighting coefficient.

[0079] The singular value decomposition results can include the first singular value, the second singular value, and the third singular value; the rank constraint residual and the singular value equality residual are used to characterize the degree of deviation of the essential matrix from the manifold constraint; the rank constraint residual can be used to characterize the degree of satisfaction of the rank constraint; the singular value equality residual can be used to characterize the degree of satisfaction of the singular value equality.

[0080] Specifically, for each adjacent image pair, the essential matrix of the adjacent image pair is determined based on the target basis matrix of the adjacent image pair and the preset prior camera intrinsic parameters; for example, the essential matrix of the adjacent image pair can be determined by the following formula:

[0081] ;

[0082] in, Indicates the first The essential matrix of adjacent image pairs; Indicates the first The target basis matrix of adjacent image pairs; This represents the transpose of the camera intrinsic parameter matrix.

[0083] Singular value decomposition is performed on the essential matrix of the adjacent image pair to obtain the first, second, and third singular values ​​corresponding to the adjacent image pair; the rank-constrained residual is determined based on the first, second, and third singular values; for example, the rank-constrained residual can be determined by the following formula:

[0084] ;

[0085] in, Indicates the first Rank-constrained residuals of adjacent image pairs; Indicates the first The third singular value of a pair of adjacent images; Indicates the first The first singular value of a pair of adjacent images; Indicates the first The second singular value of a pair of adjacent images.

[0086] The singular value equality residual is determined based on the first and second singular values; for example, the singular value equality residual can be determined using the following formula:

[0087] ;

[0088] wherein, denotes the singular value equalization residual of the i-th adjacent image pair.

[0089] The rank constraint residual and the singular value equalization residual corresponding to each of the adjacent image pairs are fused to obtain a target residual; according to the residual weight and the residual weight coefficient, a first focal length residual between the focal length of the target camera in the horizontal direction and the focal length of the target camera in the vertical direction, a principal point prior residual between the principal point coordinate of the target camera and a preset prior principal point coordinate, and a second focal length residual between the focal length of the target camera and a preset prior focal length are determined; for example, the first focal length residual can be determined by the following formula:

[0090] ;

[0091] wherein, denotes the first focal length residual; denotes the residual weight; denotes the first focal length residual weight coefficient.

[0092] The principal point prior residual can include a principal point horizontal prior residual and a principal point vertical prior residual; for example, the principal point horizontal prior residual can be determined by the following formula:

[0093] ;

[0094] wherein, denotes the principal point horizontal prior residual; denotes the prior horizontal coordinate of the principal point; denotes the width of the image; denotes the first principal point residual weight coefficient;

[0095] For example, the principal point vertical prior residual can be determined by the following formula:

[0096] ;

[0097] wherein, denotes the principal point vertical prior residual; denotes the prior vertical coordinate of the principal point; denotes the height of the image; denotes the second principal point residual weight coefficient;

[0098] For example, the second focal length residual can be determined by the following formula:

[0099] ;

[0100] wherein, denotes the second focal length residual;​ represents a prior horizontal focal length; represents a prior vertical focal length; represents a second focal length residual weight coefficient;

[0101] fusing the target residual, the first focal length residual, the principal point prior residual and the second focal length residual to obtain a target function based on an essential matrix manifold constraint; an exemplary target function can be represented by the following formula:

[0102]

[0103] wherein, represents a target function; represents a target residual; represents a number of adjacent image pairs.

[0104] It can be understood that by using the above technical solution, the essential matrix is determined, the singularity of the essential matrix is solved, the direct mapping of the camera internal parameter and the image geometry relationship is constructed relying on the algebraic characteristics that the rank of the essential matrix is 2 and the first singular value and the second singular value are equal, the error accumulation and transmission caused by the indirect constraint in the prior art are avoided, and the accuracy of the subsequent determined internal parameter is improved; by determining the regularization constraint term including the first focal length residual, the principal point horizontal prior residual, the principal point vertical prior residual and the second focal length residual, the subsequent optimization process of the target function is guided to converge to a reasonable direction through the prior knowledge when the number and quality of the adjacent image pairs are insufficient; when the number and quality of the adjacent image pairs are good, the influence of the regularization constraint term is small, and excessive constraint is avoided.

[0105] ​Optionally, the optimization of the target function to obtain the optimal intrinsic parameter of the target camera comprises: calculating a target residual weight value according to the number of the adjacent image pairs and the number of the first matching point pairs of each of the adjacent image pairs; assigning a residual weight in the target function with the target residual weight value, assigning a residual weight coefficient in the target function with a preset value, and iteratively optimizing the prior camera intrinsic parameter to obtain a first camera intrinsic parameter of the target camera, with the minimum value of the target function as the target; calculating a first residual weight coefficient value according to the first camera intrinsic parameter and the prior camera intrinsic parameter, and taking the first residual weight coefficient value as an auxiliary coefficient value; assigning the residual weight in the target function with the target residual weight value, assigning the residual weight coefficient in the target function with the auxiliary coefficient value, and iteratively optimizing the prior camera intrinsic parameter to obtain a second camera intrinsic parameter of the target camera, with the minimum value of the target function as the target; calculating a second residual weight coefficient value according to the second camera intrinsic parameter and the prior camera intrinsic parameter; if the second residual weight coefficient value is less than or equal to a preset threshold, determining the second camera intrinsic parameter as the optimal intrinsic parameter; if the second residual weight coefficient value is greater than the preset threshold, updating the auxiliary coefficient value to the second residual weight coefficient value, and returning to execute the step of assigning the residual weight in the target function with the target residual weight value, assigning the residual weight coefficient in the target function with the auxiliary coefficient value, and iteratively optimizing the prior camera intrinsic parameter to obtain a second camera intrinsic parameter of the target camera, with the minimum value of the target function as the target, until the second residual weight coefficient value is less than or equal to the preset threshold.

[0106] The first residual weight coefficient value comprises a first coefficient value corresponding to a first focal length residual weight coefficient, a second coefficient value corresponding to a second focal length residual weight coefficient, a third coefficient value corresponding to a first principal point residual weight coefficient, and a fourth coefficient value corresponding to a second principal point residual weight coefficient.

[0107] The auxiliary coefficient value comprises an auxiliary coefficient value corresponding to the first focal length residual weight coefficient, an auxiliary coefficient value corresponding to the second focal length residual weight coefficient, an auxiliary coefficient value corresponding to the first principal point residual weight coefficient, and an auxiliary coefficient value corresponding to the second principal point residual weight coefficient.

[0108] Specifically, a first average number between the first matching point pairs of each of the adjacent image pairs is determined; a target residual weight value is calculated according to the number of the adjacent image pairs and the first average number; and exemplarily, a target residual weight value is calculated according to the number of the adjacent image pairs and the number of the first matching point pairs of each of the adjacent image pairs:

[0109] ;

[0110] ;

[0111] ;

[0112] wherein, represents a target residual weight value; represents a first average number; represents a number of adjacent image pairs; 、 and represents a set coefficient constant.

[0113] The residual weight in the target function is assigned to the target residual weight value, and the residual weight coefficient in the target function is assigned to a preset value, and the first camera internal parameter of the prior camera is iteratively optimized with the value of the target function being minimum as the target, to obtain the first camera internal parameter of the target camera; in one specific embodiment, the preset value is 1.

[0114] According to the first camera internal parameter and the prior camera internal parameter, a first deviation between a horizontal focal length and a vertical focal length in the first camera internal parameter is calculated; a second deviation between a horizontal coordinate of a principal point in the first camera internal parameter and a prior horizontal coordinate of the principal point in the prior camera internal parameter is calculated, and a third deviation between a vertical coordinate of the principal point in the first camera internal parameter and a prior vertical coordinate in the prior camera internal parameter is calculated; a first focal length sum between the horizontal focal length and the vertical focal length in the first camera internal parameter is calculated, and a second focal length sum between the horizontal focal length and the vertical focal length in the prior camera internal parameter is calculated; a fourth deviation between the first focal length sum and the second focal length sum is calculated;

[0115] According to the first deviation, a first coefficient value corresponding to a first focal length residual weight coefficient is determined; according to the second deviation, a second coefficient value corresponding to a second focal length residual weight coefficient is determined; according to the third deviation, a third coefficient value corresponding to a first principal point residual weight coefficient is determined; according to the fourth deviation, a fourth coefficient value corresponding to a second principal point residual weight coefficient is determined; and the first residual weight coefficient value is taken as an auxiliary coefficient value.

[0116] For example, the first coefficient value can be determined by the following formula:

[0117] ;

[0118] wherein, represents the first coefficient value; represents a set constant; represents the first deviation; represents a set lower threshold value of the deviation of the first focal length residual weight coefficient, represents a set upper threshold value of the deviation of the first focal length residual weight coefficient.

[0119] It should be noted that the determination process of the second coefficient value, the third coefficient value and the fourth coefficient value is similar to the determination process of the first coefficient value, which will not be described here. The lower limit threshold of the deviation between the first focal length residual weight coefficient, the second focal length residual weight coefficient, the first principal point residual weight coefficient and the second principal point residual weight coefficient is different, and the upper limit threshold of the deviation is also different.

[0120] The residual weight in the target function is assigned to the target residual weight value, and the residual weight coefficient in the target function is assigned to the auxiliary coefficient value. The prior camera parameters are iterated to obtain the second camera parameters of the target camera, with the value of the target function being the minimum as the target. The second residual weight coefficient value is calculated according to the second camera parameters and the prior camera parameters by using a similar process to determining the first residual weight coefficient value, which will not be described here. If each item in the second residual weight coefficient value is less than or equal to a preset threshold, the second camera parameters are determined as the optimal parameters. If one item in the second residual weight coefficient value is greater than the preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the step of assigning the residual weight in the target function to the target residual weight value and assigning the residual weight coefficient in the target function to the auxiliary coefficient value is executed again, and the prior camera parameters are iterated to obtain the second camera parameters of the target camera, with the value of the target function being the minimum as the target. This process is repeated until each item in the second residual weight coefficient value is less than or equal to the preset threshold.

[0121] In an optional embodiment, the preset threshold can include a first threshold corresponding to the first focal length residual weight coefficient, a second threshold corresponding to the second focal length residual weight coefficient, a third threshold corresponding to the first principal point residual weight coefficient, and a fourth threshold corresponding to the second principal point residual weight coefficient. The first threshold, the second threshold, the third threshold and the fourth threshold can be different, or at least two of them are the same. Correspondingly, if each item in the second residual weight coefficient value is less than or equal to the corresponding preset threshold, the second camera parameters are determined as the optimal parameters. If one item in the second residual weight coefficient value is greater than the corresponding preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the step of assigning the residual weight in the target function to the target residual weight value and assigning the residual weight coefficient in the target function to the auxiliary coefficient value is executed again, and the prior camera parameters are iterated to obtain the second camera parameters of the target camera, with the value of the target function being the minimum as the target. This process is repeated until each item in the second residual weight coefficient value is less than or equal to the corresponding preset threshold.

[0122] In an optional embodiment, while calculating the first residual weight coefficient value, the RANSAC algorithm can also be used to re-determine the first matching point pairs in each adjacent image pair according to the first camera internal parameter to obtain more accurate first matching point pairs.

[0123] It can be understood that, by using the technical solution, the coarse solution of the camera internal parameter, i.e., the first camera internal parameter, is determined first, and then the objective function is optimized again by refining the inliers and the dynamically adjusted residual weight coefficient, so as to improve the convergence and the reliability of the result. Through the dynamic adjustment of the residual weight coefficient in the iteration process, the data constraint and the prior constraint are automatically balanced according to the data quality difference and the parameter deviation degree, which can solve the contradiction between the divergence caused by too weak prior and the convergence to the wrong solution caused by too strong prior, and improve the convergence stability and the reliability of the determined optimal internal parameter.

[0124] According to the embodiment of the application, for each adjacent image pair, the feature matching point pairs in the adjacent image pair are extracted; the feature matching point pairs include two matched feature points; the first fundamental matrix of the adjacent image pair is determined according to each feature matching point pair; the first matching point pairs satisfying the geometric constraint are selected from each feature matching point pair according to the first fundamental matrix of the adjacent image pair; the adjacent image pairs with an inlier proportion less than a set threshold are discarded; the inlier proportion is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs; for each un-discarded adjacent image pair, the target fundamental matrix of the adjacent image pair is determined according to the first matching point pairs of the adjacent image pair, which can filter out the adjacent image pairs with poor data effectiveness by feature extraction and feature point matching and filtering the adjacent image pairs according to the inlier proportion, so as to improve the data quality and further improve the efficiency of optimizing the objective function.

[0125] Embodiment three

[0126] Figure 3 A structural schematic diagram of a camera self-calibration device provided by the embodiment three of the application. The embodiment of the application can be applicable to the case of calibrating the camera internal parameter and the pitch angle. The device can execute the camera self-calibration method. The camera self-calibration device can be realized in the form of hardware and / or software. The device can be configured in an electronic device.

[0127] Referring to Figure 3 The camera self-calibration device shown in the figure includes an image pair construction module 301, a matrix determination module 302, an internal parameter determination module 303, and a pitch angle determination module 304, wherein,

[0128] The image pair construction module 301 is configured to construct at least one adjacent image pair according to a continuous image sequence collected by a target camera; the adjacent image pair includes two images; the two images in the adjacent image pair are adjacent in time of collection;

[0129] a matrix determining module 302, configured to determine, for each adjacent image pair, a target fundamental matrix of the adjacent image pair;

[0130] a parameter determining module 303, configured to determine a target function based on an essential matrix manifold constraint according to the target fundamental matrix of each adjacent image pair, and optimize the target function to obtain an optimal intrinsic parameter of the target camera;

[0131] a pitch angle determining module 304, configured to determine a pitch angle of the target camera according to the optimal intrinsic parameter and the continuous image sequence.

[0132] The embodiment of the present application constructs at least one adjacent image pair according to a continuous image sequence collected by a target camera through an image pair constructing module; the adjacent image pair comprises two images; the collection time of the two images in the adjacent image pair is adjacent; the embodiment of the present application determines, for each adjacent image pair, a target fundamental matrix of the adjacent image pair through a matrix determining module; the embodiment of the present application determines a target function based on an essential matrix manifold constraint according to the target fundamental matrix of each adjacent image pair through an intrinsic parameter determining module, and optimizes the target function to obtain an optimal intrinsic parameter of the target camera; the embodiment of the present application determines a pitch angle of the target camera according to the optimal intrinsic parameter and the continuous image sequence through a pitch angle determining module, which can determine the camera pitch angle according to the camera intrinsic parameter after the camera intrinsic parameter is determined, realizes the joint calibration of the camera intrinsic parameter and the pitch angle, ensures the matching of the camera intrinsic parameter and the pitch angle, and thus improves the accuracy of visual navigation using the camera intrinsic parameter and the pitch angle.

[0133] Optionally, the matrix determining module comprises:

[0134] an extracting unit, configured to extract, for each adjacent image pair, a feature matching point pair in the adjacent image pair; the feature matching point pair comprises two matched feature points;

[0135] a first determining unit, configured to determine, according to each feature matching point pair, a first fundamental matrix of the adjacent image pair;

[0136] a screening unit, configured to screen, according to the first fundamental matrix of the adjacent image pair, a first matching point pair satisfying a geometric constraint from each feature matching point pair;

[0137] a discarding unit, configured to discard an adjacent image pair with an inlier ratio less than a set threshold; the inlier ratio is a ratio between the number of the first matching point pair and the number of the feature matching point pair;

[0138] a second determining unit, configured to determine, for each un-discarded adjacent image pair, a target fundamental matrix of the adjacent image pair according to the first matching point pair of the adjacent image pair.

[0139] Optionally, the intrinsic parameter determination module 303 comprises:

[0140] The first matrix determination unit is configured to determine, for each adjacent image pair, an essential matrix of the adjacent image pair according to a target fundamental matrix of the adjacent image pair and a preset prior camera intrinsic parameter; the prior camera intrinsic parameter comprises a prior horizontal focal length, a prior vertical focal length, a prior principal point horizontal coordinate and a prior principal point vertical coordinate.

[0141] The first residual calculation unit is configured to perform singular value decomposition on the essential matrix, and calculate a rank constraint residual and a singular value equality residual corresponding to the adjacent image pair according to a singular value decomposition result.

[0142] The first fusion unit is configured to fuse the rank constraint residual and the singular value equality residual corresponding to each adjacent image pair to obtain a target residual.

[0143] The second residual calculation unit is configured to determine, according to a residual weight and a residual weight coefficient, a first focal length residual between a focal length of the target camera in a horizontal direction and a focal length of the target camera in a vertical direction, a principal point prior residual between a principal point coordinate of the target camera and a preset prior principal point coordinate, and a second focal length residual between a focal length of the target camera and a preset prior focal length.

[0144] The second fusion unit is configured to fuse the target residual, the first focal length residual, the principal point prior residual and the second focal length residual to obtain a target function based on an essential matrix manifold constraint.

[0145] Optionally, the intrinsic parameter determination module 303 comprises:

[0146] The weight calculation unit is configured to calculate a target residual weight value according to a number of the adjacent image pairs and a number of the first matching point pairs of each adjacent image pair.

[0147] The first optimization unit is configured to assign a residual weight in the target function to the target residual weight value, assign a residual weight coefficient in the target function to a preset value, and perform iterative optimization on the prior camera intrinsic parameter to obtain a first camera intrinsic parameter of the target camera, with a value of the target function being minimized as a target.

[0148] The first coefficient calculation unit is configured to calculate a first residual weight coefficient value according to the first camera intrinsic parameter and the prior camera intrinsic parameter, and take the first residual weight coefficient value as an auxiliary coefficient value.

[0149] a second optimization unit, configured to assign a residual weight in the target function to the target residual weight value, and assign a residual weight coefficient in the target function to the auxiliary coefficient value, and iteratively optimize the prior camera intrinsic parameter to obtain a second camera intrinsic parameter of the target camera, with a value of the target function being minimized as a target;

[0150] a second coefficient calculation unit, configured to calculate a second residual weight coefficient value according to the second camera intrinsic parameter and the prior camera intrinsic parameter;

[0151] an intrinsic parameter determination unit, configured to determine the second camera intrinsic parameter as an optimal intrinsic parameter if the second residual weight coefficient value is less than or equal to a preset threshold value;

[0152] a loop unit, configured to update the auxiliary coefficient value to the second residual weight coefficient value if the second residual weight coefficient value is greater than the preset threshold value, and return to execute the step of assigning the residual weight in the target function to the target residual weight value, and assigning the residual weight coefficient in the target function to the auxiliary coefficient value, and iteratively optimizing the prior camera intrinsic parameter to obtain the second camera intrinsic parameter of the target camera, until the second residual weight coefficient value is less than or equal to the preset threshold value.

[0153] Optionally, the image pair construction module 301 comprises:

[0154] construct at least one original image pair according to a continuous image sequence captured by the target camera; the original image pair comprises two images; the two images in the original image pair are adjacent in time of capture;

[0155] determine a structural similarity between the two images in each original image pair, and take the structural similarity as a structural similarity of the original image pair;

[0156] select adjacent image pairs from the original image pairs according to the structural similarities of the original image pairs.

[0157] Optionally, the pitch angle determination module 304 comprises:

[0158] a scene identification unit, configured to identify a scene of each image in the continuous image sequence;

[0159] a first point determination unit, configured to extract a first road boundary line and a second road boundary line of a road mask in the image if the scene is a road scene, and determine an intersection between the first road boundary line and the second road boundary line as a vanishing point;

[0160] The centerline determination unit is used to extract the centerline of each crop row mask in the image if the scene is a farmland scene;

[0161] The second point determination unit is used to determine the vanishing point based on the intersection points between the center lines.

[0162] The first angle determination unit is used to determine the camera pitch angle corresponding to the image based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters;

[0163] The second angle determination unit is used to determine the pitch angle of the target camera based on the camera pitch angle corresponding to each image.

[0164] The camera self-calibration device provided in the embodiments of the present invention can execute the camera self-calibration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the camera self-calibration method.

[0165] Example 4

[0166] Figure 4 A schematic diagram of a camera self-calibration device 410, which can be used to implement embodiments of the present invention, is shown. The camera self-calibration device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The camera self-calibration device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0167] like Figure 4 As shown, the camera self-calibration device 410 includes at least one processor 411 and a memory, such as a read-only memory (ROM) 412 or a random access memory (RAM) 413, communicatively connected to the at least one processor 411. The memory stores computer programs executable by the at least one processor. The processor 411 can perform various appropriate actions and processes based on the computer program stored in the ROM 412 or loaded from storage unit 418 into the RAM 413. The RAM 413 can also store various programs and data required for the operation of the camera self-calibration device 410. The processor 411, ROM 412, and RAM 413 are interconnected via a bus 414. An input / output (I / O) interface 415 is also connected to the bus 414.

[0168] The plurality of components in the camera self-calibration device 410 are connected to the I / O interface 415, including: an input unit 416, such as a keyboard, a mouse, etc.; an output unit 417, such as various types of displays, speakers, etc.; a storage unit 418, such as a magnetic disk, an optical disk, etc.; and a communication unit 419, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 419 allows the camera self-calibration device 410 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0169] The processor 411 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the processor 411 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 411 performs various methods and processes described above, such as the camera self-calibration method.

[0170] In some embodiments, the camera self-calibration method can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 418. In some embodiments, part or all of the computer program can be loaded and / or installed onto the camera self-calibration device 410 via the ROM 412 and / or the communication unit 419. When the computer program is loaded onto the RAM 413 and executed by the processor 411, one or more steps of the camera self-calibration method described above can be performed. Alternatively, in other embodiments, the processor 411 can be configured to perform the camera self-calibration method by any other appropriate means, such as by means of firmware.

[0171] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0172] Computer programs used to implement the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be implemented in a specialized computer or other programmable processing apparatus to provide the processor of the camera self-calibration device to cause functions / operations specified in the flow charts and / or block diagrams to be implemented when the computer programs are executed by the processor. The computer programs can be executed in whole on the machine, partially on the machine, partially on the machine as a stand-alone software package, partially on the machine and partially on a remote machine or entirely on a remote machine or server.

[0173] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store the computer program for use by or in connection with an instruction execution system, apparatus, or device. Computer-readable storage media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0174] To provide for interaction with a user, the systems and techniques described here can be implemented on a camera self-calibration device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the camera self-calibration device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0175] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0176] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, and solves the defects of large management difficulty and weak business scalability in traditional physical host and VPS (Virtual Private Server).

[0177] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present disclosure are achieved, and the present disclosure is not limited herein.

[0178] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements, and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A camera self-calibration method, characterized in that, The method includes: Based on the continuous image sequence acquired by the target camera, at least one adjacent image pair is constructed; the adjacent image pair includes two images; the two images in the adjacent image pair are acquired at adjacent times; For each adjacent image pair, determine the target basis matrix of the adjacent image pair; Based on the target basis matrix of each adjacent image pair, a target function based on the essential matrix manifold constraint is determined, and the target function is optimized to obtain the optimal intrinsic parameters of the target camera; The pitch angle of the target camera is determined based on the optimal intrinsic parameters and the continuous image sequence.

2. The method according to claim 1, characterized in that, Determining the target basis matrix for each adjacent image pair includes: For each adjacent image pair, extract the feature matching point pair in the adjacent image pair; the feature matching point pair includes two matching feature points. Based on each of the aforementioned feature matching point pairs, determine the first fundamental matrix of the adjacent image pair; Based on the first fundamental matrix of the adjacent image pair, select the first matching point pair that satisfies the geometric constraints from each of the feature matching point pairs; Discard adjacent image pairs whose inlier ratio is less than a set threshold; the inlier ratio is the ratio between the number of the first matching point pairs and the number of the feature matching point pairs. For each pair of neighboring images that is not discarded, the target basis matrix of the neighboring image pair is determined based on the first matching point pair of the neighboring image pair.

3. The method according to claim 2, characterized in that, The step of determining the objective function based on the essential matrix manifold constraint according to the target basis matrix of each of the adjacent image pairs includes: For each adjacent image pair, the essential matrix of the adjacent image pair is determined based on the target basis matrix of the adjacent image pair and the preset prior camera intrinsic parameters; the prior camera intrinsic parameters include prior horizontal focal length, prior vertical focal length, prior principal point horizontal coordinates and prior principal point vertical coordinates; Perform singular value decomposition on the essential matrix, and calculate the rank-constrained residual and singular value equality residual corresponding to the adjacent image pairs based on the singular value decomposition results; The target residual is obtained by fusing the rank-constrained residual and the singular value equal residual corresponding to each of the adjacent image pairs. Based on the residual weights and residual weight coefficients, the first focal length residual between the focal length of the target camera in the horizontal direction and the focal length in the vertical direction, the principal point prior residual between the principal point coordinates of the target camera and the preset prior principal point coordinates, and the second focal length residual between the focal length of the target camera and the preset prior focal length are determined. The target residual, the first focal length residual, the principal point prior residual, and the second focal length residual are fused to obtain the objective function based on the essential matrix manifold constraint.

4. The method according to claim 3, characterized in that, The optimization of the objective function to obtain the optimal intrinsic parameters of the target camera includes: The target residual weight value is calculated based on the number of adjacent image pairs and the number of first matching point pairs of each adjacent image pair; The residual weights in the objective function are assigned the target residual weight values, and the residual weight coefficients in the objective function are assigned preset values. The prior camera intrinsic parameters are iteratively optimized with the goal of minimizing the value of the objective function to obtain the first camera intrinsic parameters of the target camera. Based on the first camera intrinsic parameters and the prior camera intrinsic parameters, calculate the first residual weight coefficient value, and use the first residual weight coefficient value as an auxiliary coefficient value; The residual weights in the objective function are assigned the target residual weight values, and the residual weight coefficients in the objective function are assigned the auxiliary coefficient values. The prior camera intrinsic parameters are iterated with the objective function value minimized to obtain the second camera intrinsic parameters of the target camera. Calculate the second residual weight coefficient value based on the second camera intrinsic parameters and the prior camera intrinsic parameters; If the value of the second residual weight coefficient is less than or equal to the preset threshold, then the second camera intrinsic parameter is determined as the optimal intrinsic parameter; If the second residual weight coefficient value is greater than the preset threshold, the auxiliary coefficient value is updated to the second residual weight coefficient value, and the process returns to the step of assigning the residual weight in the objective function to the target residual weight value, and assigning the residual weight coefficient in the objective function to the auxiliary coefficient value, and iterating the prior camera intrinsic parameters with the objective function value as the minimum as the objective function value to obtain the second camera intrinsic parameters of the target camera, until the second residual weight coefficient value is less than or equal to the preset threshold.

5. The method according to claim 1, characterized in that, Based on a continuous sequence of images captured by the target camera, construct at least one adjacent image pair, including: Based on the continuous image sequence acquired by the target camera, at least one original image pair is constructed; the original image pair includes two images; the acquisition times of the two images in the original image pair are adjacent. For each original image pair, determine the structural similarity between the two images in the original image pair, and use the structural similarity as the structural similarity of the original image pair; Based on the structural similarity of each original image pair, adjacent image pairs are selected from each original image pair.

6. The method according to claim 1, characterized in that, Determining the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence includes: For each image in the continuous image sequence, identify the scene depicted in that image; If the scene is a road scene, then the first road boundary line and the second road boundary line of the road mask in the image are extracted, and the intersection point between the first road boundary line and the second road boundary line is determined as the vanishing point; If the scene is a farmland scene, then extract the center line of the mask for each crop row in the image; The vanishing point is determined based on the intersections between the centerlines described above; Based on the vertical coordinates of the vanishing point and the optimal intrinsic parameters, determine the camera pitch angle corresponding to the image; The pitch angle of the target camera is determined based on the camera pitch angle corresponding to each image.

7. A camera self-calibration device, characterized in that, The device includes: An image pair construction module is used to construct at least one adjacent image pair based on a continuous image sequence acquired by a target camera; the adjacent image pair includes two images; the two images in the adjacent image pair were acquired at adjacent times; A matrix determination module is used to determine the target basis matrix of each adjacent image pair. The intrinsic parameter determination module is used to determine the objective function based on the essential matrix manifold constraint according to the target basis matrix of each adjacent image pair, and optimize the objective function to obtain the optimal intrinsic parameters of the target camera; The pitch angle determination module is used to determine the pitch angle of the target camera based on the optimal intrinsic parameters and the continuous image sequence.

8. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform a camera self-calibration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the camera self-calibration method according to any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the camera self-calibration method according to any one of claims 1-6.

Images